Well from the sounds of it you are tax resident in country B then (does not need to be even with 80%)
Just an office is usually not enough to not make country B declare your company as a country B resident / taxable company if you hold the majority of shares. You will need to check your country B
CFC / foreign income / management whatever laws.
Lets suppose you pass country Bs substance tests for those and it it is ok with company being a country A company and being taxed in country A then you should probably check the double tax agreement between your two countries. They will have a dividend / passive income clause that will tell you which country is allowed to tax you in which way on those
dividends. You might get taxed in both countries but country B might give you a tax credit for the 5% paid in country A. It is very unlikely (unless we are talking Malta, Cyprus etc) that you are not going to pay proper dividend taxes in country B with country B's dividend tax rate.
The double tax agreement will also have clause about what constitutes proper local offices / substance tests in which country and what makes which country be the tax receiver for said company or local office / subsidiary of said company.
Mind you this is all very dangerous private knowledge from having to deal with it myself so just a starter for you to know where to look. I don't understand a lot of those parts correctly myself. Final word should come from a tax lawyer - i would always advise two independent lawyers.