Yes I have heard this about HK tax authorities but have never experienced any such issues. If you are tax resident in any country (outside HK) and you can evidence that the business is carried on outside HK then HK authorities should not have an issue with you. A good local accountant will help in ensuring this is the case.
