So the purpose of the
UK company is to hide the supplier's name from the customer?
Why not a UK
bank account? You might need to travel there, but if you can convince the banks to accept you (and they do sometimes accept non-resident UK companies) that's going to be the best-looking structure. I have opened accounts for non-resident companies with several UK banks. Speak to Lloyds, Barclays, and RBS. HSBC is a pain to deal with, but sometimes they will comply.
If none of the UK banks accept you, you likely have a business plan and structure that won't go over well in other non-tax havens, like
Germany, Netherlands, or Ireland (technically a
tax haven, but not in this case). However, you can try banks there; Commerzbank, Deutsche Bank, ABN Amro, ING, AIB, and so on.
If those don't work but your business plan is legitimate and structure is fine, consider speaking to banks in the UK channel islands and Isle of Man, Gibraltar, or even Malta. I know several UK companies that bank with BOV in Malta.
Since you are worried about how your company will be perceived by tax authorities, you may want to avoid banking in places like Switzerland,
Latvia, Cyprus, and so on. Those jurisdictions are often perceived to be used for hiding money. Whether this perception is right or wrong is a whole nother debate.