Treaty non-resident means exactly what you say. In a standard OECD tax treaty, there is always a definition for where a company is located. What the UK says is that if, through a DTA treaty, the company is located in the other state, then it's a "treaty non-resident" and pays no taxes in the UK. You need to do some annual filing, but it's basically just copying the balance and results sheets from your accounts. The complexity in taxation usually comes from matching your accounts to various classes of income in the tax system, and you don't need to do that. Also you can't do any business in the UK. The cost of having someone do the filing, secretary services, and address for you is around GBP 200 / year or cheaper. You also only need GBP 1 in equity for a UK Ltd company.
What I like about this setup is that while you have a real UK Ltd company and a real UK ltd company number, you do your business registered as a branch in another country (a country which has a DTA with the UK, but the UK has the most extensive network of DTAs in the world, or close to it). If you need to, you can stop being a treaty non-resident and pay taxes in the UK. You can also often use the identity of your branch when that is advantageous.
Compared to the LLP, the Ltd is not transparent. With both the LLP and the Ltd taxes must be paid, but with the Ltd the taxes are paid by the branch, while for the LLP by the members. With the Ltd, depending on the situation, you might talk about having a UK company, or a [insert other country]-company. Not all forms will ask about where a company is tax resident. For example, some EMI might not like Georgian companies, but might accept UK companies. Will it accept a Georgian tax-resident UK company? Who knows, but I assign it a higher probability than a UK LLP with a Georgian member for example.