This is basically true, but if the seed backup is found, then the finder has access to the coins. Even if another password has been set, this can potentially be brute forced.You don't even have to set up the HW wallet. You just have to store the seed somewhere. Ideally your memory + a couple of places.
I think it is more difficult to crack a hardware wallet if it gets found.
And your friend can not make a copy of your wallet (seed).