It doesn't matter if the wheel doesn't touch the lugs because the wheels do touch the lug nuts, and so that becomes a defacto point of contact between the wheel and the lug; i.e. they still touch as far as physics is concerned. Being hub-centric only the edge of the spacer is on the hub, and only the edge of the wheel is on the spacer. When it's just wheel to hub there is no lateral distance and therefore really no leverage, but the spacer adds the lateral dimension and therefore the potential leverage. So I have to think that when rolling straight and flat there is little danger, but when you add turning and travel force there most certainly is some leverage. If this was not true then use an example of even the adapter with its own lugs. If there is no leverage then in theory, using normal length studs, you should be able to potentially use a spacer/adapter of limitless width. Would you still think it was safe if it was a foot wide? No, even with 2 sets of normal length lugs, such a wide spacer of any type would create leverage that was beyond unsafe. The question becomes how much leverage is safe and acceptable. So what I've done is exagerrate the dimensions to reveal the physical dynamics that are hidden in the use of even a conservative spacer, but which are still there even though the fasteners are able to "overcome" that effect.
If the assembly was not hub-centric then you would have a new danger of shearing, but being hub-centric does not delete the danger from leverage. And yes, the added dimension of width does make the spacer essentially become a lever. The lugs are tight so it is very controlled, try using a spacer with one or two loose lugs and see how much more quickly it fails than with the same loose lugs without a spacer. I rest my case.