You have simply been lucky to this point. The same mailing address and answering questions were of no use to my wife and step daughter keeping their green cards.
Sorry to hear that, but there is something about having territorial residency that helps too I'm sure. Mainland US doesn't pay as much attention to their territories as they should. But considering that I'm a native born American, and she's my wife, you would think that it shouldn't make much difference to them as long as we are married. I'm not hiding, filed income tax with the USVI for money earned in Anguilla. I have bank accounts in DR, US, USVI, and now Anguilla that I'm sure that they are aware of. She travels to US mainland and doesn't stay long either visiting family in Miami alone without issue. I dunno about lucky. As far as I'm concerned, we are not breaking any laws. I just work differently than most Americans do.