You'd no lesser lose an envelope with US$5000 than you would your passport. Have you ever lost your passport in your lifetime?
If you get robbed, you will lose it. Have you ever been robbed? What do you think the chances are that you'll get robbed while having this somewhere amongst your things and the robber will take it?
That about sums up your chances of being parted from your money if you just bring it in an envelope.
Incidentally, I was robbed about 5 years ago in Newark airport on my way to Santo Domingo and I had US$10,000 in valuables plus US$5000 in cash on me. They stole my carry-on while I was checking in, with two friends watching the luggage and not noticing. A little bit of creativity Home-Owners insurance paid me for almost all of the loss - I may have even made out since certain electronic devices were only available in newer and better models and my insurance paid replacement costs. They don't cover cash.