As mentioned, it's your car, do what you want. i wouldn't, for all the reasons stated.
One consideration may be the resale. Given the overwhelming majority would not do this to their car, your future buyer pool may shrink enormously as they won't know exactly how you used the hitch (really heavy stuff that's bad for the car, or lots of salt water dips without rinsing. You might bec losing hundreds of dollars for every use, in which case, go rent a pickup for the day.
Drive fast, take chances.