I'd be a bit hesitant about giving the M3 up. I never owned either of them, but my dad had an M3 and has an M5. The M3 is just purely the perfect car. I can't imagine anything better. Sure the M5 has more power and the SMG is fun, but it just feels like you're driving your car, and it's not too exciting. You don't feel liek you're going fast in it. In the M3, the car becomes a part of you. You don't drive the car to somewhere, you just go and it takes you. There's also the issue of the M5 being bigger and heavier, so in-town driving just isn't as much fun, but if what you're mostly doing is on the freeway, then I suppose there isn't a big difference.
I'd at least test drive the M5 before giving up your precious M3.