If you sell your Z3, you will likely always regret it. You've undoubtedly suffered through the worst part of the depreciation, and you own it outright. You are never likely to find another one with such low miles on it, especially at the price you will give it up at.
I did the same thing with my 1997 M3/4, which I gave up for a song to trade for a new 330i with an automatic that my wife could drive everyday in San Francisco. Oh, the pain I feel when I see one drive by (although that feeling has been mitigated by my new cabrio). I have regretted that decision ever since.
Still, I hear you on the power thing, and the safety concerns. The M3 is an absolute joy. It's a little rough for an everyday car, unlike the old E36 M3, which I could have used every bit as practically as I do a 330. The new M3 is more 911 than 330 in my experience, but it makes for a wild-eyed sense of giddy happiness that you feel you're missing. And it does restore a lot of practicality to your driving experience, with extra seats and a trunk compared to the Z3.
Is there any way you can keep the Z3 and still get yourself a new car? How much are you expecting to get by giving it up?