I had and E46 M3 , but traded up to a 911 (not an S). I thought the Porsche had a superior gear ratio, which was easier to drive around town with less shifting. I thought it had more low end torque, and I definitely like the feel of the stock stick on the Porsche compared to the M3's stock stick.
Needless to say, today (just arrived) I am back into an M3. I have two small kids and unless you have two cars it is hard to beat the all around performance, but usable back seat of the M3.
Hope that helps.