It's a tough call really.
When comparing to my lighter 2800lb car I was surprised at first with how much more force the X1 needed on the non-servo. It wasn't huge, but you do notice at first. I think most men wouldn't mind it and in fact you do feel the road a little more and IMO that's better for performance driving. I decided on the 35 though, which has servo standard and no option to not have it. I think the 3 pro's for it are that anyone can drive it comfortably, if you have to turn the wheel real quick (deer, balls, avoiding an accident, autocross) you can do it much easier, and finally the adjusting ratio means more finite control when you need it and less when you don't.
If a guy is driving, don't get it unless you like a more numb road feeling. If you autocross then you might consider it.
I'd say get it though if a smaller lady is driving the car.