I have had my MY 2010 x5 35d for about a year, with 13k miles on it.
So far, it has been awesome. The torque is awesome, especially when going 25mph+, it has rock solid handling (I have adaptive drive, and drive in sport on the highway and normal on the city streets to soften the ride given potholes, construction, etc), and it gets great mileage (about 23-24mpg overall, mostly highway driving). I live nearby many diesel stations, and the chevron I frequent most is very clean.
It is very quiet for a diesel, but it is louder on the outside than the old gas engine (haven't heard the new N55 or N63 versions). I like the characteristic diesel growl, though...it sounds powerful to me. In the cabin and on the road, though, it is very quiet, especially at crusing speeds. Acceleration is audible (more so than my whisper queit 528i), but it is not unpleasant at all.
It does have some turbo lag, so it blasting off at stop lights is your thing (it isn't mine) then it isn't the right choice for you. The 50i (or M for that matter) are certainly better choices for sporty driving, but to me they weren't worth the large price premium (when I bought mine there was the 4500 eco credit, 1800 tax credit, the sales tax/income tax deduction, though there was a phase out with that, and it only applied to purchases made in 2009). I would think if you are looking for the best performance regardless of price the 50i or M are the way to go.
In any event, if you do a lot of highway driving and want the utility, this is the model for you, especially at the price point.