I've had a 540i and an X5, but now a '13 X3.
There is substantial difference in quality between a 5/X5, and having been in many 7's too, versus the x3. It is not just the interior details; it's the insulation, the fit and finish, and the overall solidity. Suffice to say that a 5 series or X5 is way WAY more solid than an X3.
Things that are better built cost more to make, and are then sold at a higher price; it's pretty much that simple.
To ME, the X3 feels like a cross between a traditional solid-as-a-rock BMW and a cheaper but competent brand like VW. I guess you could say it has a 3-series level of fit and finish and solidity, although, to be honest, the 3 series's I've been getting as rentals all these years seem more solid than the X3, too. I kinda get the impression they cut some corners and used "value" components to keep the costs down and get it slotted in under the X5 for marketing purposes. Buttons have some wiggle; doors are too light and don't go "thunk".
That said, I really enjoy the X3. It's a blast to drive. I like going places in it better than I ever did in the X5, and definitely better than the 540i, which I wasn't that crazy about, even though it was a tank. If I had to start over and pick just one BMW to drive for the next 5 years, it'd probably be an X3. I just really like this SUV a lot, even if it doesn't scream "Premium!!!!" like some other BMW models. It is lighter, and more nimble, and more casual in feel than the cut-out-of-a-single-piece-of-granite alternatives on the BMW lot.
2013 X3 28i
1997 Merc C280 Sport
Retired: 2005 X5 4.4i ... 2002 540i ... 1999 M3 ... 1997 328i ... 1988 535i