There are always functional trade offs as other posters have listed. However, one thing I feel gets lost in the x3 versus x5 conversation is driving dynamics.
Having both, I would oversimplify the driving experience to the following:
The x3 is a 3 series feeling vehicle with extra space. If you like that lighter, smaller, nimble sports feel, you will probably love driving the x3 more.
The x5 is more like a 5 or even 7 series with more room. If you like a little more heft with a little more luxury, maybe a little less of that nimble small feeling, you will probably prefer driving the x5.
Enjoy your x3!