Caveat: Never owned an X5 but have had an X6 and just upgraded to an X6M (yay!). Would agree with others here that it really depends on your driving style and needs. If you enjoy a sportier drive, the X6 is hard to beat as it does have a unique set of systems designed to enhance handling. Relatively practical: I can get two bikes, gear and a cooler in the back without taking any wheels off.
That said, the X5 is definitely roomier, especially if you have a family and it is your "utility" car to run, errands, car-pools etc.
The looks - I personally love the X6, but it is a polarizing shape so be prepared for some funny looks, especially as it is still relatively rare on the road!