So, I've driven both extensively. I find the Q5 to be more "car-like" with softer handling, brake pedal and interior detail. The X3 is more truck-like with stiffer handling, brake pedal and austere interior detail. The navigation system on the X3 is light years ahead of the Q5 with better controls, detail, screen size and input control.
Best of both worlds? Q5 supercharged 3.0 V6 with BMW brakes, BMW i-Drive and BMW service converage (why should I have to pay an extra $820 for Audi service contract).