I had this same issue on my 2003 X5 3.0i and could not figure it out. I replaced the tie-rod ends but that was not the issue. It ended up being the front control arm bushings and lower ball joints. Also, had the right side CV joint half shaft replaced - the CV boot was torn. My X5 drives like brand new now with 128,500 miles.
If you can move the front control arm (wishbone shape) then the bushings and lower ball joints are the issue.