I know it's not directly about an E53, but I was having similar issues on my E46 (325i). I ended up replacing my lower control arms and the associated bushings, and it solved the "wandering" issues that my car had been having previously. Tire wear seems to be improved as well. Not saying it's the same on your car, but my car is a 2001, and has gobs of mileage, so it's possible that your car is experiencing the same thing. You might also want to look into having at least the outer tie rod ends replaced as well, as worn ball joints on those can lead to some alignment and "tracking" issues as well.
As a recommendation, if you do decide to have any suspension components replaced, take it to a reputable BMW mechanic (either an independent or to a dealership) and have it aligned as part of the service. It's a good idea to have an alignment done every couple of years anyway, but any time I replace parts of my suspension or steering system, I have an alignment done just to be sure.