Surge Problem Found & Fixed
I recently bought a very clean 2002 530i MT w/ 73K miles that I soon learned had that same "surge" problem that others have experienced. The car would suddenly and randomly drop or add a few hundred rpms without any throttle input. It was really maddening.
I first had a full tune-up and injector cleaning done, but those efforts made no difference. This problem doesn't "throw a code", which makes it very hard to diagnose. I later tested many things myself and thorougly searched the engine for bad vacuum lines, dirty electrical connections, etc., but found nothing. Finally I took it to an excellent independent shop in my area - Bavarian Specialties in King of Prussia, PA. They quickly diagnosed and fixed the problem: worn seals in the VANOS unit.
The VANOS unit controls the variable valve cams. It's located at the front of the engine, and has electrical connections on each side. In essence it consists of two electrically-actuated hydraulic pumps. It works constantly, and Bavarian's mechanics told me that the rubber seals eventually wear until they get hard, leak or break, and then don't actuate properly.
They took the time to show me the internal workings of a VANOS unit after they fixed my car. They said there's a specific diagnostic test that can be performed to check the unit, but that a bad unit sometimes passes all of the tests. My unit passed all the tests except the last one, which they told me checked the length of time it takes to respond to a certain input. That failure indicated to them that the unit was losing pressure, which meant that the seals were bad.
I literally would have sold this car if I hadn't been able to get the problem fixed. Now the throttle is perfectly smooth and I love the car. I hope my experience helps someone else, because I was unable to find any definitive answer in my own research.
Also, I recommend following the suggestion made by some people I've read, as well as Bavarian's mechanics, and having your car's computer software updated at the same time.