I have driven the E39 M5 and the 2005 and 2006 E55 AMG (very similar to the 2003).
My thoughts about the E39 M5; beautiful interior, great quality, smooth ride, good brakes, good power, good transmission, good audio system. Only fault, the steering wasn't very communicative, not terrible, but a little vague. Only complaint is the steering.
THe 2005/6 E55; beautiful interior but compared to the E39 M5 the dashboard is not full leather. Good ride, but slightly vague with the air suspension, also long-term reliability of the air suspension is a question mark. The POWER is amazing, pointless because in the USA the police would stop you, but it's a really nice to have that kind of effortless power, amazing power. Brakes are very capable, but the feeling is a bit strange because it's a computer controlled braking system so the brake pedal has artificial feedback; but the hardware is a 8 piston front caliper by Brembo. The steering ratio is a little slow, not a huge amount of feel, and a little bit on the light side; but the steering is probably better than the E39 M5 overall because it's more precise.
The E Class AMG isn't perfect, but it's a nice car. And with all the acclaim the E39 M5 gets, nobody seems to mention the steering isn't perfect. In the end, I would say the M5 has fan boys, and if I try not to be a fan boy, I would say the E55 is worth looking at.
And I've driven the E60 M5 as well. And while I really like the E60 M5, it genuinely gets an emotional response from me, I would really not like to have it on a daily basis. The car is not a car you can take clients or colleagues out to lunch in. It's not something you can take your mother or grandmother in. If you get the SMG version, the SMG doesn't shift smoothly no matter what "level of aggression mode" you're in (it has 6 modes) unless you shift at something higher than 3500 rpms. It gives your passengers a lot of noise, some vibration, and your passengers will be constantly wondering what's going on. The M5 makes a very good weekend car in my opinion. The E60 M5 has good handling, good braking, good acceleration, an annoying transmission, too firm of a suspension, and a beautiful full leather interior. Nice car. I want one, just to hear the sound of the V10 engine. That sound is so good...