Ed--
I've noticed that too. I think this has become very prevalent in recent years due to advances in Computer Aided design software used to design the bodyshells and maybe increased used of high strenght steel. I think those "increase" numbers are starting to taper off again-- for example, I think the E65 7 series is only about 10% more rigid than the e38. But basically the answer to your question is that the body shell designers make the car as rigid as they can given the technology, time and budget they have at the time they design the car. Obviously, these factors change with each new designs. I also read somewhere that rigidty has only recently become such a big topic for carmakers-- probably due to higher saftey requirements, and the desire to improve the suspension performance/comfort. So, in the past 5 years, they've focused a lot of energy on it. Initially, it was easy to make big gains over previous designs-- but I'd bet they're reaching a plateau now without going to new materials like carbon fiber.
The other interesting this about this whole rigidity topic is that you never see the actual figures published (yes, people on this board have managed to dig them up for the e46)-- or comparisons between cars. I think someone posted that you can't really compare these numbers between car makers because they all use different techniques to make these measurements-- and some measure the bodyshell w/ glass installed-- others do it without, etc. Despite this, i've tried digging up the numbers for other cars, but haven't found anything.