First, try a search, whether through this forum or google. There are plenty of answers out there to your questions, but I'll summarize.
There are essentially four pieces to your exhaust system:
2. s-pipe ("section 1")
3. x-pipe ("section 2")
4. mufflers ("section 3")
The s-pipe looks like an S and x-pipe looks like an X. Gains are typically largest from swapping out exhaust pieces closest to the engine. Catalytic converters are located in your headers and s-pipe, and a resonator is located in your x-pipe.
1. Your primary catalytic converters are located in the headers. Installing catless aftermarket headers yields 15-25 hp depending on quality and design. In header design, you can focus torque gains in specific rpm bands. **There's no magic header that does it all** Supersprint stepped headers yield the most power gains of any headers overall, but they are focused around high rpm horsepower. V1 stepped > V2 stepped; however, V1 stepped require a special Supersprint s-pipe, whereas V2 stepped mate to the OEM s-pipe. Non-stepped regular headers add more torque (and therefore more power) at low rpm. Buy headers tuned to the rpm band in which you want to maximize your gains.
If you delete your primary cats by installing aftermarket headers, your car will not be emissions legal and the secondary O2 sensors will cause a check engine light. The light can be defeated with a tune, simulator, or by relocating the secondary O2 sensors beyond the cats in the s-pipe. The ECU compares the secondary O2 sensor readings with primary O2 sensor readings to determine if your cats are running efficiently. Obviously if you delete the primary cats with a header install, the computer will sense that the cats aren't running efficiently because they simply aren't there. By relocating the sensors to post-cats in the s-pipe, the computer will see a catalytic converter between the primary and secondary O2 sensors and viola- no light. Your car still won't be legal, but you likely won't get caught unless you're in Cali and are being inspected by a thorough tester.
Using a tune to defeat the O2 sensor code will cause you to fail an OBDII readiness test during inspection. However, if inspections aren't an issue, a tune is a an easy way to go.
I've yet to see a simulator that avoids a light on a Z4M. I'm going to be making my own when I install my headers on my 3.0i.
Headers add a lot of good sound and a little bit of bad sound. Most dislike the added rasp (a resultant from deleting the primary cats), but the tradeoff in other sounds and extra power is considered worth it.
2. Swapping your OEM s-pipe can add up to 10 hp. However, beware that installing catless headers and a catless s-pipe will add A LOT of rasp. The majority of the population will think the car sounds terrible unless you have additional resonators welded in. The sound will be exceptionally polarizing. And, without cats in the headers or s-pipe you have no way of fooling the computer into thinking you have primary cats installed through relocation of the O2 sensors.
You CAN keep your stock headers in place and only swap the s-pipe without triggering a check engine light. As you are removing an emissions device, it's still not legal... but it requires much less work and is cheaper than a header swap.
3. There's little power to be gained from swapping the x-pipe. The OEM x-pipe has a resonator in place that eliminates some rasp and improves sound. However, if you are keeping the stock headers and s-pipe in place, the x-pipe is a good place to get some burble and pop on downshifts without a lot of extra noise. The car will sound a little meaner and dirtier with an aftermarket x-pipe. Again, most notice little to no performance gain.
4. Mufflers offer the least gain of any piece, but have the greatest impact on sound. RPI mufflers are the loudest. Many consider them too loud even with an otherwise stock car, and owners add sound deadening materials just to make the car tolerable. I personally have Rogue Diablos, which are about one notch down from RPI (maybe 75%). If you are replacing the rest of the exhaust system, consider going for "sport" rather than "race" mufflers... especially if you've never done this before. People deal with exhaust drone differently. Remus, Supersprint Sport, Eisenmann Sport, and Stromung are good places to start.
My advice is to spend your money learning how to drive the car first, then worry about trying to get more power. Add some mufflers and enjoy the car.
If you're set on spending tons of money on the car, browse the forums (zpost, bimmerforums, etc) for some second hand parts. I bought my Z4M V1 stepped headers for $450 shipped; my 3.0i's regular Supersprint headers for $700 shipped (Jet Hot coated too, and brand new!); and a new Supersprint race muffler for $300.
Finally, use the search feature!
G Power Z4M
and VF Engineering Z4 3.0i