The E46 M3 ring and pinion (3.62 ratio) are definitely compatible with the M5 diff. However, I don't know if the internals of the M5 diff are compatible with the 540i diff, which means the M3 3.62 may not be compatible with your 540i diff.
If I were you I'd buy a stock E39 3.15 out of an auto car as it is a direct bolt in. Before I took my car off the road for the LQ4, I swapped in one of those to replace my OEM 2.81 and enjoyed the better acceleration; highway cruising was only slightly more annoying (higher revs), but above 80mph the engine was really screaming. I would do it again.
If you really have time and money to burn, buy a used E46 M3 ring and pinion and see if it's compatible with the 210mm M3 housing. I probably wouldn't run that combo on the street though-too many revs while cruising on the highway.