the first thing that comes to mind with what your describing is a problem with your driveshaft and/or your transfer case (pray its not). after doing alot of researching online about my new(used) x5 i found that a major design flaw with the E53's was its transfer case (one of very few american made parts on the E53) and the front drive shaft. alot of people have many different ideas as why this happens to the X5. some people say that drive shaft is far too short in the Spline area. some people say its weak material used to make the drive shaft and some people think it can be due to an unbalance driveshaft or a faulty transfer case. the grinding noise you described is possibly the chain in the TC rubbing against the walls of the case. the chains can stretch and need to be replace it usually costs around 150$CAD not including labour if its just the drive shaft they go anywhere from 400-700$ VERY EASY install. if its the transfer case thats gone it runs around 1200$ and if thats the case chances are you need new driveshaft too. sometimes you can take the TC into a transmission place and get them to replace the chain and gears to save alot of cash. if it is only your driveshaft i recommend upgrading to a new driveshaft with extended Splines because its about an 1.5'' longer than stock which helps get maximum contact and its also made with a stronger material so they dont wear as easily. dont buy a re-manned driveshaft to save cash because this problem happens to the E53 every 80xxx-120xxx kms ebay sells an upgraded driveshaft and i found a site online that sells TC parts called cobra transmission.
from what you explained with the start of grinding turning into clunking and knocking it sounds very much like what i described up top. i would get it checked out asap because if you keep driving and it breaks on you, you will be stranded and towing always sucks. one way to tell if your DS has broken is if you put it in park or drive and it still rolls. so keep your parking breaks on when not driving for the time being untill you find out exactly whats wrong with you E53
hope i helped