I agree with you on the having to look at a touchscreen and potentially distracting. However on the flip side.. the premium brand companies spend a lot of money in R&D into developing good user interfaces for when having to select features 'directly' on the touchscreen or, on some of the hard buttons on an aftermarket head unit. They're then evaluated and tested at the same places where the OEM manufacturers get their systems tested at etc. They have to pass tests and safety related aspects prior to being allowed to be sold etc. At that point, aftermarket and oem systems are at a fairly equal footing. Add to that you can get steering wheel interfaces to control the some of the most often control functions, it further reduces distraction and this is evident on oem and aftermarket audio setups.
Common sense then plays a big part, trying to adjust bass and treble controls, or adjusting your crossover frequency etc on a after market head unit whilst driving is down to the sensibility of the driver. 'Most' drivers won't do that whilst driving and will pull over if it's *really* that important. The ones who take a risk can or will have chances of being distracted for sure, but then a driver in a totally oem level car could be distracted by a child in the back seat, and still cause an incident.