The XDM does have a grip safety like a 1911, so it has another layer of unobtrusive safety built in over something like a Glock. You have to grip the gun for sure, and then pull the trigger for it to go off. With that, and a decent kydex holster that covers the trigger area, I can’t imagine that any halfway deliberate person would need a manual safety.
Which reminds me of an incident that happened with my hunting buddy when we were scouting for deer. A vicious prairie dog was attempting to attack him (honest

) and I watched him pull out his manual-safety-equipped M&P to dispatch it. Of course he tried to pull the trigger and nothing happened. The safety was on because it is his normal habit to carry it with the manual safety engaged. So even with plenty of practice drawing and disengaging the lever, under pressure he failed to remember to do this. The prairie dog had disappeared. If it had been the vicious rabbit from Monty Python, his throat would have been ripped out! The moral is: if more that casual speed or urgency is involved, I think the grip safety of the XDM offers plenty of redundant AD safety.