Probably the most cringe-worthy thing I've seen on a drill press was chucking up an end mill in a drill chuck. Even a simple counterboring operation will trash the jaws on a chuck. You'll booger your end mill AND the workpiece.
I agree, most drill presses, except for some of the higher end ones,
have thrust bearings, that are meant to take pressure "head on" from pressure put downward when drilling.
Mills have "Timken" style bearings that can take side thrust.
Unless you have a very heavy duty drill press and use light feeds,
I'm afraid you will wreck the bearings using side thrust on a drill press.
Once bearings start getting slop in them, the bit will start to "chatter"
and it will be hard to get a nice finish on the work.