Huh? Manners went where?

So I get this email from an acquaintance (a coworker of my N&D):

Do you remember last night when I said that you were looking attractive & stylish, to which you replied with an insulting comment about how I must be stoned and shouldn't be allowed to drive? This is not the first time you've needlessly shot down acclaims from me, though this time I've taken note.
I should no longer bother to pay you compliments. You can make all the self-depricating declarations you want, but when you attack a real friend for thinking well of you. Well, you fucked up and hurt some feelings.
I apologize for having to express this in email, but I felt too uncomfortable last night to explain my feelings in person.

So the thing is, he really was stoned and my comment was regarding a different conversation about who should drive. And if he truly was my friend, he would know that I generally ignore any comments regarding my appearance as I find them to be rude, shallow, or often both. So where does white privileged boy get off thinking he has the right to describe his opinion but I don't have the right to state a more immediate reality? He pulls an attitude that doesn't even border on pretension -- it is -- and he thinks I've "hurt some feelings"? I don't get it. So I email back in veiled jokey-sarcasm just how white and privileged he's acting, but he's so there (and it's email) that it slides right over him. And when we finally do actually see each other, his first comment is so disgustingly snobbish, it sets me off. I know he's smart, and no one could be raised to be this obtusely annoying, so he's doing it on purpose.
Whatever happened to the idea that if you don't want to hang out with someone, you just stop hanging out with them?