You often remember something complimentary that someone said about you. Like when the head of my graduate program in musical theatre writing told me she was excited about my potential. (If only potential paid the bills.) Or the person in high school that told me at a party, "When I first met you, I thought you were going to be a total geek, but actually you turned out to be really cool." Well, OK, that one is kind of a mixed bag.
But you ALWAYS remember something truly negative. The doctor that looked at a malady I had at the time and said, "Oh, my god!" The person that, when I expressed a dream that my song be played by Colin Meloy (of the Decemberists), told me my song could not hold a candle to that songwriter.
I've got off rather easy on this account, but I have students who have said they will never get over a bully's words that they were ugly, to a thoughtless teacher telling them they would never be good at math. It will stick with them as long as they live.
Be careful what you say.