There is nothing wrong with being tall first of all, lol. I would just take what they say as a compliment.
I’m a little above 5’4 (So average)
Plus you’ll always encounter people who are taller than you, or shorter. Even if your height is considered average or not average. I’m average height according to google, but a lot of people in my class and high school in general are taller than me.