I despise formal restaurants. I find all of that formality to be very base and vile. I would much rather eat potato chips on the sidewalk.
Freedom of speech means freedom for those who you despise, and freedom to express the most despicable views. It also means that the government cannot pick and choose which expressions to authorize and which to prevent.
I am content with nothing, restless and ambitious... and I despise myself for the vanity, which formed half the stimulus to my exertions. Oh would that I were one of those plodding wise fools who having once set their hand to the plough go on nothing doubting.
Allah says in the Qur'an not to despise one another. So the criterion in Islam is not color or social status. It's who is most righteous. If I go to a mosque - and I'm a basketball player with money and prestige - if I go to a mosque and see an imam, I feel inferior. He's better than me. It's about knowledge.
If we don't believe in freedom of expression for people we despise, we don't believe in it at all.