You can say what you want about me, but the only way that you're truly going to affect me is if I allow your words or deeds to affect me. I can keep my dignity only on my terms, and no one can take it away on theirs.
That is, as long as I make sure that's so.
There truly is nothing that anyone can take from us if we don't let them. When they threw Thoreau in jail for not paying poll taxed, he was amazed that they thought they had taken away his freedom. "I did not for a moment feel confined, and the walls seemed a great waste of stone and mortar."
If he had allowed himself to feel as if they had taken his freedom, then guess what? He would have felt that he had lost it. But that wasn't the case at all--since he refused to surrender his freedom, he still held on to it even while locked away in a jail.