True patience isn't a sign of weakness, but of courage. It certainly doesn't mean to let everything happen passively. Patience gives you the capacity to act correctly without being blinded by hatred and a thirst for revenge, which deprive you of any capacity for judgment. As the Dalai Lama often says, true tolerance isn't a question of saying, ‟Come on, do me some harm!” It's neither submission nor resignation—it's accompanied by the strength of mind and intelligence.
Sweet words, if spoken with the intention to deceive, look like kindness but are actually violent. Conversely, for a mother to push her child strongly away from a fast approaching car may look like violence but is really saving the child's life. What counts is the motivation behind our actions and the final result of those actions.
Violence fuels violence, and usually has disastrous effects that keep on repeating themselves. So it's best to avoid it by all means and resolve conflict through skillful dialogue.