Compassion is not the same as letting people get away with whatever they want. Compassion is also expressing limits.
Survivors of abuse may often go into over-justifying our boundaries more than being poor at communicating boundaries in general.
If we were to believe abusers and toxic people, then enforcing boundaries is the most selfish act in the world. But are boundaries really selfish?