If someone asks wouldn't be correct to say "I was just thinking about what you said?".If they seem bothered about the fact that you are not rapidly responding to them.
Most of the time time when an opinionated conversation takes place I often think "do I want to be right or do I want to be happy?"
And when conversations turn to trash talking/gossip about other people I tend not to participate in the negatives and look for the light in the person being discussed.Once you have done this enough the people that do that kind of discussion will know that this does not interest you.(any more?)
remmeber this question? What is the best way not to get influenced by other people’s negativity? and this one Why do you need to be right?

and in the words of timone and pumba "hakuna matata"
peace

Acceptance and removing the veil of duality. Remove the judgement you carry for them and the things they do that annoy you. Maybe try to find compassion through understanding of why they are the way they are. It's not about good, bad, right or wrong. When you judge someone you take on their energy and that is why you become so irritated. Honor & respect who other people are. This does not mean you have to be close to them in a relationship, but when you remove the judgment carried for them the rest will fall away. It will no longer be an issue. Happiness comes from within and not from judgment or being right. And there is no devil. There are negative and positive energies such as anger vs love. And love is the strongest energy there is. So remember when you become irritated by someone, this is your test... will you come from a place of non-judgment and love or a lower vibration of anger and judgment. there is a great book called, "Lifting the veil of duality" by Andreas Moritz. It helped me see a much bigger picture in how to see things and how I attract them into my space. Judgment is what is attracting these energies.

Just so you know... I had to work through this myself. Hope this helps.

It seems to me as though the root of your challenge lies with your belief that you have the power to hurt another's feelings. While it's true that they may not like what you say or your actions, if any, they are the only ones who can hurt their feelings; not you.

While what I'm about to say may seem 'hard core' to some, I believe that starting to practice it was a turning point in many of my relationships and hence in my life. I care about how I feel so much that I absolutely refuse to listen to or participate in other people's pain.

Whenever I sense a situation arising I politely ask the person to change the subject and talk about something else. If they refuse or persist even after agreeing not to I show them my back by turning 180 degrees and walking away...

Each of us can only be of service to ourselves and others when we maintain our vibrational frequency. So what seems hard on the surface is really the best thing one can do under all circumstances.

And ironically, I have so many people wanting to become friends on Facebook that any fear of becoming a social island can be dismissed. Non-participation is the right thing to do :)

EFT is an excellent tool to remove the negative emotions that are causing you distress. FasterEFT.com is great for the right now stresses. Matrix Reimprinting is great for traumatic events of the past, you can rewrite your past, hence rewrite your future. So for your situation, you could use the Matrix Reimprinting and go back to the first time you remember feeling the way you do when people are behaving in a way that causes you stress. FasterEFT would be good at the time they cause you the stress to relieve it now.

I let people be most of the time,but on occasions have been called apathetic because I don't get drawn into what's going on.I'm not apathetic I just don't think a lot of what is going on is important to me.

You can volunteer to be a good listener, without participating in the conversation, or you can choose to remove yourself from the environment immediately by saying, I have an urgent matter to attend too, so I cannot talk with you now!

I know what you mean, you see others as a reflection on you. Do not do this, see others as separate they do not need agree with you, they don't need to do things your way, they can be totally different from you and it doesn't matter. It is nothing that affects you, they do not need agree with you, they can completely oppose you even. Everyone is his/her own self, that is not a reflection on you if someone disagrees, that is not an attack on him/her either.

If we see others as separate, others do not need to agree with us, we can in our heads say well they can believe it that way it doesn't affect my belief.