Any idea how to not let it get to me?

The simple and obvious answer would be to just simply quit caring what other people think :)

I know sometimes that particular simplistic type of answer isn't enough, so I'll elaborate a little further.

I know it can be easier, more enjoyable but really, sometimes, I start doubting myself too

As long as you continue to pulse out the vibration of doubt, that is exactly what you will receive back. You need to allow yourself to just play around with the puzzle pieces of physical reality a little bit and start knowing what works and doesn't work for you. It's totally your decision and your personal opinion as to what is real in your reality and what isn't.

For example:

Lets say you were very proficient in mathematics and you were in a room full of people who only know how to add, subtract, and multiply. You tell these people that this complex equation equals a particular formula. Every person in the room disagrees with you and starts to tell their own individual story of how wrong you are because anything other than addition, subtraction, and multiplication are airy fairy and only weirdo new-agey people believe in that stuff.

But you absolutely love complex mathematics and you even went to a very special school to learn it so you absolutely know for a fact that it is valid in your reality. These people can't phase you whatsoever because you have such a passion and strong knowing that this is legitimate in your life.

This makes it easy for you to just get up and leave that room and find a new room that fits your preference (complex mathematics) all while you happily whistle and stroll down the hall with a big smile on your face going over "rocket scientist" type numbers in your head.

The point is, when you know something to be true for you in your physical reality, then it is...period.

Allow everyone else to sit or stand there and experience their unique reality. They cannot penetrate your knowing (or state of being) unless you allow them to do it. The vibration of doubt is pretty much like saying... "I have all these voices in my head telling me what to do and what not to do, so I guess I'll just submit and kinda believe what they say even though I'm not totally sure."

When we get to the point of just choosing an option of what we believe, and leave all the middle stuff out like doubt, being unsure, hesitation, worry, etc., that is when we start to build confidence and day by day get closer and closer to simply just knowing because that is our choice.

As @spacemetalfantasy mentioned in their answer already, everyone you experience in your reality is a reflection of yourself. Every person and circumstance that you encounter in your reality is really just a big mirror made to look like other people and objects to keep you in the game of physical reality on planet earth.

I personally like to use the reflection or mirror analogy as a challenge. This person or these people are acting like this towards me in a way that I don't prefer - light bulb goes on in my head - it is time to get in my proper feeling state that I want to be in.

If you can get yourself in a repetitive habit of experiencing what you don't want (others opinions) and immediately shifting your attention to something you do want (no matter how small that thing may be) it gets to a point where it becomes a habit, just like doubting yourself was a habit as soon as someone else shared their opinion about you.

So my question is basically- How do I overlook their opinions? How do I get unaffected by so many people's views on life?

Put how you feel before anything else.

Sit for a second and think about how you feel when everyone in your family is ranting about suffering and hardships. Really sit for a few moments, close your eyes and focus on how you feel inside.

Now sit and think about things that you want in your life. Think about being with your family and having them talk positively about life and about you. Do this with your eyes closed for a few moments as well.

Your done? Which one felt better to you?

Remember, once you get that good feeling flowing on a regular basis and you allow people to have their opinions with no resistance while you focus on the things you want, the experience you have of those people will shift into something completely different than what you are used to.

Caring about their negative opinions = staying in that reality over and over again.

Caring about how you feel and allowing their negative opinion to automatically shift you to good feeling thoughts = completely different reality in the way you prefer it to be.