Does it really matter being the 4th or 5th author? I wonder if it is worth being the "peace-breaker" for something not important.
Not even a little bit. In some fields, authors are alphabetical. In others, being first (or sole) author is important. In others, the last author has a particular meaning (usually the lab's PI). But I seriously doubt there are any instances ever where being fourth instead of fifth would make a difference.
So, I definitely would not expend any goodwill on this.
I am the most junior one in the team. I guess that is the reason they put one person who contributed way less than me in front of my name. Is it common practice to give less experienced people lower rankings.
Varies widely. But do bear in mind that the more senior people may have contributed to the infrastructure / lab generally, even if their contributions to this specific paper are more modest.
Since I am only unhappy with one person's ranking. How to raise this issue without targeting one particular person.
Tactfully and privately. In your case, you would probably discuss this with your boss and they would take care of communicating the decision, if they decide to bump you up. (But in this case, I would not raise the issue at all.)
My boss is the corresponding author. And the first author is an external collaborator. Should I discuss this directly with the first author who put everything together directly or check with my boss first?
Definitely check with your boss first. Jumping over your boss's head is generally a nuclear option, even outside of academia. And in this case, even if the external collaborator agreed with you, they would have to discuss with your boss in any case, so jumping over your boss's head would have no real upside (and plenty of downside).
As you gain more experience, you may eventually be able to go to the external collaborator directly, and it will be sufficient to inform your advisor that you have done so. But you should definitely err on the side of getting permission from your advisor before contacting external people (especially for something this delicate).