Context: United States. I'm not sure how applicable what I wrote is to other countries.
Moving departments within the same institution is not common, but should usually be possible through an administrative process that is a lot less cumbersome and competitive than getting a position at the institution in the first place. At my institution where I have seen this happen, the main hurdle that needs to be cleared is that the faculty of department B have to take a vote and confirm that they approve of the faculty member from department A joining their department. Then there is an approval process where various deans and committees have to sign off on the move, but if the faculty of department B are happy with it I think that's basically the end of the story and no one else is likely to object.
For this reason, if you are interested in moving from department A to department B at the same university, I would think that the correct process to follow is to start by exploring the idea informally with department B's department chair, and/or other colleagues at that department who you think may be supportive of your move. Most likely you will either find that department B is not really open to that idea (e.g., maybe they think your research profile is a poor fit for them), or instead you will find that they are indeed open to the idea and would be willing to get the ball rolling on the relevant administrative process.
In any case, I'd bet that you would not be required at any point to apply to an advertised tenure track position the way outside candidates need to do. And I'm also reasonably sure that if you do just decide to apply to such a position on your own initiative using the regular application process, you will be informed that this is not a correct or practical way to approach the department switch idea.
Any advice on navigating this while maintaining good relationships in both departments?
If you approach department B politely and respectfully with your request, I don't see why this would hurt your relationship with anyone there. (If anything, they should be flattered that you want to join them.) Possibly they will tell you no, which could be a bit awkward, but that does not seem like a huge problem to worry about.
On the other hand, I can see that under some circumstances your colleagues in department A may take offense to your wanting to leave their department, especially if you are someone they worked hard to recruit and/or whose presence is strategically important to them. So there is a chance this might sour your relationship with some of them. This could potentially be quite awkward, especially if your gambit to move to department B fails. But again, if you approach the various conversations with tact and respect and offer good explanations of why you are trying to move, hopefully your department A colleagues would understand and you will avoid upsetting them too much. But in the end, even with the best of intentions it may be impossible to avoid some damage to your relationships with them.
Good luck!
