This professor is not my official thesis supervisor. I have taken his class. I often discuss research with him and has sent him my work which he read and discussed with me in depth. So he is going to recommend me for PhD. The schools ask me to specify my relationship with him. I don't think "course instructor" or "research advisor" would be appropriate because the former may be interpreted as I only take courses with him and he does not know my research whilst the latter is not strictly speaking true because he is not actually my thesis supervisor and I don't have an official relationship with him as advisee and advisor. That said, I feel I have talked to him almost as often as I speak to my thesis advisor. I feel he is more like a "research mentor" but what is the right way to say it on an application?
Sounds like this person is a “course instructor and mentor”. Honestly it does not matter what you put on the form - the letter is what matters. Typically I don’t look at what the student says about the relationship on the form, I only look at the letter.
If you have to check a box from a set of fields, I think course instructor is closer to accurate than research advisor. If you can write one free hand than feel free to use the "course instructor and mentor" if you choose.
I actually don't think it's that critical what you put in the label. It will be normal for many students to pick instructors that they had some interaction with above and beyond the classroom. I would just request your letter writer to emphasize this distinction: "It might help me if you emphasized our collegial research interactions, not just the course work."