I did my masters research project with Prof X. While doing this I joined his lab, where he used to ask me to do random tasks i.e. taking part in different projects, conducting lectures to undergrad students, help him preparing presentations, doing literature reviews not related to my project. I did all those things because of my curiosity and passion for learning as many things as I could. Unfortunately, this delayed my progress in my masters research project.
The Prof. X seemed not at all concerned not only about me but also his other students, except when he wanted us to publish. Although the research field was new to me and I had to do everything on my own and I learned a lot. In the end, he was quite happy and satisfied with my work. He offered me PhD position at his lab but having bad experience with him, I politely refused that right now I don't have any plan to pursue my PhD.
After that, I could notice he is very cold towards me. During my writing stage, he continuously burdened me more with different tasks related to lab. Moreover, after submission he totally ignored to pass the examiner's names to faculty and it delayed my masters results further two months, even though I requested him multiple times. Once I came to know about this delay through faculty, I told him very politely that how can he do that? He just said (laughingly): sorry I forgot about it.
This is not the first time I found him selfish and not consistent while having no concern about any of his student.
Now, I am applying for PhD position and I need references. I have other other faculty members with whom I have worked with or taken courses and they are willing to witness my motivation and good performance. The problem is that they require one of the referee must be the main research supervisor, in my case Prof X. I don't trust Prof. X in this matter. What should I do now?