It is actually quite common that internships are not the source of reference letters to graduate school. The ideal letter writer knows about you and your work in an academic research setting, with longer periods of time being better - and also ideally has worked with lots of other students, and thus can meaningfully compare you to the "typical" student.
Internships relatively rarely will meet all of these guidelines, and students are generally advised to stick with professors or other similar professional researchers - so to not see a letter from your internship mentor would not be at all unusual. It would perhaps only catch someone's attention if the mentor would otherwise have been an ideal candidate for a letter (a prominent professional researcher in the field and former professor, for instance) - but even then, if your letters are otherwise good from well-chosen people I can't imagine this really being a problem.
My main concern is how you speak of the position. If you or your letter writers mention it - or if only your letter writer mentions it and you didn't include it in any of your materials - that could cause an issue, as it would look a bit odd. If your PhD admission process includes pre-admission interviews at all, I could see that being something you are asked about. If you are asked, how will you respond? Saying your mentor refused to write a letter for you and that it was a position you didn't really want in the first place would raise a number of red flags.
It's something you did that's relevant and took up a significant block of your time, so list it honestly and focus on the positive - the opportunity, whaty ou did, what you learned, the experience you gained. If you feel like you have something to hide - or you are actually trying to hide something - that's extremely likely to come out in other ways and negatively effect how you approach the application process as a whole.