He reasoned that I lacked the understanding of the subject and so not competent enough for a PhD. I agree to his concerns, since I was studying this concept for the first time (for my thesis) and was a bit slow in my approach. On the other hand I'm very passionate about this particular subject and intend to pursue an academic career in this field. I did manage to get a decent score in my thesis and graduated with cum laude. The question is should I give up and take up some job since nothing will move forward without a recommendation.
From what you've written, it sounds like your adviser doesn't think you know enough to be able to research a subject. Knowledge can always be learned. It's actually a very important trait for a PhD student to be able to sit down and learn what they need on their own.
There's no reason not to apply to PhD programs. It's going to be very hard to get into one without a letter from your adviser, but it's not theoretically impossible. The worst thing that can happen is that you spent some time and money and fail to get accepted anywhere. The best thing that can happen is that you get into a program and have the chance to follow your passion.
On advice from Pete L. Clark, I'm adding a caveat: You should be careful as to what sort of programs you're applying to and think carefully about whether those programs will lead to your desired outcome. Going into a PhD program is not something to be taken lightly. It will takes years of your life and be a potential drain on you mentally, emotionally, and economically. To land a decent academic position you need to graduate from a top program and produce high level research. Afterwards you may have to put in more time and effort as a post-doc. Once you do finally get a position, you'll have to work your way through the tenure process.
While I was an undergraduate student at a lower ranked institution in my country, all the professors in the department were thinking that I don't have any academic merit. Just after that I took some time outside the school, and later attended classes at a top-ranked school in my country just as a visiting student.
I was the best in the class and the professor was overly blunt to state that fact in front of the other students. The people in my undergraduate institution were so biased and mediocre that they have never had an intention to give me a chance or were so insidious and sinister to undermine me.
So, sometimes, at some places you simply do not fit with others. You should try your chance at other places, internships etc. A single institution or professor does not suffice to assess your abilities.
A last note: some academics are real jerks, do not give up at the first obstacle.
There are two aspects of this. One is Should I forget about a PhD? and another is Is this a show stopper?
But for both questions, a lot depends on the opinions of other elders in your department. Are there other professors in your department who know your work, whom you could approach for career advice?
When you meet with such a professor, try to keep your questions open ended. It's okay to express an interest in pursuing further studies, but in that preliminary conversation try to keep all options on the table, even if in your heart you feel a strong drive to jump straight into further studies.
Unless, of course, the professor asks you what your preference is.
The question is should I give up and take up some job since nothing will move forward without a recommendation.
You are right, a recommendation from your thesis adviser plays an important part in getting a PhD position (experience!). But giving up is out of question if your words I'm very passionate comes from your heart. I strongly disagree with those who commented, one have to be good at the subject if you want any hope of pursing an academic career. It bear a high degree of vagueness with it because the terms good and hope are incredibly inappropriate in this context as it is not at all clear that OP indeed lacks the potential to be good at the subject.
I think you can do a couple of things here
- Talk to your thesis adviser and ask him politely to explain why he thinks you are not apt for the recommendation.
- Value your adviser's words and spent some time on working out yourself on the subject core.
- If you think that your adviser misjudged you, try for a short term project position, which can be relatively easy to find even without your adviser's recommendation.
I think there are plenty of choices before you and tremendous room for improvement, once again in my humble opinion, its too early to give up.