Your friend seems to have gotten the idea that "no thanks" is needed to indicate that the answer is "no", while "thanks" is needed to indicate gratitude for the offer. That is not the case. If the email were to contain nothing but the word "thanks", that would be inferred to be acceptance, but "thanks" in an email that otherwise indicates other plans would be understood to be only gratitude, and not acceptance. And "no thanks", as the words indicate, communicates both a negative answer and gratitude.
Thus, "thanks, but no thanks" is redundant. Unfortunately for your friend, it is a form of redundancy that has been used to indicate disdain. To be precise, I would say that it not so much is rude, but has been associated so much with rude intent that it should be avoided (although of course there is a point at which "is taken to be rude" becomes the same as "is rude").
You say that you are not a native speaker, but don't mention whether your friend is. If I were speaking with a non-native speaker, I would give them the benefit of the doubt and not expect them to be familiar with the entirety of English-speaking culture. Even with a native speaker, I would be willing to entertain the possibility that they chose their words poorly without considering the pop-culture context, especially if I had had previous dealings with them in which they were polite. Other people here have indicated that they would be less charitable, which I find unfortunate, so your friend will have to consider the possibility that the professor will share their attitude.