In my mind, i think there are different aspects.
The purpose of the recording is one of the main aspects => for personal use only, for the students of this university, for public publishing, etc
The context of the event recorded is another important aspect. If it's a public meeting, it may be difficult to any party to restrict the recording.
If it's private, it's another story, even if the lecturer is a public personality i think. In some context, you only can be present because you implicitly agreed on some specific rules linked to the event.
If it's secret, it's another another story, ... etc
In a university course for example, do not forget that what your record is owned by the university.
This is not exactly public stuff, so you really should tell and ask the professor if he does agree to be recorded, and also, who will use the record later.
The same should be done for everyone present that could be identified on the record.
I think that everything but "personal use only" may even need any kind of written agreement between all the parties (the recorder, the professor, the university, and even the students that could be identified on the record).
This agreement may contain some conditions, like "you can publish it only if you hide the face of the professor, or of a students, ... "
Some party may also want to keep the possibility to forbid the use of the record afterward.
Anyway, i suppose that if all the parties don't mind, it eventually can stay only a verbal agreement assuming that every party is a witness for the others. But that's pure speculation from my part.
To answer the original question, the reasons for the refusal of one party (not just the lecturer), may be VERY variate (rational or not), and even if it may be frustrating or sounds like an injustice, i am not sure the party even has to tell you why if he don't want.
It is his right to refuse and you have to respect his right.