What standards should I apply when evaluating a PhD thesis from a weaker university?

Question

I have been asked to serve on the examination committee for a PhD thesis. It's in the area of machine learning. It's from a university in a developing country. My job is to provide a written report, recommending whether to award the degree (possibly with minor or major changes), or even not to proceed to defence. I won't be participating in the defence itself.

The quality is very bad. I would say it would receive a low grade, if evaluated as a 3-month MSc dissertation, in my university (in a developed country, but not a top-tier university). It's bad in every respect - novelty, experimental design, understanding, motivation, contact with literature, writing, reliance on "predatory" journals, even some (fairly benign) plagiarism. So, my instinct is to recommend it should not proceed to the defence.

My problem is that standards vary between countries and between institutions. How much allowance should I make for this? I do not want to perpetuate a "vicious cycle" which prevents high-quality work in developing countries (eg Breaking the (perhaps perceived) academic (poverty) cycle).

Part of my problem is that I don't know how much weight my report will carry. Maybe the department is used to receiving harsh reports from international committee members, and de-weights them. Or maybe a harsh report from an international member is enough for an immediate fail.

If I could allocate the blame in my review, I would say the supervisor surely deserves most of it. But it is the student who will suffer the most, if they fail. On the other hand, perhaps the supervisor's own PhD was the beneficiary of some lenient reviewing in the past, leading them not to know what standards are expected. I should not perpetuate that cycle either.

I hope my question does not sound condescending or insulting to scholarship in developing countries. I know that great research is done everywhere, and I have great respect for many individual researchers from the country in question.

Edit: of course, there are no right answers here, but I accepted @anon's answer because the "PhD = passport to worldwide community" concept was the most helpful in my thinking. In the end, I wrote a fully honest review from my point of view, and recommended it should not go to defence, but I added a note that I understand the local committee will apply the institutions' own standards. It goes without saying that I was constructive, not negative, and gave a lot of advice for accessible improvements. Thanks to all for a useful discussion!

Answer 1

External examiner = independent, disinterested, and candid expert

Universities engage external examiners in order to add the legitimacy of disinterested expert approval. Since the independence of the external examiner is a vital criterion, it is essential that the university not be allowed to dictate the terms on which you, as this disinterested expert, grant (or refuse to grant) approval. This is an occasion where you can and must be completely candid, acting without fear or favour. Whilst the internal examiner(s) and convenor are supposed to be similarly candid, they may be reluctant to be candid, since they may be too scared to "rock the boat".

[When I say disinterested, I mean you have no vested interests in the university or the candidate; of course, you have a vested interest in maintaing the integrity of scholarhip generally, and it is that interest which you represent in your capacity as external examiner]

As the external examiner, your responsibility is to evaluate the PhD thesis independently, since your duty is to scholarship, not to the university engaging you.

Therefore, you should judge it by the same standard as you would judge any PhD in the relevant field, irrespective of institution. Unlike Bachelor or Master's degrees, the principal criterion for the award of a PhD is universal and generic: a substantial original contribution to scholarship. As an expert in the relevant field (I hasten to add that only the candidate can be an expert in the work itself), it is your prerogative and responsibility to make a determine whether the thesis meets that criterion independent of what the university may or may not require.

A PhD is a 'passport' to recognition as a legitimate academic worldwide. Universities (and some vocational Higher Education institutions) are entrusted with the sacred responsibility to act as 'gatekeeper' to issuing this 'passport'. University regulations for PhD degrees are just a means to an end, existing to guide all parties towards determining what consitutes a substantial original contribution to scholarship. Ensuring compliance with such regulations is the responsibility of the internal examiner(s) and the convenor. The external examiner should disregard the bureaucracy and evaluate a PhD thesis solely on its own merits.

If the candidate has failed to comply with a university regulation (e.g.: he/she submitted too early or too late, did not attend some compulsory workshop, or did not comply with the word-limit) but his/her thesis is worthy of a PhD award (i.e.: the non-compliance is not a matter of academic integrity), that is for the internal examiner(s) and convenor to address, and not your problem as an external examiner.

If the candidate has complied with all university regulations, but his/her thesis is not worthy of a PhD award, it is your duty as external examiner to either recommend against awarding the degree (if the thesis is nowhere close to the standard required) or require corrections before the degree can be awarded (if the thesis is close to the standard required -- "pass subject to corrections" is the usual outcome in the UK). Of course, you should make your criticism constructive and useful, but you must not be lenient.

You are working for the global academic community, NOT for the university (despite the fact that it is the university which pays your fee and travel expenses)

Your responsibility as external examiner is to hold both the candidate and the university to account, and act as a vital line of defence (on behalf of the global academic community) against universities being too lenient (or, conversely, being too hostile to their own candidate... it can happen!). You also have a responsibility to defend the candidate against any absurdities of the university -- that is to say, you should have no qualms about disagreeing with university regulations if you feel they are inappropriate to the work at hand (e.g.: if you think the abstract is too long, you should demand, in your report, that the candidate make it shorter, regardless of how long the university regulations require the abstract to be).

Answer 2

I suggest that you remember that you are evaluating the work, not its author, or the supervisor, or the university. Give it an honest evaluation, based only on what you see before you. If it doesn't measure up to your standards, then say so. Say why. Make suggestions if you have the time. But honesty is required.

One of the reasons, actually, for international reviewers is to give the home institution some assurance that they are producing students on a par with others.

Answer 3

The question is very difficult. One reason developing countries are developing because there are issues in the culture of learning, teaching and research which hold them back, amongst other authority-based or rote-based learning, focusing on unproductive criteria etc.

Now, there are amazing talents in these countries and some very interesting curricula (whose carriers often end up in good position in a 1st world country), but, on the whole, the system is not going to change overnight. On the other hand, the advent of the internet has brought the world's greatest library (ignoring the paywalls) to the fingertips of virtually anybody on this planet. In other words, the essence of a great academic institution which used to be its library, is no longer exclusive. Thus, even developing countries have now, in less resource-intensive topics, the perspective of improving the standing of their academic institutions.

To avoid the structural problems of nepotistic and political considerations, neutral external evaluations are increasingly sought. Research programs are now more and more evaluated by evaluators from established academic locations. It is clear that it is unlikely that the quality of such programs is going to match anything that, say, a top university regularly comes up with, but in a comparative setup, one can still fairly assess which of the proposals are the better of the submitted lot.

The OP's situation is more difficult. They have not been asked to rank, but to decide on a pass/fail. I would be careful not to assign blame to the supervisor, and possibly not even the academic institution - they have to work with what candidates they have, and their education level, existing tradition and resources. The question is, why did they select OP as a reviewer? Because of their expertise field, to rubber stamp the thesis, or indeed because they wanted an objective, developed-country view on the quality of the thesis?

As an academic, I would find it difficult to just waive the requirements I expect from a doctoral level work. But, understanding the context in which it was created, you might consider formulating it politely, compassionately, constructively, and perhaps with indications of what would need to be done to get the work up to passable level.

[I would not put too much weight on downgrading due to publications that fell for predatory outlets - after all, not only can you not expect less internationally experienced supervision teams to be fully aware of that, predatory outlets are sometimes very well disguised. For instance, I remember a case of a predatory conference which closely mimicried another conference which is well known, which managed to push their version to the top search engine ranking - this included name, location (it was in the same country as the real one inducing you to not note the discrepancy; basically a form of gaslighting). We ourselves discovered it by chance, by on/off noticing that there were two different versions of a Program Committee, one of them with none of the researchers you would expect.]

They invited you to evaluate this work. This means they want your opinion. Academic integrity would probably demand that you give them your real opinion, but, in view of the circumstances, with a compassionate and constructive attitude and showing the candidate a way towards passing.

Finally, in the end, I assume it is their local committee that has to evaluate the opinion and give the ultimate verdict. If you feel that the mitigating circumstances still might call for a possible pass, you might consider formulating your opinion in such a way that gives them a back route to pass the candidate should they choose to do so.

In my opinion, the thing to remember that, ideally, with a PhD, that person should be able to apply for a position as a postdoc in a developed country. The long-term goal is that, with time, a PhD from that institution should become trustworthy enough that any good university in the developed world would realistically consider a PhD from that institution as a viable candidate. You have now a small role in helping to move towards this endeavour.

Answer 4

Somewhere in the information pack that the university sent you, there should be an explicit statement of what the requirements for award of a PhD at that university are. At some universities, the only requirement is the classic "substantial original contribution to knowledge". Other universities ask for a substantial original contribution to knowledge and a couple of other things too. I've never been involved in a PhD exam at a university that doesn't require a substantial original contribution to knowledge, but I guess such universities must exist. So the direct answer is: the standard you should apply is the (presumably published) standard for award of a PhD of the university for which you're examining.

Answer 5

If you have the time, you could consider giving him the full truth. Not only what parts are poor, but also why they are poor and more importantly, what would have been a better way of doing it. You can't give him the full tutelage of a supervisor, but since he apparently has a poor supervisor now, some good advice might help this student a lot.

Answer 6

Being honest is not incompatible with adjusting for context. Give your assessment honestly, but not just in absolute terms — also state how you judge the thesis relative to the expected standards of the institution.

Of course, you will probably have to ask/research a bit to find out those standards. Perhaps ask your contacts to send previous examples of what they consider a strong thesis and a weak-but-passable thesis; perhaps ask them to send any official guidelines to follow; perhaps ask for a phone call for an off-the-record chat to get their personal guidelines.

If these turn out to basically agree with your usual standards, then things are straightforward. If they’re much lower than your usual standards, then you can be clear in the report that you are writing with reference to them, along the lines of:

This thesis has major problems XYZ. These are significant scientific flaws, which must be addressed in order to produce good-quality research, and by my own university’s standards, I would not recommend proceeding to defence. However, the thesis does meet Target Institution’s standard of acceptability to proceed to defence, according my best judgement and the guidelines I have received.

You are not compromising your honesty or candour, nor hiding the ways in which the candidate’s (and perhaps the institution’s) research standards need to be improved; but you are also not unilaterally imposing an external standard.

Answer 7

Let's first consider the question of whether a thesis from a weaker university should be evaluated differently. First, we must define what we mean by 'weaker' and 'stronger.' I believe that there are two relevant definitions for this use: one is degree-granting institution and the other is quality of institution. We may also wish to consider institutional prestige as an additional factor.

Now, we must consider the effects of evaluating based on these factors. If one evaluates based on institutional prestige or degree-granting status, then their evaluation will be heavily influenced by reputation and hearsay. Such evaluations can be prone to personal bias as well.

If one evaluates based on institutional quality, then they are evaluating solely on the merits of the research and not hearsay. However, I believe that this can also be prone to bias.

Consider that there are many excellent institutions which may not be well-known. These institutions may have excellent professors and students who produce fantastic research, but because the institution is unknown to most people this work will go unnoticed.

Conversely, there are many institutions which may not be so excellent. These institutions have professors and students who produce less-than-stellar research, but because the institution is well known this work will be noticed!

Therefore, it seems that there are two factors which influence whether or not a thesis will be noticed: quality of the work itself and reputation/prestige.