1

When discussing the ability for a faculty candidate to find a job, SE users often say that the quality of graduate research overshadows the prestige of the school at which they received a graduate degree. As viewed by a a hiring committee, what are characteristics of "high quality" research?

3 Answers 3

2

Ultimately quality is a judgement call that every reader gets to make.

For example, a paper in mathematics might be considered to be "high quality" to half a dozen people worldwide who happen to be interested in the main result of the paper (or other work). Especially if the question has been open for a while. But other mathematicians may just say "meh" because that result neither interests them specifically, nor advances their own studies.

Reviewers at journals and conferences need to be able to make such judgements and, generally, they do a good job and we respect those judgments when we find published papers. But not everything that is published is of high quality.

But, some of the criteria that reviewers and others will use are as follows.

Is the solution correct? And reproducible?

Is the problem long standing? Or does it break new ground?

Does it open new areas of research?

Is it novel in some sense? Or just an easy derivation from what is already known?

Could I have done this easily? Or does it surprise me in some way?

Is the result widely helpful to either researchers or society as a whole?

Note that all of the above are judgement calls and all imply some scale with, not interesting at one end and very interesting at the other.

But I think the real intent of what you have read, is that it is more important what you do in your research than the general reputation of the place in which you do it. Reputation is based on many things, not all of which are directly relevant to the quality of research. In particular some places that seem quiet generally may be hotbeds of ideas in some niche of some field.

And note that hiring committees may rely on the judgements of others, such as the work being published. They might also rely on just the judgements of writers of letters of recommendations if those writers themselves are in a position both to know and to be honest. But they might also, in some cases, rely on their own judgement. "Well, yes, this is interesting, but not interesting to us at this time."

1

The UK has an official answer that might be of use. In particular, the general criteria are:

• scientific rigour and excellence, with regard to design, method, execution and analysis
• significant addition to knowledge and to the conceptual framework of the field
• actual significance of the research
• the scale, challenge and logistical difficulty posed by the research
• the logical coherence of argument
• contribution to theory-building
• significance of work to advance knowledge, skills, understanding and scholarship in theory, practice, education, management and/or policy
• applicability and significance to the relevant service users and research users
• potential applicability for policy in, for example, health, healthcare, public health, food security, animal health or welfare.

There are also specific criteria for each quality level for the various panels (which I have merged):

In assessing work as being four star (quality that is world-leading in terms of originality, significance and rigour), sub-panels will expect to see evidence of, or potential for, some of the following types of characteristics:
• agenda-setting
• research that is leading or at the forefront of the research area
• great novelty in developing new thinking, new techniques or novel results
• major influence on a research theme or field
• developing new paradigms or fundamental new concepts for research
• major changes in policy or practice
• major influence on processes, production and management
• major influence on user engagement
• outstandingly novel in developing concepts, paradigms, techniques or outcomes
• a primary or essential point of reference
• a formative influence on the intellectual agenda
• application of exceptionally rigorous research design and techniques of investigation and analysis
• generation of an exceptionally significant data set or research resource.
• a primary or essential point of reference
• of profound influence
• instrumental in developing new thinking, practices, paradigms, policies or audiences
• a major expansion of the range and the depth of research and its application
• outstandingly novel, innovative and/or creative.

In assessing work as being three star (quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence), sub-panels will expect to see evidence of, or potential for, some of the following types of characteristics:
• makes important contributions to the field at an international standard
• contributes important knowledge, ideas and techniques which are likely to have a lasting influence, but are not necessarily leading to fundamental new concepts
• significant changes to policies or practices
• significant influence on processes, production and management
• significant influence on user engagement
• novel in developing concepts, paradigms, techniques or outcomes
• an important point of reference
• contributing very important knowledge, ideas and techniques which are likely to have a lasting influence on the intellectual agenda
• application of robust and appropriate research design and techniques of investigation and analysis
• generation of a substantial data set or research resource.
• an important point of reference
• of considerable influence
• a catalyst for, or important contribution to, new thinking, practices, paradigms, policies or audiences
• a significant expansion of the range and the depth of research and its application
• significantly novel or innovative or creative

In assessing work as being two star (quality that is recognised internationally in terms of originality, significance and rigour), sub-panels will expect to see evidence of, or potential for, some of the following types of characteristics:
• provides useful knowledge and influences the field
• involves incremental advances, which might include new knowledge which conforms with existing ideas and paradigms, or model calculations using established techniques or approaches • influence on policy or practice
• influence on processes, production and management
• influence on user engagement.
• providing important knowledge and the application of such knowledge
• contributing to incremental and cumulative advances in knowledge
• thorough and professional application of appropriate research design and techniques of investigation and analysis. • a recognised point of reference
• of some influence
• an incremental and cumulative advance on thinking, practices, paradigms, policies or audiences
• a useful contribution to the range or depth of research and its application.

In assessing work as being one star (quality that is recognised nationally in terms of originality, significance and rigour), sub-panels will expect to see evidence of, or potential for, some of the following types of characteristics:
• useful but unlikely to have more than a minor influence in the field
• minor influence on policy or practice
• minor influence on processes, production and management
• minor influence on user engagement.
• providing useful knowledge, but unlikely to have more than a minor influence
• an identifiable contribution to understanding, but largely framed by existing paradigms or traditions of enquiry
• competent application of appropriate research design and techniques of investigation and analysis.
• an identifiable contribution to understanding without advancing existing paradigms of enquiry or practice
• of minor influence.

-4

It's published in a reputable journal or reputable book press. "Reputable" means it has a reputation for having some quality control, as opposed to a predatory publisher.

10
  • 1
    @Buffy Maybe in your field, but in my field new faculty often have 30+ publications in reputable journals. Feb 13, 2020 at 1:07
  • 2
    I generally find that, as long as the journal is reputable, that there is not a dramatic difference in paper "quality" for many papers in a higher impact journal versus a mid range journal in my area. Some of the papers in higher impact journals maybe fancier, but that doesn't make them higher quality.
    – User2341
    Feb 13, 2020 at 1:09
  • 3
    Peer reviewed conference papers count as journal articles in, for example, CS. Feb 13, 2020 at 1:25
  • 2
    Downvoted. The quality of any particular research paper is at best only loosely correlated with the reputation of the publication venue. Some papers on arXiv have stellar quality; some papers in Science are sensationalist garbage.
    – JeffE
    Feb 14, 2020 at 15:12
  • 1
    @JeffE however, this discussion is not about the objective quality of research papers but about what hiring committees look for - and I can certainly imagine many hiring committees which will really prefer candidates with some sensationalist garbage in Science, and not care about stellar papers on arXiv. It's also quite plausible that they won't read any of these papers in detail, they won't try to evaluate the merits of any particular paper, but they'll look at general metrics - the impact factors, citation rates and other external proxies, so that is what will matter for hiring.
    – Peteris
    Feb 15, 2020 at 20:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .