9

Many academic institutes are routinely evaluated by an advisory board. In addition to the public performance metrics, such as publications and funding, the board usually goes on site to meet the personnel and get a better picture of the institute, its culture and its performance.

As a panel member, what information should we try to obtain in priority during the site visit? How should we evaluate what we see and hear, and how much importance should we give it?

9
  • 1
    Are you asking what an on-site inspector should look for to evaluate an institution, or are you asking what an institute should do to look good for the on-site inspectors?
    – David
    Jan 9, 2017 at 20:09
  • 3
    @David "As a panel member" clearly indicates, that the former is meant.
    – Dirk
    Jan 9, 2017 at 20:13
  • 4
    I would find it rather odd, to be on such a panel without having either guidelines to follow or experience with on-site visits (on one side or the other).
    – Dirk
    Jan 9, 2017 at 20:16
  • @Dirk I'd disagree that the meaning is clear. The evaluating board is going to send a panel or committee to do the evaluation, and the institution is going to have a panel or committee whose job is to ensure that the evaluators wind up happy. The follow on questions could apply to either party as well.
    – David
    Jan 9, 2017 at 20:16
  • 1
    @Dirk That's one of the reasons I'm asking. My uncertainty mainly stems from my perception that hopefully the evaluator already knows what they're looking for.
    – David
    Jan 9, 2017 at 20:18

2 Answers 2

11
+50

I have been on two such panels, both for departments in my field, one at a 'sister' university within the same system as my own, and another at an unrelated institution in another state.

My first observation is that you need to understand who will receive, evaluate, and perhaps act upon findings and recommendations of your panel, and what criteria they will use. If you are not personally in substantial agreement with stated institutional goals, you should probably decline the opportunity to serve on the panel. You might be able to 'score a few points' for an alternative institutional policy in the panel report, but typically these reports are not made public and it is unlikely a lone voice will have much effect changing goals. Also, there may be an element of unfairness in judging a program by goals at odds with stated institutional goals.

Before your visit you will probably get a huge packet of information about faculty achievements, student and researcher outcomes, enrollment and employment trends, and budgets. Beware that the most important information may not be highlighted, and look for obvious gaps in this information. Omissions and inconsistencies can be a guide as to what questions to ask when you get to campus. Of course, the institute, school, or department under review has a right to present its best case, but you will do your best job if you look beyond the PR gloss.

Focus on what is actually going on, as well as historical information and stated goals for the future. It is easy for a faculty member or researcher to claim a passionate interest in various trendy fields, but you should look at what she/he has actually published in the past few years and evidence of current activity. It is easy for a department to claim increased student and staff diversity as a goal, but you should look around to see what is actually happening. You should have as many one-on-one meetings with faculty, staff, and students as possible. Ideally, some of these meetings will not be pre-arranged. On your own, have a slice of pizza at a local hangout; sometimes that can give you more information (or clues to search for information) than prepared binders of documents and fancy receptions with carefully chosen people.

0

I would like to guide you on this errand. However, my answer is applicable from the point of view of a student of life and not indicative of this specific evaluative experience. Firstly, one should not indulge oneself in mystery shopping in such a position. That means one must formlly ask the host at the beginning for each required data point / meeting and should not walk into these places looking for hidden secrets. Such responsibilities will likely be part of larger teams by accreditation agencies. Evaluations should also rely on one to one meetings and in the little time possible build trust with the agents being interviewed. Its not very hard to make your interviews about the other person's comfort and realising his grounded point of view. Then one can proceed to realising if they set the right objectives and how much they can and indeed do achieve. Most of your evaluation report would thus like to be positive while pointing out real improvements required and you can choose to be critical exactly where it needs to be said. Also, you should probably want to work in smaller groups and not alone in discusing issues within the evaluation team and probably initiate a discussion to firm up each sub group's responsibility.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .