Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It only takes a minute to sign up.
Sign up to join this community
Many academic institutes are routinely evaluated by an advisory board. In addition to the public performance metrics, such as publications and funding, the board usually goes on site to meet the personnel and get a better picture of the institute, its culture and its performance.
As a panel member, what information should we try to obtain in priority during the site visit? How should we evaluate what we see and hear, and how much importance should we give it?
I have been on two such panels, both for departments in my field, one at a 'sister' university within the same system as my own, and another at an unrelated institution in another state.
My first observation is that you need to understand who will receive, evaluate, and perhaps act upon findings and recommendations of your panel, and what criteria they will use. If you are not personally in substantial agreement with stated institutional goals, you should probably decline the opportunity to serve on the panel. You might be able to 'score a few points' for an alternative institutional policy in the panel report, but typically these reports are not made public and it is unlikely a lone voice will have much effect changing goals. Also, there may be an element of unfairness in judging a program by goals at odds with stated institutional goals.
Before your visit you will probably get a huge packet of information about faculty achievements, student and researcher outcomes, enrollment and employment trends, and budgets. Beware that the most important information may not be highlighted, and look for obvious gaps in this information. Omissions and inconsistencies can be a guide as to what questions to ask when you get to campus. Of course, the institute, school, or department under review has a right to present its best case, but you will do your best job if you look beyond the PR gloss.
Focus on what is actually going on, as well as historical information and stated goals for the future. It is easy for a faculty member or researcher to claim a passionate interest in various trendy fields, but you should look at what she/he has actually published in the past few years and evidence of current activity. It is easy for a department to claim increased student and staff diversity as a goal, but you should look around to see what is actually happening. You should have as many one-on-one meetings with faculty, staff, and students as possible. Ideally, some of these meetings will not be pre-arranged. On your own, have a slice of pizza at a local hangout; sometimes that can give you more information (or clues to search for information) than prepared binders of documents and fancy receptions with carefully chosen people.
I would like to guide you on this errand. However, my answer is applicable from the point of view of a student of life and not indicative of this specific evaluative experience. Firstly, one should not indulge oneself in mystery shopping in such a position. That means one must formlly ask the host at the beginning for each required data point / meeting and should not walk into these places looking for hidden secrets. Such responsibilities will likely be part of larger teams by accreditation agencies. Evaluations should also rely on one to one meetings and in the little time possible build trust with the agents being interviewed. Its not very hard to make your interviews about the other person's comfort and realising his grounded point of view. Then one can proceed to realising if they set the right objectives and how much they can and indeed do achieve. Most of your evaluation report would thus like to be positive while pointing out real improvements required and you can choose to be critical exactly where it needs to be said. Also, you should probably want to work in smaller groups and not alone in discusing issues within the evaluation team and probably initiate a discussion to firm up each sub group's responsibility.
