Qualitative approach to evaluation of academic production

Question

In Brazil, a government agency, named CAPES, takes care of evaluation of the postgraduate courses. Among other things, they receive criticism, because such evaluation is based on bibliometric approach. Explaining briefly, the main problem of following this approach is that it becomes counterproductive, because the academy spends time on increasing the number of publications, instead of focusing on quality research (it is considered more valuable to increment the postgraduate course classification).

A logical advice would be that the organization needs switch to a qualitative approach (here is a report, where four international guests shared this observation). My questions are the following.

Are there today any countries that can be models in regard to their system of the academic production evaluation? How such transition can be initiated, focusing on a qualitative approach?

Answer 1

I'm from Brazil and I understand you very well about our research/funding system.

Let's branch the problem in topics:

• CAPES classify the papers by "research impact", it's basically how hard is the congress/conference... to accept your paper: (A1,A2,B1,B2,B3,B4,B5,C).

• CAPES is responsible for funding research presentations and they have a very limited money.

• Only A1, A2 and sometimes B1 are really relevant for the society. Nevertheless both are commonly international congress. (let's consider international travels are expensive in Brazil, plus the conference taxes)

• Let's also consider, poor relevance conferences (B2,B3,B4,B5,C) are commonly in Brazil, sometimes on places where there are a lot of universities, e.g São Paulo, Rio, Rio Grande do Sul, etc. Furthermore, all of them are in Portuguese.

Well, at this time I guess you already see the problem... Summarizing, the Brazilian research is very correlated with local congress and that makes everything easy even for CAPES (due to funding), even for researchers (due to easy paper acceptance), then you create a perfect productivity system... With a lot of useless research.

In my opinion, poor impact research are irrelevant. Then, I guess the solution would be change the impact rank and funding to only classify (A1, A2 and B1). This probably would make everyone work hard and then you probably would change the status quo in a long-term.

• Note that is theoretical scenario. As I said before, this kind of system is perfect for lazy researchers and their financiers.