-3

Talking a lot about people in the academic world, I experienced a general unsatisfiedness of the customs of how the funding agencies (mostly, governmental entities) measure the scientific results. Their general view is that the stats are easy to trick, a large part of the science world is doing that, and that makes most of the publications wrong.

Also various trick were explained, like if "friend" departments cite regularly each other in various publications and so on.

For example, if some U.S. federal ministry decides, how to divide up R&D budget in the next year. I believe, the decision makers doing there the hard job, decide about not little sums and probably not anyone can get into there. Also their obvious goal is to do the best for the science (of the U.S.).

How can it happen that they do not use a better stat? Considering the importance of the budget to divide, I believe there should be human resources available to invent stats which can not be tricked even by scientists, particularly if even the details of the algorithm are not public.

(Related material, partially to show "basic research")

19
  • 5
    I believe every worthwhile stat can be tricked. The people devising the tricks are just as smart if not smarter than the people devising the stats. Nov 9, 2023 at 15:27
  • 4
    This seems like more of a rant than an answerable question.
    – Buffy
    Nov 9, 2023 at 15:27
  • 5
    As a suggestion to improve your "question", consider looking through US funding agencies to see how they state they evaluate research proposals. Then, cite these in your question. Nov 9, 2023 at 17:43
  • 3
    Proposals are evaluated on a very clear matrix by a panel of other scientists. Not sure where you are getting the idea that citation rates play a big role beyond pointing out that one’s previous research was well received and impactful
    – Dawn
    Nov 9, 2023 at 18:02
  • 1
    @peterh The answer suggested in that link is "not very big" - it suggests that "fake papers" are at least an order of magnitude less common (~1%) than legitimate papers whose results cannot be reproduced (>10%). Nov 9, 2023 at 19:35

2 Answers 2

11

I don't believe there is an objective, quantitative/algorithmic way to solve this problem. The systems for "gaming" that you refer to exist because those things are used as quantitative measures. It's a far bigger problem in systems that have used algorithmic solutions entirely to make hiring/grant giving decisions.

The alternative is to bring in more subjective evaluation, to let experts decide what the value of some scientific proposal or accomplishment is. The problem with that is that subjective evaluations are...subjective. They suffer from bias or appearance of bias. Sometimes forces outside science demand a more quantifiable approach because they fear abuse otherwise.

In the US most grant agencies and universities/hiring committees use a combination of these methods, not either in isolation.

2
  • 2
    +1. However, it's not only "forces outside science" that push for more objective criteria, it is certainly also people within academia who fear that subjective criteria give too much influence to "old boys' networks", and I am using the term "boys" advisedly. It is not the case that there is a "good" solution here that is only suppressed by sinister forces with an agenda. Nov 9, 2023 at 18:24
  • 1
    Yes indeed; I didn't mean to imply only outside forces have that influence, rather that sometimes the decisions to use those tools and even which to use are not up to academics themselves.
    – Bryan Krause
    Nov 9, 2023 at 18:28
3

There are two misconceptions in your question:

  • You think that the fact that there are no better ways to detect fake papers is because nobody has cared enough to develop them. But that's certainly not true, many very smart people have looked at this problem, but found that it is just very difficult to automatically detect whether a paper is "fake" or not. The problem is that a paper is just several pages of text, and to detect issues with natural language text is just difficult all around.

  • At a deeper level, it is difficult to detect "fake papers" if you have trouble defining what exactly a fake paper is. These papers are not just a random collection of words, but they are written by humans, oftentimes on topics for which they have made experiments or calculations, that are shown in the papers. To a non-expert, they will look exactly like any other scientific paper. To an expert, they will (i) show results from experiments that are unsuitable to illustrate a specific issue, (ii) use inappropriate statistics, (iii) are on topics that are simply not of interest to anyone. But these "fake papers" are generally not complete fabrications, showing figures with data completely made up, machine-generated text, etc. -- most of these papers took actual work to write, though perhaps less work than writing legitimate papers.

Not the answer you're looking for? Browse other questions tagged .