Peer Review Has Real Problems. Abolishing It Would Be Worse.

Critics of peer review have a point, but the process is still invaluable

---

“The Fence Builders” by N.C. Wyeth, 1915. Brandywine Conservancy & Museum of Art, Public Domain.

According to academic publisher Wiley, peer review is the process whereby experts “assess the validity, quality, and often the originality” of a peer’s research with the aim of “safeguarding the quality and integrity of scholarly research.” And while it is often thought of in the context of academic circles (research papers and grants), the basic concept of peer review is integrated throughout our everyday lives, from the judicial system (“jury of one’s peers”) to sports refereeing and even job references.

There are an estimated 2.82 million academic articles published every year, partly attributed to the “publish or perish” culture in academia. While this high volume is putting an increasing strain on peer review, the practice is still generally supported and is currently the norm. And yet, there is growing sentiment that peer review should be abolished, or at a minimum, undergo major reform. After all, the “peers” in peer review are people, with all the usual biases and blind spots, and so, unsurprisingly, peer review carries these same flaws.

Calls for Reform

The primary criticism of peer review is that it serves as a “gatekeeper” dedicated to preserving the status quo. And since peer review holds substantial influence over what research ultimately gets funded, critics say it stops, or at least slows, progress in science and other scholarly areas. When serious consideration of new ideas can only occur after the gatekeepers literally die (“science advances one funeral at a time”), it’s obvious that there’s a problem. As author David Sinclair put it bluntly, “Peer reviewers are in fact the midwit, careerist paradigm enforcers. They shut down all new thinking.” Even Nobel Prize-winning research gets rejected.

Sociologist and philosopher Zygmunt Bauman sums up the outdated and bureaucratic system this way: “With my tongue in one cheek only, I’d suggest that were our paleolithic ancestors to discover the peer-review dredger, we would be still sitting in caves.” With these sentiments in mind, calls for reform are not surprising.

In addition, critics of peer review point out that it has not been able to assuage the public’s mistrust of research reliability. In particular, they point to the continuing replication crisis, which has plagued so much of scientific and other research. While these critics all have a point, there’s reason to be wary of viewing peer review solely as an outdated, useless, and status-driven process. This perspective is overly pessimistic and misses peer review’s value and unique strengths. Which is why, rather than eliminating peer review or staunchly defending it, the best way forward is to find ways to improve the process.

Start with Transparency

One possible way to make peer review better would be to improve its transparency. The lack of transparency, particularly the anonymity of those conducting the review, sows mistrust and leads to questions like, “Who are these reviewers?” and “Why did they reject my research?” Most review processes maintain the reviewers’ anonymity and provide limited rationale for their decisions, especially when the decision is “no.”

So as a start, one way to lessen the sense of “gatekeeping” could be to ensure that peer reviewers are made known to researchers and that meaningful feedback is provided, even if that feedback is highly critical. This could help strengthen trust, accountability, and fairness in the process.

Losing anonymity could also have the added benefit of curtailing some reviewers from being “outright cruel” and unprofessional with their feedback. The leading scientific journal Nature is already adopting these kinds of strategies and now publishes all reviewer reports and author responses for every research article. Furthermore, the open-access journal eLife uses an alternative approach to the traditional “accept or reject” decision and publishes all pieces that undergo peer review, along with their reviews, even if reviewer feedback is uniformly negative. This removes the power of reviewers to decide which ideas get through while maintaining a form of peer review, but without the gatekeeping.

Although worthwhile, these and other transparency efforts might have an uphill battle to gain wider acceptance. For instance, modern academics often self-censor, and research suggests that only a small minority of academic referees would remove anonymity in their peer reviews.

Can AI Make a Difference?

Other reformers look to AI as another possible way to make peer review fairer and more efficient. Proponents of AI in general view it as a deus ex machina, a one-size-fits-all solution to many of society’s problems. However, when it comes to peer review, we should be cautious about integrating AI too quickly into the process.

In many ways, AI embodies the same flaws of human-led review—errors, bias, and lack of transparency. A recent editorial in Leadership Quarterly clearly presents the risks, detailing special considerations for AI in publishing, such as tendencies to reinforce biases, reward what’s popular over what’s needed, and allow a lack of accountability. I’m reminded of the writer and philosopher Aldous Huxley when he quipped that “progress has merely provided us with more efficient means for going backwards.” Just because it’s new, doesn’t necessarily mean it’s good. On top of these concerns about accuracy and bias, overreliance on AI takes away the reviewer’s unique “voice.”

Still, there may be a role for AI in peer review. For instance, we could possibly use AI for task-oriented work (e.g. proper formatting), while reserving the artistry of reviewing for reviewers. This could help address the criticism that peer review is laborious and slow. Who will make the distinction between “tasks” and “artistry” is up for debate, but these delineations are needed, and soon.

Meanwhile, scientific publishing is already making its position known. For example, Elsevier, the world’s largest scientific publisher, states their policy is for AI to support human judgment, not replace it. Overall, when used appropriately, AI, like any tool, can enhance our efforts. But additional wisdom and guidance are needed to help clarify the optimal use of AI in peer review.

A New Paradigm

Finally, we must commit ourselves to continuously improving the process, in large part because getting rid of peer review would be disastrous for the simple reason that it remains an indispensable tool. Indeed, while rigorous review is resource intensive, it also is crucial for developing and disseminating high-quality research. The very fact that those who submit their work for review know it will be under scrutiny by definition raises standards. A less rigorous process might speed efforts, but would also likely lead to false, misleading, and poor-quality information, as was seen a few years ago with COVID-19 research dissemination. Furthermore, inadequate vetting could lead to disastrous effects, as evidenced by the U.S. Public Health Services’ Tuskegee Syphilis Study.

We should choose to live by the mantra “Feedback is a gift,” and embrace our work being challenged. When done right, peer review refines ideas via an iterative process. What’s more, since academics are infamous for their lack of intellectual humility, peer review is a critical opportunity to receive feedback and get outside of one’s echo chamber. As the late Carl Sagan pointed out, while most scientists don’t like criticism, “the hard but just rule is that if the ideas don’t work, you must throw them away.”

It takes humility and vulnerability to submit one’s ideas to external criticism, but peer review is a powerful tool in promoting self-reflection and professional and even personal growth. Peer review also has relational benefits. Indeed, we often forget about the “peer” in peer review. Sharing ideas with colleagues is one of the great benefits of going to a workplace. Academic life, and life in general, is filled with the proverbial “faculty lounge” and “water cooler talk.” Specific feedback from colleagues in your discipline is an especially important part of developing as a scholar.

Unfortunately, growing mistrust of peer review puts these benefits at risk. So, one important step forward then is to improve the review process while simultaneously coaching researchers to be more receptive to feedback. The goal here is to entirely reframe the perception of peer review from an adversarial dynamic to a welcomed, colleague-based relationship, assuming the positive intent of reviewers who are focused on open inquiry and constructive disagreement.

Researchers would be wise to heed John Stuart Mill’s argument in On Liberty when he writes, “Nor is it enough that he should hear the opinions of adversaries from his own teachers, presented as they state them, and accompanied by what they offer as refutations…. He must be able to hear them from persons who actually believe them.… He must know them in their most plausible and persuasive form.”

At the same time, reviewers should adopt an open mindset as described by C.S. Lewis: “The first demand any work of art makes upon us is surrender. Look. Listen. Receive. Get yourself out of the way.”

There’s no obvious solution or “silver bullet” to peer review’s problems. Even though there are clear flaws in the process, the alternative is much worse. Completely open publishing with no quality checks or rigor? Total AI control of research, funding, and resultant policymaking? No thanks. I’ll take the system we have and trust that moving forward we can reform it for the better.

---