Scientists Are People — And Their Politics Can Show Up In Their Results
A new study highlights why viewpoint diversity on research teams matters.
Viewpoint diversity is an essential part of academic culture. But, individual scholars (like everyone else) are vulnerable to their own ideological biases, which can directly impact their scholarship and teaching.
A new paper has been published showing how scientists’ political opinions about a topic can influence the way they conduct research, and ultimately, the conclusions they draw about hot-button topics like the effects of immigration on society.
For this ambitious experiment, George Borjas and Nate Breznau led a project in which they shared one dataset with 158 social scientists (in 71 research teams), and asked them to run analyses on the dataset with the purpose of understanding the consequences of immigration. The key question was whether scientists’ prior viewpoints on this topic would influence the way they conducted analyses and their general conclusions. Would scientists who were already pro-immigration analyze the dataset differently than those who were anti-immigration?
Before conducting any analyses, the participants responded to a question asking whether they thought immigration laws “should be made tougher” or “should be relaxed” to establish their viewpoints on immigration. Based on this, Borjas and Breznau categorized the research teams as either pro-immigration (scores of 5-6 on the viewpoint scale; approximately 43% of the research teams), anti-immigration (1-2 on the scale; approximately 12% of the research teams), or moderate (3-4 on the scale; approximately 43% of research teams). Moderate teams could have a mix of both very pro and anti immigration researchers, or an entire team of people who reported moderate viewpoints. Overall, the participant researchers in the study skewed strongly pro-immigration (see figure below).
The dataset contained variables on immigration patterns (increases over time) and public sentiment toward social welfare programs. Importantly, the dataset itself was exactly the same for all research teams and the instructions they were given were identical: to test whether increased immigration reduces public support for social policies. In theory, the participant researchers could find that the relationship between immigration and public sentiment could be positive, negative, or neutral (no impact). But in practice, the observed results depend on their analytic decisions. After running analyses, research teams submitted reports to be evaluated by others in a process of peer review. The peer reviewers rated the extent to which they believed that the research teams used sound statistical approaches, and whether their analytical decisions were defensible.
The key result from this experiment was that researchers’ ideology did predict the conclusions they drew from the dataset, and also led to research teams receiving worse peer review scores. In the words of the authors (emphasis added):
The data suggest that the team’s pre-existing preference toward immigration restrictions plays a role in producing some of the observed differences. The estimated effect of immigration on social cohesion is more positive if the researchers are pro-immigration and more negative if the researchers are anti-immigration.
What Borjas and Breznau observed is that the researchers with ideological views on immigration chose analytical methods which were more likely to yield their preferred conclusions. Their suggestion is that scientists may “play around” with the data in order to derive the results that they desire. Scientists with strong views on immigration (either pro- or anti-) were not acting as neutral “umpires” calling metaphorical “balls and strikes” or merely “following the data” toward its rightful conclusions. They were instead more like the players themselves, trying to use data to score points in a political battle against their opponents.
But what about “moderate” research teams? They reported a result that was not statistically distinguishable from zero, so basically, a null effect. Not only did the moderate research teams get a meaningfully different result from either the pro- or anti-immigrant researchers, their research reports also received better peer review scores, meaning that their peer reviewers believed that their statistical analyses were more appropriate and their analytical decisions were more defensible.
It’s important to note that we don’t actually have an objectively “correct” answer to the core research question at hand. Borjas and Breznau never declared in their paper that the moderate researchers were more likely to arrive at the proper conclusion regarding the effects of immigration, nor that those with stronger viewpoints came to incorrect conclusions. They only argue that moderates did a better job in terms of how their work was reviewed by their peers, and that both the pro- and anti-immigrant researchers were at odds with each other.
So, is this evidence that the best science is produced by dispassionate, enlightened centrists? Perhaps, although even moderate individuals can be bad at overcoming their own biases. Although Borjas and Breznau didn’t say this in their paper, I think this study provides substantial evidence for the value of viewpoint diversity through adversarial collaboration in research, especially those on topics with real world implications. When scholars pursue knowledge on important and divisive issues — such as the effects of immigration (or abortion, military policy, or conflicts in the Middle East) — an ideal research team could comprise those with a variety of different viewpoints which could challenge each other’s biases.
Borjas and Breznau’s ambitious study provides further evidence for the importance of viewpoint diversity in academic research and scholarship — and how ideological conformity of research teams can effectively skew research results.
 | A guest post by
|
The stories we tell ourselves about the world shape how we interpret data. This study has shown that we will use data to re-create – and scientifically certify – the stories we’re already telling ourselves about what is true, whether we are on the left or the right. We are storytelling animals, and we are wired this way. The challenge is to create systems that reliably enable us to correct for bias – not to impose a different bias, but to actually get us to something like dispassionate analysis and accurate assessment. That's the only way we can begin to approach what is real, and what is actually true.
How comforting when science affirms what we intuitively feel is so obvious. Now the hard work begins; researchers may be reluctant to acknowledge bias (aren’t we all?). How can we most effectively induce receptivity to these findings, and foster efforts to ensure viewpoint diversity on research teams? Reaching individuals? Reaching institutions?