RedacTek’s tool alerts users to PubPeer discussions, and indicates when a study, or the papers that it cites, has been retracted.Credit: deepblue4you/Getty
A free online tool released earlier this month alerts researchers when a paper cites studies that are mentioned on the website PubPeer, a forum scientists often use to raise integrity concerns surrounding published papers.
Studies are usually flagged on PubPeer when readers have suspicions, for example about image manipulation, plagiarism, data fabrication or artificial intelligence (AI)-generated text. PubPeer already offers its own browser plug-in that alerts users when a study that they are reading has been posted on the site. The new tool, a plug-in released on 13 April by RedacTek, based in Oakland, California, goes further — it searches through reference lists for papers that have been flagged. The software pulls information from many sources, including PubPeer’s database; data from the digital-infrastructure organization Crossref, which assigns digital object identifiers to articles; and OpenAlex, a free index of hundreds of millions of scientific documents.
It’s important to track mentions of referenced articles on PubPeer, says Jodi Schneider, an information scientist at the University of Illinois Urbana-Champaign, who has tried out the RedacTek plug-in. “Not every single reference that’s in the bibliography matters, but some of them do,” she adds. “When you see a large number of problems in somebody’s bibliography, that just calls everything into question.”
The aim of the tool is to flag potential problems with studies to researchers early on, to reduce the circulation of poor-quality science, says RedacTek founder Rick Meyler, based in Emeryville, California. Future versions might also use AI to automatically clarify whether the PubPeer comments on a paper are positive or negative, he adds.
Third-generation retractions
As well as flagging PubPeer discussions, the plug-in indicates when a study, or the papers that it cites, has been retracted. There are existing tools that alert academics about retracted citations; some can do this during the writing process, so that researchers are aware of the publication status of studies when constructing bibliographies. But with the new tool, users can opt in to receive notifications about further ‘generations’ of retractions — alerts cover not only the study that they are reading, but also the papers it cites, articles cited by those references and even papers cited by the secondary references.
The software also calculates a ‘retraction association value’ for studies, a metric that measures the extent to which the paper is associated with science that has been withdrawn from the literature. As well as informing individual researchers, the plug-in could help scholarly publishers to keep tabs on their own journals, Meyler says, because it allows users to filter by publication.
In its ‘paper scorecard’, the tool also flags any papers in the three generations of referenced studies in which more than 25% of papers in the bibliography are self-citations — references by authors to their previous works.
Future versions could highlight whether papers cited retracted studies before or after the retraction was issued, notes Meyler, or whether mentions of such studies acknowledge the retraction. That would be useful, says Schneider, who co-authored a 2020 analysis that found that as little as 4% of citations to retracted studies note that the referenced paper has been retracted1.
Meyler says that RedacTek is currently in talks with scholarly-services firm Cabell’s International in Beaumont, Texas, which maintains pay-to-view lists of suspected predatory journals, which publish articles without proper quality checks for issues such as plagiarism but still collect authors’ fees. The plan is to use these lists to improve the tool so that it can also automatically flag any cited papers that are published in such journals.
Policymakers often work behind closed doors — but the documents they produce offer clues about the research that influences them.Credit: Stefan Rousseau/Getty
When David Autor co-wrote a paper on how computerization affects job skill demands more than 20 years ago, a journal took 18 months to consider it — only to reject it after review. He went on to submit it to The Quarterly Journal of Economics, which eventually published the work1 in November 2003.
Autor’s paper is now the third most cited in policy documents worldwide, according to an analysis of data provided exclusively to Nature. It has accumulated around 1,100 citations in policy documents, show figures from the London-based firm Overton (see ‘The most-cited papers in policy’), which maintains a database of more than 12 million policy documents, think-tank papers, white papers and guidelines.
“I thought it was destined to be quite an obscure paper,” recalls Autor, a public-policy scholar and economist at the Massachusetts Institute of Technology in Cambridge. “I’m excited that a lot of people are citing it.”
The top ten most cited papers in policy documents are dominated by economics research. When economics studies are excluded, a 1997 Nature paper2 about Earth’s ecosystem services and natural capital is second on the list, with more than 900 policy citations. The paper has also garnered more than 32,000 references from other studies, according to Google Scholar. Other highly cited non-economics studies include works on planetary boundaries, sustainable foods and the future of employment (see ‘Most-cited papers — excluding economics research’).
These lists provide insight into the types of research that politicians pay attention to, but policy citations don’t necessarily imply impact or influence, and Overton’s database has a bias towards documents published in English.
Interdisciplinary impact
Overton usually charges a licence fee to access its citation data. But last year, the firm worked with the London-based publisher Sage to release a free web-based tool that allows any researcher to find out how many times policy documents have cited their papers or mention their names. Overton and Sage said they created the tool, called Sage Policy Profiles, to help researchers to demonstrate the impact or influence their work might be having on policy. This can be useful for researchers during promotion or tenure interviews and in grant applications.
Autor thinks his study stands out because his paper was different from what other economists were writing at the time. It suggested that ‘middle-skill’ work, typically done in offices or factories by people who haven’t attended university, was going to be largely automated, leaving workers with either highly skilled jobs or manual work. “It has stood the test of time,” he says, “and it got people to focus on what I think is the right problem.” That topic is just as relevant today, Autor says, especially with the rise of artificial intelligence.
Walter Willett, an epidemiologist and food scientist at the Harvard T.H. Chan School of Public Health in Boston, Massachusetts, thinks that interdisciplinary teams are most likely to gain a lot of policy citations. He co-authored a paper on the list of most cited non-economics studies: a 2019 work3 that was part of a Lancet commission to investigate how to feed the global population a healthy and environmentally sustainable diet by 2050 and has accumulated more than 600 policy citations.
“I think it had an impact because it was clearly a multidisciplinary effort,” says Willett. The work was co-authored by 37 scientists from 17 countries. The team included researchers from disciplines including food science, health metrics, climate change, ecology and evolution and bioethics. “None of us could have done this on our own. It really did require working with people outside our fields.”
Sverker Sörlin, an environmental historian at the KTH Royal Institute of Technology in Stockholm, agrees that papers with a diverse set of authors often attract more policy citations. “It’s the combined effect that is often the key to getting more influence,” he says.
Has your research influenced policy? Use this free tool to check
Sörlin co-authored two papers in the list of top ten non-economics papers. One of those is a 2015 Science paper4 on planetary boundaries — a concept defining the environmental limits in which humanity can develop and thrive — which has attracted more than 750 policy citations. Sörlin thinks one reason it has been popular is that it’s a sequel to a 2009 Nature paper5 he co-authored on the same topic, which has been cited by policy documents 575 times.
Although policy citations don’t necessarily imply influence, Willett has seen evidence that his paper is prompting changes in policy. He points to Denmark as an example, noting that the nation is reformatting its dietary guidelines in line with the study’s recommendations. “I certainly can’t say that this document is the only thing that’s changing their guidelines,” he says. But “this gave it the support and credibility that allowed them to go forward”.
Broad brush
Peter Gluckman, who was the chief science adviser to the prime minister of New Zealand between 2009 and 2018, is not surprised by the lists. He expects policymakers to refer to broad-brush papers rather than those reporting on incremental advances in a field.
Gluckman, a paediatrician and biomedical scientist at the University of Auckland in New Zealand, notes that it’s important to consider the context in which papers are being cited, because studies reporting controversial findings sometimes attract many citations. He also warns that the list is probably not comprehensive: many policy papers are not easily accessible to tools such as Overton, which uses text mining to compile data, and so will not be included in the database.
The top 100 papers
“The thing that worries me most is the age of the papers that are involved,” Gluckman says. “Does that tell us something about just the way the analysis is done or that relatively few papers get heavily used in policymaking?”
Gluckman says it’s strange that some recent work on climate change, food security, social cohesion and similar areas hasn’t made it to the non-economics list. “Maybe it’s just because they’re not being referred to,” he says, or perhaps that work is cited, in turn, in the broad-scope papers that are most heavily referenced in policy documents.
As for Sage Policy Profiles, Gluckman says it’s always useful to get an idea of which studies are attracting attention from policymakers, but he notes that studies often take years to influence policy. “Yet the average academic is trying to make a claim here and now that their current work is having an impact,” he adds. “So there’s a disconnect there.”
Willett thinks policy citations are probably more important than scholarly citations in other papers. “In the end, we don’t want this to just sit on an academic shelf.”
It was in just the second article of more than 1,000 that Otto Kalliokoski was screening that he spotted what he calls a “Photoshop masterpiece”.
The paper showed images from western blots — a technique used to analyse protein composition — for two samples. But Kalliokoski, an animal behaviourist at the University of Copenhagen, found that the images were identical down to the pixel, which he says is clearly not supposed to happen.
Image manipulation in scientific studies is a known and widespread problem. All the same, Kalliokoski and his colleagues were startled to come across more than 100 studies with questionable images while compiling a systematic review about a widely used test of laboratory rats’ moods. After publishing the review1 in January, the researchers released a preprint2 documenting the troubling studies that they uncovered and how these affected the results of their review. The preprint, posted on bioRxiv in February, has not yet been peer reviewed.
Their work “clearly highlights [that falsified images] are impacting our consolidated knowledge base”, says Alexandra Bannach-Brown, a systematic-review methodologist at the Berlin Institute of Health who was not involved with either the review or the preprint. Systematic reviews, which summarize and interpret the literature on a particular topic, are a key component of that base. With an explosion of scientific literature, “it’s impossible for a single person to keep up with reading every new paper that comes out in their field”, Bannach-Brown says. And that means that upholding the quality of systematic reviews is ever more important.
Pile-up of problems
Kalliokoski’s systematic review examined the reliability of a test designed to assess reward-seeking in rats under stress. A reduced interest in a reward is assumed to be a proxy symptom of depression, and the test is widely used during the development of antidepressant drugs. The team identified an initial pool of 1,035 eligible papers; 588 contained images.
By the time he’d skimmed five papers, Kalliokoski had already found a second one with troubling images. Not sure what to do, he bookmarked the suspicious studies and went ahead with collating papers for the review. As the questionable papers kept piling up, he and his colleagues decided to deploy Imagetwin, an AI-based software tool that flags problems such as duplicated images and ones that have been stretched or rotated. Either Imagetwin or the authors’ visual scrutiny flagged 112 — almost 20% — of the 588 image-containing papers.
“That is actually a lot,” says Elizabeth Bik, a microbiologist in San Francisco, California, who has investigated image-related misconduct and is now an independent scientific-integrity consultant. Whether image manipulation is the result of honest error or an intention to mislead, “it could undermine the findings of a study”, she says.
Small but detectable effect
For their final analysis, the authors examined all the papers that met their criteria for inclusion in their review. This batch, consisting of 132 studies, included 10 of the 112 that the team had flagged as having potentially doctored images.
Journals adopt AI to spot duplicated images in manuscripts
Analysis of these 10 studies alone assessed the test as 50% more effective at identifying depression-related symptoms than did a calculation based on the 122 studies without questionable images. These suspicious studies “do actually skew the results”, Kalliokoski says — although “not massively”, because overall variations in the data set mask the contribution from this small subset.
Examples from this study “cover pretty much all types of image problems”, Bik says, ranging from simple duplication to images that showed evidence of deliberate alteration. Using a scale that Bik developed to categorize the degree of image manipulation, the researchers found that most of the problematic images showed signs of tampering.
The researchers published their review in January in Translational Psychiatry without telling the journal that it was based in part on papers that included suspicious images. The journal’s publisher, Springer Nature, told Nature that it is investigating. (The Nature news team is editorially independent of its publisher, Springer Nature).
When they published their preprint the following month, the researchers included details of all the papers with suspicious images. They also flagged each study on Pubpeer, a website where scientists comment anonymously on papers. “My first allegiance is towards the [scientific] community,” Kalliokoski says, adding that putting the data out is the first step.
Bring reviews to life
The process of challenging a study’s integrity, giving its authors a chance to respond and seeking retraction for fraudulent studies can take years. One way to clear these muddied waters, says Bannach-Brown, is to publish ‘living’ systematic reviews, which are designed to be updated whenever papers get retracted or new research is added. She has helped to develop one such method of creating living reviews, called Systematic Online Living Evidence Summaries.
Systematic-review writers are also keen to see publishers integrate standardized ways to screen out dubious studies — rather than waiting until a study gets retracted.
Authors, publishers and editorial boards need to work together, Bannach-Brown says, to “catch some of these questionable research practices before they even make it to publication.”
Women and early-career researchers: Nature wants to publish your research.Credit: Getty
Researchers submitting original research to Nature over the past year will have noticed an extra question, asking them to self-report their gender. Today, as part of our commitment to helping to make science more equitable, we are publishing in this editorial a preliminary analysis of the resulting data, from almost 5,000 papers submitted to this journal over a five-month period. As well as showing the gender split in submissions, we also reveal, for the first time, possible interactions between the gender of the corresponding author and a paper’s chance of publication.
The data make for sobering reading. One stark finding is how few women are submitting research to Nature as corresponding authors. Corresponding authors are the researchers who take responsibility for a manuscript during the publication process. In many fields, this role is undertaken by some of the most experienced members of the team.
The giant plan to track diversity in research journals
During the period analysed, some 10% of corresponding authors preferred not to disclose their gender. Of the remainder, just 17% identified as women — barely an increase on the 16% we found in 2018, albeit using a less precise methodology. By comparison, women made up 31.7% of all researchers globally in 2021, according to figures from the United Nations science, education and cultural organization UNESCO (see go.nature.com/3wgdasb).
Large geographical differences were also laid bare. Women made up just 4% of corresponding authors of known gender from Japanese institutions. Of researchers from the two countries submitting the most papers, China and the United States, women made up 11% and 22%, respectively. These figures reflect the fact that women’s representation in research drops at the most senior levels. They also mirror available data from other journals1, although it is hard to find direct comparisons for a multidisciplinary journal such as Nature.
At Cell, which has a life-sciences focus, women submitted 17% of manuscripts between 2017 and 2021, according to an analysis of almost 13,000 submissions2. The most recent data on gender from the American Association for the Advancement of Science (AAAS), which publishes the six journals in the Science family, is collected and reported differently. Some 27% of their authors of primary and commissioned content, and their reviewers, are women, according to the AAAS Inclusive Excellence Report (see go.nature.com/3t6yyr8). Nonetheless, all of these figures are just too low.
Another area of concern is acceptance rates. Of the submissions included in the current Nature analysis, those with women as the corresponding author were accepted for publication at a slightly lower rate than were those authored by men. Some 8% of women’s papers were accepted (58 out of 726 submissions) compared with 9% of men’s papers (320 out of 3,522 submissions). The acceptance rate for people self-reporting as non-binary or gender diverse seemed to be lower, at 3%, although this is a preliminary figure and we have reason to suspect that the real figure could be higher, as described below. Once we have a larger sample, we plan to test whether the differences are statistically significant.
Sources of imbalance
So, at what stage in the publishing process is this imbalance introduced? Men and women seem to be treated equally when papers are selected for review. The journal’s editors — a group containing slightly more women than men — were just as likely to send papers out for peer review for women corresponding authors as they were for men. For both groups, 17% of submitted papers went for peer review.
Our efforts to diversify Nature’s journalism are progressing, but work remains
A difference arose after that. Of those papers sent for review, 46% of papers with women as corresponding authors were accepted for publication (58 of 125) compared with 55% (320 of 586) of papers authored by men. The acceptance rate for non-binary and gender-diverse authors was higher at 67%. However, this is from a total of only three reviewed papers, a figure that is too small to be meaningful.
This difference in acceptance rates during review tallies with the findings of a much larger 2018 study of 25 Nature-family journals, which used a name-matching algorithm, rather than self-reported data3. Looking at 17,167 papers sent for review over a 2-year period, the authors found a smaller but significant difference in acceptance rates, with 43% for papers with a woman as corresponding author, compared with 45% for a man. However, they were unable to say whether the difference was attributable to reviewer bias or variations in manuscript quality.
Peering into peer review
How much bias exists in the peer-review process is difficult to study and has long been the subject of debate. A 2021 study in Science Advances that looked at 1.7 million authors across 145 journals between 2010 and 2016 found that, overall, the peer-review and editorial processes did not penalize manuscripts by women4. But that study analysed journals with lower citation rates than Nature, and its results contrast with those of previous work5, which found gender-based skews.
Moreover, other studies have shown that people rate men’s competence more highly than women’s when assessing identical job applications6; that there is a gender bias against women in citations; and that women are given less credit for their work than are men7. Taken together, this means we cannot assume that peer review is a gender-blind process. Most papers in our current study were not anonymized. We did not share how the authors self-reported, but editors or reviewers might have inferred gender from a corresponding author’s name. Nature has offered double-anonymized peer review for both authors and reviewers since 2015. Too few take it up for us to have been able to examine its impact in this analysis, but the larger study in 2018 looked at this in detail3.
Data limitations
There are important limitations to Nature’s data: we must emphasize again that they are preliminary. Moreover, they provide the gender of only one corresponding author per paper, not the gender distribution of a paper’s full author list. Furthermore, they don’t describe any other differences between authors.
There are also aspects of the data that need to be investigated further. For example, we need to look into the possibility that the option of reporting as non-binary or gender diverse is being misinterpreted by some authors with English as a second language. We think that ironing out such misunderstandings could result in a higher acceptance rate for non-binary authors.
Nature’s under-representation of women
Most importantly, these data give no insight into author experiences in relation to race, ethnicity and socio-economic status. Although men often have advantages compared with women, other protected characteristics also have a significant impact on scientists’ careers. Nature is participating in an effort by a raft of journal publishers to document and reduce bias in scholarly publishing by tracking a range of characteristics. This is a work in progress and sits alongside Springer Nature’s wider commitment to tackling inequity in research publishing.
So what can Nature do to ensure that more women and minority-gender scientists find a home for their research in our pages?
First, we want to encourage a more diverse pool of corresponding authors to submit. The fact that only 17% of submissions come from corresponding authors who identify as women might reflect existing imbalances in science (for example, it roughly tracks with the 18% of professor-level scientists in the European Union who are women, as reported by the European Commission8).
But there remains much scope for improvement. We know that the workplace climate in academia can push women out or see them overlooked for senior positions9. A 2023 study published in eLife found that women tend to be more self-critical of their own work than men are and that they are more frequently advised not to submit to the most prestigious journals10.
Second, just as prestigious universities should not simply lament their low application numbers from under-represented groups, we should not sit back and wait for change to come to us. To this end, our editors will actively seek out authors from these communities when at conferences and on laboratory visits. We will be more proactive in reaching out to women and early-career researchers to make sure they know that Nature wants to publish their research. We encourage authors with excellent research, at any level of seniority and at any institution, to submit their manuscripts.
Third, in an effort to make peer review fairer, Nature’s editors have been actively working to recruit a more diverse group of referees; 2017 data found that women made up just 16% of our reviewers. We need to double down on our efforts to improve this situation and update readers on our progress. In the future, we also plan to analyse whether corresponding authors’ gender affects the number of review cycles they face, and whether there are differences in relation to gender according to discipline and prestige of their affiliated institution. We need to improve our understanding of the sources of inequity before we can work on ways to address them. Nature’s editors will also strive to minimize our own biases through ongoing unconscious-bias training.
Last but not least, we will keep publishing our data on authorship and peer review, alongside complementary statistics on the gender of contributors to articles outside original research. Although today’s data present just a snapshot, Nature remains committed to tracking the gender of authors, to regularly updating the community on our efforts, and to exploring ways to make the publication process more equitable.
