I’m a paediatrician based in South Africa. Last year, my colleagues and I were invited to submit an editorial to a medical journal. We felt that the article, about medicine in resource-limited settings, should be published open access (OA) because it contains information that health-care workers and researchers in sub-Saharan Africa need access to. The problem is that the OA fee for that journal is US$1,000, which is more than most doctors earn per month in, say, Uganda. Now, we’re not sure whether we can move forward with the editorial. Are there any resources or funds available to authors in low-income countries to cover OA fees? — A paediatrician on a budget
The advice
Nature reached out to three researchers for tips on article processing charges (APCs). These fees can range from several hundred to thousands of dollars, and are requested by journals in return for making their articles OA — free for everyone to read.
According to a study published in 2023, the average fee for publishing an OA article is close to US$1,4001. OA fees can create significant barriers to publishing and sharing one’s work, especially for researchers based in low- and middle-income countries (LMICs). For instance, among the top 40 journals in ecology, the average OA fee was $3,150, according to a 2021 study2. The authors described it as a hardship for African scholars, who typically do not receive grant funding and whose monthly salaries at the time of the study ranged from $365 to $2,300.
I’m worried I’ve been contacted by a predatory publisher — how do I find out?
Most scientific journals are transparent about their publishing fees, which are typically included in the author guidelines or stated on their website. “If a journal suddenly asks for payment” having not mentioned such a requirement initially, says Kit Magellan, an independent behavioural ecologist based in Siem Reap, Cambodia, “it is likely a predatory journal — run away!” Predatory journals present themselves as legitimate publications, but use the OA publishing model to dupe authors into paying them fees.
If the APCs for a legitimate journal are too steep for you to afford, there are multiple ways to tackle the cost. “The first thing to do is check in with your co-authors to see if they have any funds available,” says Magellan, because scientists might be eligible to have APCs covered by their grants or by funding organizations. If not, she recommends asking your institution if it provides researchers with financial support to publish OA.
Institutional support for APCs is highly variable, ranging from offering no funding to covering the full cost. “Processing fees can get prohibitively expensive,” says Thulani Makhalanyane, a microbial ecologist at Stellenbosch University in South Africa. “My institution will reimburse half the cost, but I still have to think about where the other half of that expense will come from.”
Both Magellan and Makhalanyane note that scientific societies often offer their members grants or financial support — separate from funding for day-to-day laboratory work — to pay for APCs. For example, in December 2023, the American Physical Society announced a partnership with the non-profit organization Research4Life to cover APCs for paper submissions from scientists in 100 LMICs. Since 2002, Research4Life has helped researchers at more than 11,500 institutions in 125 LMICs access peer-reviewed papers from over 200,000 journals and books. Other governmental partnerships and programmes, such as the European Commission’s Open Research Europe and the library partnership SCOAP, pay OA fees directly to publishers, to avoid publishers passing those costs on to authors.
Another option is to contact the journal you want to publish with, to see whether it can offer assistance or flexibility with APCs. When approaching a journal editor, Makhalanyane recommends being upfront and open about your budget. “Tell the editor you’d like to submit your paper to their journal because you think it’s a good fit, but that you can’t afford the fee,” he says. As a journal editor himself, Makhalanyane receives several OA fee waivers from the publisher each year that he can offer to researchers. “Most of these vouchers are never taken,” he adds.
Springer Nature was asked whether it provides assistance with APCs for researchers in LMICs. (Springer Nature publishes Nature, but the magazine’s careers team is editorially independent of its publisher.) “Enabling open-access equity remains a key part of our focus,” said a spokesperson, who made reference to the publisher’s waiver policy for fully OA journals, Transformative Agreements and partnerships with organizations such as Research4Life.
The spokesperson also noted that the company has an initiative for Nature and the Nature research journals that means that accepted papers by authors from more than 70 LMICs are published at no cost to them. Finally, a tiered-pricing pilot adjusts the APC on the basis of the lead author’s country of residence, the spokesperson said.
Other researchers who want to pursue the OA route wait until their paper is close to publication before approaching an editor about the cost. “I don’t consider budget issues when I submit papers,” says Noam Shomron, a genomicist and computational biologist at Tel Aviv University in Israel. The peer-review and publication process can span months to a year or longer, and researchers’ budgets can fluctuate drastically over that period, he explains. “If I’m running out of funding at the time, I just tell the publication I don’t have the money. Very often they give me a 10% or 20% discount, which is nice.” Even if a discount isn’t possible, Shomron says that journals might defer payment for a year or two.
Magellan, who also has experience as a journal editor, emphasizes that vouchers and fee waivers are meant for exceptional circumstances, in which the author lacks access to funding to cover APCs. For those who are paying the standard charges, she is keen to see more-flexible payment plans from publishers. “It would be good for journals to allow authors to pay in instalments so the APC vouchers can remain available for the people who really need it,” she says.
Collection: Careers toolkit
“The recent proliferation of online fee-paying journals seems to sometimes result in the perception that you have to pay to publish,” says Magellan. But researchers who can’t afford OA fees can still publish their work for free in many scientific journals, with the caveat that their articles might be hidden behind a paywall. “You can still share your article with colleagues in the field, use it in presentations and cite it; it just can’t be freely accessed,” she says. However, researchers at eligible institutions in LMICs can access paywalled papers through resources such as Hinari, a branch of Research4Life that provides access to thousands of medical and health journals.
“Submissions that come from the parts of the world where researchers can’t afford to publish are usually such a minor fraction of the papers that end up being published,” says Makhalanyane. “I would encourage people who want to publish and genuinely cannot afford the APCs to ask for vouchers. The fees shouldn’t stop you from showcasing your science in the best journals you can.”
The ‘file-drawer problem’, where findings with null or negative results gather dust and are left unpublished, is well known in science. There has been an overriding perception that studies with positive or significant findings are more important, but this bias can have real-world implications, skewing perceptions of drug efficacies, for example.
Multiple efforts to get negative results published have been put forward or attempted, with some researchers saying that the incentive structures in academia, and the ‘publish or perish’ culture, need to be overturned in order to end this bias.
This is an audio version of our Feature: So you got a null result. Will anyone publish it?
Article retractions have been growing steadily over the past few decades, soaring to a record-breaking figure of nearly 14,000 last year, compared with less than 1,000 per year before 2009 (see go.nature.com/3azcxan and go.nature.com/3x9uxfn).
Retracting flawed research papers is part of a healthy scientific process. Not all retractions stem from misconduct — they can also occur when mistakes happen, such as when a research group realizes it can’t reproduce its results.
But regardless of how erroneous results found their way into a published paper, it is important that they are not propagated in the scientific literature. No one wants to base their reasoning on false premises. In the same way that many people wouldn’t accept a medical treatment bolstered by shaky clinical trials, the scientific community doesn’t want researchers, the public and, increasingly, artificial intelligence (AI) to rely on erroneous data or conclusions from retracted articles.
How to improve assessments of publication integrity
One aspect that is often overlooked is what happens to the papers that cite retracted research. For example, in June, a Nature paper1 on stem cells was retracted amid concerns about the reliability of the data shown — 22 years after its publication, having amassed nearly 5,000 citations. Of course, an article lists references for a variety of reasons, such as to provide context, to introduce related work or to explain the experimental protocol. A retraction doesn’t mean that all the papers that cited the retracted article are now unreliable, too — but some might be. At a minimum, researchers should be aware of any retractions among the studies that they have cited. This would enable them to assess potential negative effects on their own work, and to mention the relevant caveats clearly should they continue to cite the retracted paper in the future. But, as far as I know, no systematic process is in place for publishers to alert citing scholars when an article is retracted. There should be.
Beyond retractions, what is needed is a large-scale mechanism to stop errors from propagating in the scientific literature. The tools exist — now, practices need to change.
Shaking up the status quo
Publications and citations are important currency in academia. Yet dubious papers or citations can be difficult to distinguish from genuine ones. Combined with the fact that the editorial, peer-review and publishing processes are highly reliant on trust, this has led to many distortions.
Biomedical paper retractions have quadrupled in 20 years — why?
A researcher’s performance metrics — including the number of papers published, citations acquired and peer-review reports submitted — can all serve to build a reputation and visibility, leading to invitations to speak at conferences, review manuscripts, guest-edit special issues and join editorial boards. This can give more weight to job or promotion applications, be key to attracting funding and lead to more citations, all of which can build a high-profile career. Institutions generally seem happy to host scientists who publish a lot, are highly cited and attract funding.
Unscrupulous businesses known as paper mills have popped up that capitalize on this system. They produce manuscripts based on made-up, manipulated or plagiarised data, sell those fake manuscripts as well as authorship and citations, and engineer the peer-review process.
But reputable publishers are also complicit, when they churn out papers from high-profile researchers — including some who might have built visibility quickly through dubious or dishonest practices — and regularly use those individuals as reviewers and editors. The publishing industry benefits from large volumes of papers, including those that are not scientifically sound.
Illustration: Phil Wheeler
Tools for change
Researchers, publishers, institutions and funders must all act to uphold the integrity of the scientific record.
Scientists who discover a suspicious or problematic paper can flag it through the conventional route by contacting the editorial team of the journal in which it appeared. But it can be difficult to find out how to raise concerns, and who with. Furthermore, this process is typically not anonymous and, depending on the power dynamics at play, some researchers might be unwilling or unable to enter these conversations.
And journals are notoriously slow. The process requires journal staff to mediate a conversation between all parties — a discussion that authors of the criticized paper are typically reluctant to engage in and which sometimes involves extra data and post-publication reviewers. Most investigations can take months or years before the outcome is made public.
Retractions are increasing, but not enough
Other avenues exist to question a study after publication, such as commenting on the PubPeer platform, where a growing number of papers are being reported. As of 20 August, 191,463 articles have received comments on PubPeer — nearly all of which were critical (see https://pubpeer.com/recent). But publishers typically don’t monitor these, and the authors of a criticized paper aren’t obliged to respond. It is common for post-publication comments, including those from eminent researchers in the field, to raise potentially important issues that go unacknowledged by the authors and the publishing journal.
In February 2021, I launched the Problematic Paper Screener (PPS; see go.nature.com/473vsgb). This software originally flagged randomly generated text in published papers. It now tracks a variety of issues to alert the scientific community to potential errors.
I devised a tool for the PPS to comb the literature for nonsensical ‘tortured phrases’ that are proliferating in the scientific literature (see ‘Lost in translation’). Each tortured phrase first needs to be spotted by a human reader, then added as a ‘fingerprint’ to the tool that regularly screens the literature using the 130 million scientific documents indexed by the data platform Dimensions. So far, 5,800 fingerprints have been collated. Humans are involved in a third step to check for false positives. (Dimensions is in the portfolio of Digital Science, which is part of Holtzbrinck, the majority shareholder in Nature’s publisher, Springer Nature.)
Lost in translation
Nonsensical phrases in scientific papers can sound alarm bells.
Co-authors, editors, referees and typesetters should keep an eye out for unnatural phrases in articles. They can expose text that has been generated by artificial intelligence or by an elaborate form of copy-and-paste that uses a translation tool to make phrases unrecognizable to plagiarism-detection tools.
Yet, even published articles that are riddled with dozens of these ‘tortured phrases’ are slow to be investigated, corrected or retracted. As of 20 August, the Problematic Paper Screener that I launched in 2021 had flagged more than 16,000 papers citing 5 or more such tortured phrases — only 18% of which have been retracted (see go.nature.com/3mbey8m).
Since my colleagues and I reported that tortured phrases had marred the literature2, publishers — and not just those deemed predatory — have been retracting hundreds of articles as a result. Springer Nature alone, for example, has retracted more than 300 articles featuring nonsensical text (see go.nature.com/3ytezsw).
And I am increasingly concerned by the number of articles that cite retracted studies — even after their retraction3.
To facilitate ongoing checks and continuous clean-up of the literature, I have devised two other tools for the PPS. One is the Annulled Detector, which keeps track of papers that have been retracted, withdrawn or removed — these are the various labels used by publishers to flag that a study is no longer valid. The detector harvests data from individual publishers, the Crossref database (which includes the Retraction Watch database) and the biomedical database PubMed to track the global landscape of retractions, withdrawals and removals. Some 62,000 such ‘annulled’ articles have now been cited more than 836,000 times overall (see go.nature.com/4dp5d7f).
The other is the Feet of Clay Detector, which serves to quickly spot those articles that cite annulled papers in their reference lists (see go.nature.com/3ysnj8f). I have added online PubPeer comments to more than 1,700 such articles to prompt readers to assess the reliability of the references.
Prevent and cure
There are simple steps, using widely available tools, that would significantly bolster the reliability of the scientific literature.
Authors should check for any post-publication criticism or retraction when using a study, and certainly before including a reference in a manuscript draft.
Retractions are part of science, but misconduct isn’t — lessons from a superconductivity lab
Two PubPeer extensions are instrumental. One plug-in automatically flags any paper that has received comments on PubPeer, which can include corrections and retractions, when readers skim through journal websites. The other works in the reference manager Zotero to identify the same articles in a user’s digital library. For local copies of downloaded PDFs, the publishing industry uses Crossmark: readers can click on the Crossmark button to check the status of the article on the landing page at the publisher’s website.
Tools exist to check reference lists, such as RetractoBot, which alerts scholars when papers they have cited are retracted. And the Feet of Clay Detector can be used, for free, to check whether the reference list of a published article has any red flags. It can run checks using just the title of an article or entire publishers’ portfolios, making it easy for individual researchers and journals to check the literature that is of interest to them.
Science would also benefit from more active post-publication scrutiny by an increased number of readers reporting concerns on PubPeer and to publishers. Conversely, the authors of a criticized paper should engage in good faith in discussions with their peers and/or the relevant journal, and work towards a swift resolution.
Readers — especially reviewers — should be aware of red flags, such as tortured phrases and the possible machine-generation of texts by AI tools (including ChatGPT). Suspicious phrases that look like they might be machine-generated can be checked using tools such as the PPS Tortured Phrases Detector2.
Journals should also contact researchers who reviewed an article that went on to be retracted on technical grounds — for their own information and, if the technical issue is in their area of expertise, to prompt them to be more cautious in future.
Publishers are best placed to make impactful changes to their practices and processes. They should routinely run submitted manuscripts through tools that check for plagiarism, doctored images, tortured phrases, retracted or questionable references, non-existent references erroneously generated (‘hallucinated’) by AI tools and citation plantations (large shares of references that benefit certain individuals).
Pay researchers to spot errors in published papers
These checks and balances are being integrated into the STM Integrity Hub currently in development by STM, the association for the academic publishing industry that serves the editorial boards of subscribed publishers. The software aims to detect duplicate submissions across publishers, and to flag to editors any suspicious signals such as tortured phrases, comments on PubPeer or retracted references.
As well as preventive measures, publishers should speed up and scale up their curative efforts when it comes to investigations, corrections and retractions. They should take firm responsibility for the articles they have published, and conduct regular checks so that retractions in their portfolios do not go unnoticed.
To help independent tools such as the Feet of Clay Detector to harvest data on the current status of articles in their journals, all publishers should publicly release the reference metadata of their entire catalogue.
They should also “unmistakably” identify retracted articles, as stated in the guidelines from the Committee on Publication Ethics (COPE; see go.nature.com/4dh7fdg). Most publishers edit the article PDF file to include a ‘Retracted’ watermarked banner. But any copy downloaded before a retraction took place won’t include this crucial caveat.
‘Expressions of concern’ from publishers should be more widespread. Such notes serve to alert readers that the reliability of a paper’s conclusions has been called into question.
And when a study is retracted, it should trigger a cascade reaction and, in some cases, a ‘cascade retraction’. This would mean that all the papers that cite that study should be reassessed, and if their conclusions hinged on now-retracted results, they should be corrected or retracted as appropriate.
Overall, to facilitate all these steps, publishers must update their practices and attribute more resources to both editorial and research-integrity teams.
Hold all parties accountable
Finally, another aspect that could curb the propagation of errors in the literature is accountability. At the moment, there are few consequences for failing to correct or retract erroneous papers, and little reward for flagging them — a time-consuming endeavour. Universities and funders must give priority to good, solid science over indirect metrics such as numbers of publications and impact factors of the journals they appeared in. Contributions to correcting the scientific record should be viewed more positively, perhaps in terms of community service.
As publishers retract ever more articles, I nudge them to transfer to charity the article processing charges they received on publication. For instance, IOP Publishing, owned by the Institute of Physics in London, was among the first publishers to retract articles on the basis of tortured phrases. It donates revenues from its retracted articles to Research4Life, an organization that provides institutions in low-and middle-income countries with online access to the academic literature.
A combined preventive and curative effort from all involved is key to sustaining the reliability of the scientific literature — a crucial undertaking for science and for public trust.
In January, a review paper1 about ways to detect human illnesses by examining the eye appeared in a conference proceedings published by the Institute of Electrical and Electronics Engineers (IEEE) in New York City. But neither its authors nor its editors noticed that 60% of the paper it cited had already been retracted.
The case is one of the most extreme spotted by a giant project to find papers whose results might be in question because they cite retracted or problematic research. The project’s creator, computer scientist Guillaume Cabanac at the University of Toulouse in France, shared his data with Nature’s news team, which analysed it to find the papers that most heavily cite retracted work yet haven’t themselves been withdrawn (see ‘Retracted references’).
Chain retraction: how to stop bad science propagating through the literature
“We are not accusing anybody of doing something wrong. We are just observing that in some bibliographies, the references have been retracted or withdrawn, meaning that the paper may be unreliable,” Cabanac says. He calls his tool a Feet of Clay Detector, referring to an analogy, originally from the Bible, about statues or edifices that collapse because of their weak clay foundations.
The IEEE paper is the second-highest on the list assembled by Nature, with 18 of the 30 studies it cites withdrawn. Its authors didn’t respond to requests for comment, but IEEE integrity director Luigi Longobardi says that the publisher didn’t know about the issue until Nature asked, and that it is investigating.
Cabanac, a research-integrity sleuth, has already created software to flag thousands of problematic papers in the literature for issues such as computer-written text or disguised plagiarism. He hopes that his latest detector, which he has been developing over the past two years and describes this week in a Comment article in Nature, will provide another way to stop bad research propagating through the scientific literature — some of it fake work created by ‘papermill’ firms.
Further scrutiny
Cabanac lists the detector’s findings on his website, but elsewhere online — on the paper-review site PubPeer and on social media — he has explicitly flagged more than 1,700 papers that caught his eye because of their reliance on retracted work. Some authors have thanked Cabanac for alerting them to problems in their references. Others argue that it’s unfair to effectively cast aspersions on their work because of retractions made after publication that, they say, don’t affect their paper.
Scientific sleuths spot dishonest ChatGPT use in papers
Retracted references don’t definitively show that a paper is problematic, notes Tamara Welschot, part of the research-integrity team at Springer Nature in Dordrecht, the Netherlands, but they are a useful sign that a paper might benefit from further scrutiny. (Nature’s news team is independent of its publisher, Springer Nature.)
Some researchers argue that retraction of references in a narrative review — which describes the state of research in a field — doesn’t necessarily invalidate the original paper. But when studies assessed by a systematic review or meta-analysis are withdrawn, the results of that review should always be recalculated to keep the literature up to date, says epidemiologist Isabelle Boutron at Paris City University.
Picking up fraudsters
Some of the papers that cite high proportions of retracted work are authored by known academic fraudsters who have had many of their own papers retracted.
These include engineering researcher Ali Nazari, who was dismissed from Swinburne University of Technology in Melbourne, Australia, in 2019, after a university misconduct investigation into his activities. He previously worked at Islamic Azad University in Saveh, Iran, and his current whereabouts are unclear. After Nature told publishers about his extant papers2,3 topping Cabanac’s lists — including Elsevier and Fap-Unifesp, a non-profit foundation that supports the Federal University of São Paulo in Brazil — they said that they would look into the articles. One of the relevant journals was discontinued in 2013, Elsevier noted.
Cabanac’s detector also flags papers4 by Chen-Yuan Chen, a computer scientist who worked at the National Pingtung University of Education in Taiwan until 2014. He was behind a syndicate that faked peer review and boosted citations, which came to light in 2014 after an investigation by the publisher SAGE. Some of Chen’s papers that are still in the literature were published by Springer Nature, which says it hadn’t been aware of the issue but is now investigating. Neither Chen nor Nazari responded to Nature’s requests for comment.
Another flagged study5 is by Ahmad Salar Elahi, a physicist affiliated with the Islamic Azad University in Tehran who has already had dozens of his papers retracted, in many cases because of excessive self-citation and instances of faked peer review. In 2018, the website Retraction Watch (which also wrote about the Nazari and Chen cases) reported that according to Mahmoud Ghoranneviss, then-director of the Plasma Physics Research Centre where Elahi worked, Elahi was likely to be dismissed from the university. Now, Ghoranneviss — who has retired — says that Elahi was barred only from that centre and not the rest of the university. Elahi continues to publish papers, sometimes listing co-authors including Ghoranneviss, who says he wasn’t aware of this. Neither Elahi nor the university responded to Nature’s queries. The IEEE and Springer Nature, publishers of the journals that ran the Elahi papers, say they’re investigating.
Unhappy authors
Some authors are unhappy about Cabanac’s work. In May 2024, editors of the journal Clinical and Translational Oncology placed an expression of concern on a 2019 review paper6 about RNA and childhood cancers, warning that it might not be reliable because it cited “a number of articles that have been retracted”. The journal’s publishing editor, Ying Jia at Springer Nature in Washington DC, says the team was alerted by one of Cabanac’s posts on social media last year.
Computer scientist Guillaume Cabanac has flagged more than 1,700 papers that caught his eye because of their reliance on retracted work.Credit: Fred Scheiber/SIPA/Shutterstock
Cabanac’s analysis finds that just under 10% of the article’s 637 references have been retracted — almost all after the review was published. However, the paper’s corresponding author, María Sol Brassesco, a biologist at the University of São Paulo, says that removing these references doesn’t change the conclusions of the review, and that she has sent the journal an updated version, which it hasn’t published. Because the cited works were retracted after publication, the expression of concern “felt like we were being punished for something that we could not see ahead”, she says. Jia says that editors felt that adding the notice was the most appropriate action.
In other cases, authors disagree about what to do. Nature examined three papers7,8,9 in which between 5 and 16% of the references have now been retracted, all co-authored by Mohammad Taheri, a genetics PhD student at Friedrich Schiller University of Jena in Germany. He says that criticisms of his work on PubPeer “lack solid scientific basis”. Yet, in May, a co-author of two of those works, Marcel Dinger, dean of science at the University of Sydney in Australia, told Retraction Watch that he was reassessing review papers that cited retracted articles. He now says that his team has submitted corrections for the works, but Frontiers, which published one paper, says it hasn’t received the correspondence and will investigate. Elsevier — which published the other two papers — also says that it is examining the issue.
Catching problems early
Examples in which papers cite already-retracted work suggest that publishers could do a better job of screening manuscripts. For instance, 20 studies cited by a 2023 review paper10 about RNA and gynaecological cancers in Frontiers in Oncology had been retracted before the article was submitted. Review co-author Maryam Mahjoubin-Tehran, a pharmacist at Mashhad University of Medical Sciences in Iran, told Nature that her team didn’t know about the retractions, and does not plan to update or withdraw the paper. The publisher, Frontiers, says it is investigating.
Until recently, publishers have not flagged citations to retracted papers in submitted manuscripts. However, many publishers say they are aware of Cabanac’s tool and monitor issues he raises, and some are bringing in similar screening tools.
Last year, Wiley announced it was checking Retraction Watch’s database of retracted articles to flag issues in reference lists, and Elsevier says it is also rolling out a tool that assesses manuscripts for red flags such as self-citations and references to retracted work. Springer Nature is piloting an in-house tool to look for retracted papers in manuscript citations and Longobardi says the IEEE is considering including Feet of Clay or similar solutions in its workflow. A working group for the STM Integrity Hub — a collaboration between publishers — has also tested the Feet of Clay Detector and “found it useful”, says Welschot.
Medical trend
Medical reviews that cite studies in areas later shown to be affected by fraud are a recurring theme in Cabanac’s findings.
In theory, meta-analyses or systematic reviews should be withdrawn or corrected if work they have cited goes on to be retracted, according to a policy issued in 2021 by the Cochrane Collaboration, an international group known for its gold-standard reviews of medical treatments.
Boutron, who directs Cochrane France in Paris, is using Cabanac’s tool to identify systematic reviews that cite retracted work, and to assess the impact the retracted studies had on the overall results.
However, a 2022 study11 suggests that authors are often reluctant to update reviews, even when they are told the papers cite retracted work. Researchers e-mailed the authors of 88 systematic reviews that cited now-retracted studies in bone health by a Japanese fraudster, Yoshihiro Sato. Only 11 of the reviews were updated, the authors told Nature last year.
Retraction alerts
Authors aren’t routinely alerted if work cited in their past papers is withdrawn — although in recent years, paper-management tools for researchers such as Zotero and EndNote have incorporated Retraction Watch’s open database of retracted papers and have begun to flag papers that have been taken down. Cabanac thinks publishers might use tools like his to create similar alerts.
In 2016, researchers at the University of Oxford, UK, began developing a tool called RetractoBot, which automatically notifies authors by e-mail when a study that they have previously cited has been retracted. The software currently monitors 20,000 retracted papers and about 400,000 papers, published after 2000, that cite them. The team behind it is running a randomized trial to see whether papers flagged by RetractoBot are subsequently cited less than those not flagged by the tool, and will publish its results next year, says project lead Nicholas DeVito, a integrity researcher at Oxford.
The team has alerted more than 100,000 researchers so far. DeVito says that a minority of authors are annoyed about being contacted, but that others are grateful. “We are merely trying to provide a service to the community to reduce this practice from happening,” he says.
Studies that try to replicate the findings of published research are hard to come by: it can be difficult to find funders to support them and journals to publish them. And when these papers do get published, it’s not easy to locate them, because they are rarely linked to the original studies.
A database described in a preprint posted in April1 aims to address these issues by hosting replication studies from the social sciences and making them more traceable and discoverable. It was launched as part of the Framework for Open and Reproducible Research Training (FORTT), a community-driven initiative that teaches principles of open science and reproducibility to researchers.
The initiative follows other efforts to improve the accessibility of replication work in science, such as the Institute for Replication, which hosts a database listing studies published in selected economics and politics journals that academics can choose to replicate.
The team behind the FORTT database hopes that it will draw more attention to replication studies, which it argues is a fundamental part of science. The database can be accessed through the web application Shiny, and will soon be available on the FORTT website.
Nature Index spoke to one of the project’s leaders, Lukas Röseler, a metascience researcher and director of the University of Münster’s Center for Open Science in Germany.
Why did you create this database?
We’re trying to make it easier for researchers to make their replication attempts public, because it’s often difficult to publish them, regardless of their outcome.
We also wanted to make it easier to track replication studies. If you’re building on previous research and want to check whether replication studies have already been done, it’s often difficult to find them, partly because journals tend to not link them to the original work.
We started out with psychology, which has been hit hard by the replication crisis, and have branched out to studies in judgement and decision-making, marketing and medicine. We are now looking into other fields to understand how their researchers conduct replication studies and what replication means in those contexts.
Who might want to use the database?
A mentor of mine wrote a textbook on social psychology and said that he had no easy way of screening his 50 pages of references for replication attempts. Now, he can enter his references into our database and check which studies have been replicated.
The database can also be used to determine the effectiveness of certain procedures by tracking the replication history of studies. Nowadays, for instance, academics are expected to pre-register their studies — publishing their research design, hypotheses and analysis plans before conducting the study — and make their data freely available online. We would like to empirically see whether interventions such as these affect how likely a study is to be replicable.
2024 Research Leaders
How is the database updated?
It is currently an online spreadsheet, which we created by manually adding the original findings, their replication studies and their outcomes. So far, we have more than 3,300 entries — or replication findings — of just under 1,100 original studies. There are often multiple findings in one study; a replication study might include attempts to replicate four different findings, constituting four entries.
There are hundreds of volunteers who are collecting replications and logging studies on the spreadsheet. You can either just enter a study so that it’s findable, or include both the original study and the replication findings.
We are in contact with teams that conduct a lot of replication research, and we regularly issue calls for people to add their studies. This is a crowdsourced effort and a large proportion of it is based on the FORTT replications and reverses project, which is also crowdsourced. It aims to collate replications and ‘reversal effects’ in social science, in which replication attempts have results in the opposite direction compared with the original.
Do you plan to automate this process?
We are absolutely looking into ways to automate this. For instance, we are working on a machine-readable manuscript template, in which people can enter their manuscript and have it automatically read into the database.
We have code that automatically recognizes DOIs and cross checks them with all the original studies in the database to check whether there is a match. We are working on turning this into a search engine, but it’s beyond our capabilities and resources at the moment.
Does your database provide any data on the replications it hosts?
If you go to our website, there is a replication tracker, where you can see the percentage of studies that were able to replicate original findings, and those that failed to do so.
In a version of the database that we will launch in the coming months, users will be able to choose the criteria by which they judge whether a study successfully replicated the original findings. Right now, it’s all based on how strong the effect sizes — a measure of the relationship between two variables — were on both the original study and the replication attempts, but there are many other criteria and metrics of replication success that we are considering.
We’re also planning to launch a peer-reviewed, open-access journal at FORTT to publish replication studies from various disciplines.
This interview has been edited for length and clarity.
Nature Index’s news and supplement content is editorially independent of its publisher, Springer Nature. For more information about Nature Index, see the homepage.
South Korea’s Share in natural-science journals in the Nature Index is shown alongside its closest competitors in the database. Among these countries, India is the only one to have increased its natural-science output between 2019 and 2023, with a 14.5% jump in adjusted Share between 2022 and 2023 alone.
People power
South Korea has the most researchers in science, technology and innovation roles (full-time equivalent) per million inhabitants — and by a large margin. It is notably the only non-European country in the top 10 by this measure in 2021. Japan, not shown here, is ranked 11th.
Subject strengths
A breakdown of subject contributions to countries’ overall 2023 output in journals tracked by the Nature Index is shown for South Korea and some of its closest competitors. With almost 55% of its output attributed to the physical sciences, South Korea joins India in this group as having a dominant subject that is worth more than half of its total output. France and Switzerland, by comparison, have a more balanced output.
On the up
The fastest rising South Korean institutions for the period 2019 to 2023 have recorded very modest gains in natural-science output in the Index, which could speak to the country’s relatively stable performance in recent years. Samsung Group, the only corporate institution in this list, had the largest percentage increase over the period, at 57.98%. This was from a much smaller adjusted Share of 10.77 in 2019, however, compared with Seoul National University at 174.97.
Most improved
The fastest-rising institutions in four natural-science subjects, and in the natural sciences overall, are shown for the period 2019 to 2023. Institutions are ranked according to change in adjusted Share, which for the Pohang University of Science and Technology was larger in the physical sciences than it was in the natural sciences overall.
Source: Nature Index
Big spender
Although South Korea spends more on its research and development as a proportion of its gross domestic product than most other countries in the world, this does not translate to higher Share in the Nature Index. Its output does seem relatively stable, however; among the selected countries shown below, many recorded a decline in Share (per million people) in natural-science journals over the past five years, but South Korea’s drop was smaller than most.
Citations for cash: researchers have identified services where scholars can buy citations to their papers in bulk.Credit: Vergani_Fotografia/Getty
Research-integrity watchers are concerned about the growing ways in which scientists can fake or manipulate the citation counts of their studies. In recent months, increasingly bold practices have surfaced. One approach was revealed through a sting operation in which a group of researchers bought 50 citations to pad the Google Scholar profile of a fake scientist they had created.
The scientists bought the citations for US$300 from a firm that seems to sell bogus citations in bulk. This confirms the existence of a black market for faked references that research-integrity sleuths have long speculated about, says the team.
The science that’s never been cited
“We started to notice several Google Scholar profiles with questionable citation trends,” says Yasir Zaki, a computer scientist at New York University (NYU) Abu Dhabi, whose team described its sting operation in a February preprint1. “When a manuscript acquires hundreds of citations within days of publication, or when a scientist has an abrupt and large rise in citations, you know something is wrong.”
These practices are troublesome because many aspects of a researcher’s career depend on how many references their papers garner. Many institutions use citation counts to evaluate scientists, and citation numbers inform metrics such as the h-index, which aims to measure scholars’ productivity and the impact of their studies.
Citation manipulation can have real consequences. In June, Spanish newspaper El País reported that the country’s Research Ethics Committee has urged the University of Salamanca to investigate the work of its newly appointed rector, Juan Manuel Corchado, a computer scientist accused of artificially boosting his Google Scholar metrics. (Corchado did not respond to Nature’s request for comment.)
References for sale
Research-integrity watchers had already suspected that citations are for sale at paper mills, services that churn out low-quality studies and sell authorship slots on already-accepted papers, says Cyril Labbé, a computer scientist at Grenoble Alpes University in France. “Paper mills have the ability to insert citations into papers that they are selling,” he says.
In November 2023, analytics firm Clarivate in Philadelphia, Pennsylvania, excluded more than 1,000 researchers from its annual list of highly cited researchers because of fears of citation gaming and ‘hyper-publishing’.
Hundreds of extreme self-citing scientists revealed in new database
In their sting operation, Zaki and his colleagues created a Google Scholar profile for a fictional scientist and uploaded 20 made-up studies that were created using artificial intelligence.
The team then approached a company, which they found while analysing suspicious citations linked to one of the authors in their data set, that seemed to be selling citations to Google Scholar profiles. The study authors contacted the firm by e-mail and later communicated through WhatsApp. The company offered 50 citations for $300 or 100 citations for $500. The authors opted for the first option and 40 days later 50 citations from studies in 22 journals — 14 of which are indexed by scholarly database Scopus — were added to the fictional researcher’s Google Scholar profile.
The team didn’t share the company’s name with Nature, citing concerns that revealing it could draw attention to its website, or the fake Google Scholar profile they created, because this might reveal the identities of the authors of the studies that planted the fake citations. Asked by Nature whether Google Scholar is aware that faked profiles can be created on its site, Anurag Acharya, distinguished engineer at the company said: “While academic misbehaviour is possible, it’s rare because all aspects are visible — articles indexed, articles included by an author on their profile, articles citing an author, where the citing articles are hosted and so on. Anyone in the world can call you on it.”
In another demonstration of citation manipulation, last month researchers created a fake Google Scholar profile for a cat called Larry listing a dozen fake papers with Larry as the sole author. The researchers posted a dozen more nonsensical studies on the academic social-networking site ResearchGate that cited Larry’s papers. A week or so after Larry’s identity was revealed, Google Scholar removed the cat’s studies, those citing Larry, and the accumulated citations. ResearchGate has also removed the bogus studies citing Larry.
Fake preprints
Zaki and colleagues’ sting operation was born out of a broader effort to assess the scale of the fake-citation problem. They used software to examine about 1.6 million Google Scholar profiles that had at least 10 publications. They searched for profiles with more than 200 citations and instances in which researchers’ citations increased by 10 times or more each year or when the rise represented a jump of at least 25% of their total citation count. The team found 1,016 such profiles.
The fight against fake-paper factories that churn out sham science
Zaki says that many citations to the papers on those profiles are from preprint articles that haven’t been peer reviewed and that they are typically listed in the bibliographies of papers but not cited in the main body of the manuscripts.
“Citations can easily be manipulated by creating fake preprints and through paid services,” says co-author Talal Rahwan, a computer scientist at NYU Abu Dhabi.
The authors also surveyed 574 researchers working at the 10 highest-ranked universities in the world. They found that of those universities that consider citation counts when evaluating scientists, more than 60% obtain these data from Google Scholar.
Fishy patterns
Labbé isn’t convinced by the survey’s claim that Google Scholar is widely used to obtain researchers’ citation metrics. Allegations of citation manipulation on Google Scholar have surfaced in the past, he says, and academics have long suspected that there are vendors offering this sort of service. But the sting operation to reveal a citation seller is the first of its kind, he says.
Guillaume Cabanac, a computer scientist at the University of Toulouse in France who has created a tool that flags fabricated papers that contain odd turns of phrase added to circumvent plagiarism-detection software, says that many studies are cropping up with citations to work that has nothing to do with the topic of the study.
Labbé’s team is building a tool that automatically flags fishy citation patterns that might point to manipulation.
How big is science’s fake-paper problem?
To help with that, Zaki’s team proposes a metric called the citation-concentration index, designed to detect cases in which a scientist receives many citations from few sources. Such activity is often a sign of a ‘citation ring’, in which scientists agree to cite one another to inflate each other’s metrics. “Suspicious ones tend to have massive citations stemming from just a few sources,” Rahwan says.
One fear among integrity sleuths is that fraudsters will conceive subtler practices to avoid being found out. For instance, one way to avoid being detected by the citation-concentration index, Labbé notes, is to buy a few citations at a time and not in bulk.
For Labbé, the way to address citation gaming is to change the incentives in academia so that scientists are not under pressure to accumulated as many citations as possible to progress their careers. “The pressure for publication and citation is detrimental to the behaviour of scientists,” he says.
Omitting journal names in your CV could help to prioritize the quality of your science over the prestige of the publication.Credit: Nature
Can you name the journal in which microbiologist Alexander Fleming first reported on the antibacterial properties of penicillin? Or where engineer John O’Sullivan and his colleagues presented the image-sharpening techniques that led to Wi-Fi?
Most of you can easily name the benefits of these breakthroughs, but I expect only a few would know where they were published. Unfortunately, in modern scientific culture, there is too much focus on the journal — and not enough on the science itself. Researchers strive to publish in journals with high impact factors, which can lead to personal benefits such as job opportunities and funding.
But the obsession with where to publish is shaping what we publish. For example, ‘negative’ studies might not be written up — or if they are, they’re spun into a positive by highlighting favourable results or leaving out ‘messy’ findings, to ensure publication in a ‘prestigious’ journal.
Illuminating ‘the ugly side of science’: fresh incentives for reporting negative results
To shift this focus in my own practice, I have removed all the journal names from my CV. Anyone interested in my track record will now see only my papers’ titles, which better illustrate what I’ve achieved. If they want to read more, they can click on each paper title, which is hyperlinked to the published article.
I’m not alone in thinking of this. The idea for removing journal names was discussed at a June meeting in Canberra on designing an Australian Roadmap for Open Research. A newsletter published by the University of Edinburgh, UK, no longer includes journal titles when sharing researchers’ new publications, to help change the culture around research assessment. Celebrating the ‘what’ rather than the ‘where’ is a great idea. This simple change could be extended to many types of research assessment.
Quality over journal titles
It is disorienting at first to see a reference that does not contain a journal title, because this bucks a deeply ingrained practice. But journal names are too often used as a proxy for research excellence or quality. I want people reading my CV to consider what I wrote, not where it was published, which I know is sometimes attributable to luck as much as substance.
Of course, anyone who really wants to judge me by where I’ve published will simply be able to google my articles: I haven’t anonymized the journals everywhere. But removing the names in my CV discourages simplistic scans, such as counting papers in particular journals. It’s a nudge intervention: a reminder that work should be judged by its content first, journal second.
Because I’m a professor on a permanent contract, it’s easier for me to make this change. Some might think that it would be a huge mistake for an early-career researcher to do the same. But there is no stage in our scientific careers at which decisions about hiring and promotion should be based on the ‘where’ over the ‘what’. It would be easier for early-career scientists to make this change if it became normalized and championed by their senior colleagues.
A potential criticism of removing journal names is that there is nothing to stop unscrupulous academics from publishing shoddy papers in predatory journals to create a competitive-looking CV, which could put candidates with genuine papers at a disadvantage. Promotion and hiring committees need to be made aware of the growing problem of faked and poor-quality research and receive training on how to spot flawed science.
However, when a job gets 30 or more applicants, there can be a need for short-cuts to thin the field. I suggest that reading the titles of each applicant’s ten most recent papers would work better than any heuristic based on paper counts or journal names, for only a slight increase in workload.
Imagine a hiring or fellowship committee that receives plain or preprint versions of the every applicant’s five best papers. Committee members who previously relied on simplistic metrics would have to change their practice. Some might simply revert to Google, but others might welcome the challenge of judging the applicants’ works.
Judging researchers is much more difficult than counting impact factors or citations, because science is rarely simple. Simplistic promotion and hiring criteria ignore this wonderful complexity. Changing typical academic CV formats could bring some of it back.
This is an article from the Nature Careers Community, a place for Nature readers to share their professional experiences and advice. Guest posts are encouraged.
Competing Interests
A.B. is a member of the Australian National Health and Medical Research Council (NHMRC) Research Quality Steering Committee, which provides national guidance on good research practice. A.B. is paid for his time to attend committee meetings. A.B. was on the organizing committee for the Policy Roundtable: An Australian Roadmap for Open Research meeting, which is mentioned in the article, and received paid accommodation to attend the meeting.
The metascience unit will fund studies that analyse UK research in the hope of boosting its quality and efficiency.Credit: Sanjeri/Getty
With the launch of a metascience unit late last year, the United Kingdom became one of the first countries to formalize the practice of using scientific methodology to study how research is done. The question now is whether it will produce insights to help elevate UK research and, if so, whether this will influence other countries to launch their own metascience initiatives.
The UK metascience unit was announced in November, under the previous Conservative government, and will continue under the Labour government that took power last month. Its remit is to explore better ways of conducting, publishing and reviewing UK research, as well as distributing and funding it. More broadly, the unit is focused on improving the overall quality and efficiency of UK research.
The project will be run jointly by the UK Department for Science, Innovation and Technology and national funder UK Research and Innovation, in partnership with Open Philanthropy, a charitable research funder based in San Francisco, California, which is contributing £2 million (US$2.6 million) of the total £5 million for the first call for applications.
The first call for grant proposals went out in April, seeking applications from UK researchers from all disciplines who want to conduct metascience studies in their area of expertise, says Stian Westlake, executive chair of the UK Economic and Social Research Council, who was involved in setting up the unit. Applications are also welcomed from dedicated metascience researchers, such as those working in science and technology studies, scientometrics and bibliometrics. Topics of interest include innovations in peer review, funding processes and reproducibility.
Studying science
Applications will be reviewed by a panel of representatives of the UK government, Open Philanthropy and academics in relevant fields, and a maximum of £300,000 will be distributed to each successful grant applicant for a period of between six months and two years.
“It’s not a huge amount of money,” says James Wilsdon, a research policy scholar at University College London, who will be the panel’s chair, but “it’s certainly a very positive step forward”. He adds: “It’s encouraging to see any government or funding organization taking seriously the need for more systematic, robust meta-scientific evidence to inform decision strategies and the managing and monitoring of research innovation systems.” Wilsdon is also founding director of the Research on Research Institute, a UK-based consortium.
2024 Research Leaders
The review process for grant applications will be similar to what UK researchers are used to when applying for government grants, at least for the first funding call, says Westlake. He adds that there are no plans yet to experiment with the review process, although insights from the studies funded by the metascience unit could inform future processes. “We’re not quite in the realm of meta-metascience yet,” he says.
Katy Börner, an information scientist at Indiana University Bloomington, hopes that findings that stem from research funded by the metascience unit will inform decisions by policymakers, funders, educators and other influential groups. Börner notes that it has been difficult to secure funding for research on research in the past.
Behavioural nudges
The launch of the metascience unit follows another initiative in 2010, when the coalition UK government at the time opened a ‘nudge unit’. The unit was dedicated to using insights from behavioural sciences to inform public policy, by ‘nudging’ people to make certain decisions that were in their own interest — for example, prompting people to meet their tax deadlines and avoid fines. The idea stemmed from the 2008 book Nudge, by US economists Richard Thaler — who won the Nobel prize in economic science for his work on the topic in 2017 — and Cass Sunstein.
After the United Kingdom, nudge units were launched in countries including the United States, the Netherlands, Canada, Germany, India, Indonesia, Singapore and Peru. The UK nudge unit, formally called the Behavioural Insights Team, produces regular working papers and reports on the outcomes of experiments it conducts. It was acquired in 2021 by London-based innovation charity Nesta.
Wilsdon says it’s possible that other countries will see the UK metascience unit and want to launch similar initiatives. “Things become fashionable and they travel around,” he says. “Maybe this is the first of many. Who knows?”
Barbara Lancho Barrantes, a scientometrician at the University of Brighton, UK, is not sure whether the idea of a metascience unit can be rolled out in or be applicable to all countries. Every country has its own priorities and resources, she says, and research funding is already scarce in many regions.
Long-term outlook
The search for innovative approaches to funding science has been gaining traction elsewhere. In September, the US National Science Foundation (NSF) announced that it is partnering with the Institute for Progress, a think tank based in Washington DC, to find new ways of funding research. A standout feature of the deal is that it sought to give academics better access to internal NSF data, which is typically not released to external entities, even if it is to analyse their own processes.
The first big challenge for the UK metascience unit will be to ensure that it is funding a good mixture of short-term, eye-catching projects and projects that aim to produce longer-term evidence and insights into how well the system is working, says Wilsdon.
“The big win for me would be less about short-term project wins and more about long-term system reform,” says Wilsdon. This would mean the United Kingdom has the “structures and systems of metascientific data and analysis that it needs over the medium and long term to actually make intelligent decisions about the system”, he adds.
Global warming is costing lives, deepening health inequality and driving the spread of disease-carrying ticks and parasites across Europe, according to a major report.
The report reviewed hundreds of studies on the health effects of climate change — as well as the actions being taken in response — in Europe. Climate and health researcher Rachel Lowe and her colleagues tracked 42 indicators, including those on heat-related deaths, the spread of infectious diseases and trends in research on health and climate change.
Extreme heat harms health — what is the human body’s limit?
“We really need some drastic action to be taken by European countries to help keep the European population, and also populations across the globe, safe from the health impacts of climate change,” says Lowe, who is at the Barcelona Supercomputing Center and at the Catalan Institution for Research and Advanced Studies in Spain.
The report, published last month in Lancet Public Health1, is the second — after one published in 20222 — from a study called ‘The Lancet Countdown: Health and Climate Change in Europe’.
“The report emphasizes the alarming increase in mortality and morbidity linked to rising temperatures, and the proliferation of climate-sensitive diseases,” says Ana Raquel Nunes, a health and environment researcher at the University of Warwick, UK.
Researchers say that further studies should take a holistic approach to the climate–health nexus. “You can’t treat all these health impacts of climate change in isolation,” says Ruth Doherty, a climate-change and health researcher at the University of Edinburgh, UK. “We really need to know about how these multiple exposures affect the population.”
In three graphics, Nature outlines how a warmer world is affecting health and research across Europe.
Deadly heat
Lowe and her colleagues used mortality and temperature data, as well as prior evidence for how heat influences mortality, to estimate that, from 2003–12 to 2013–22, heat-related mortality increased by an average of 17 deaths per 100,000 people per year across Europe. The increase in heat-related mortality was higher in women compared with men (See ‘Heat kills’).
Source: Ref 1.
“Gender disparities may be explained by differences in terms of losing heat from the body and maximum sweat rates,” says Kim van Daalen, who studies climate change, disease and gender inequity at the Barcelona Supercomputing Center. Women might also generally be at greater risk of heat stress after ovulation, when they tend to have higher body temperature, she says.
Another factor that could be driving the gender gap is that women generally reach older ages than men, and older people are generally more vulnerable to heat-related stresses, says Lowe. Older people are also more likely to live alone, which puts them in greater danger from heat, she says.
Ticks and parasites
Warmer temperatures are enabling disease-carrying parasites to expand into more regions and spurring the growth of tick populations. One pathogen that is becoming more widespread owing to climate change is the single-cell parasiteLeishmania infantum. It is transmitted to people when female sandflies (Phlebotomus sp.) bite human skin to feed on blood. The parasite usually causes skin ulcers across the body, which can be debilitating. In extreme cases, it can cause fevers and the swelling of the spleen and liver, and could be fatal.
Source: Ref 1.
The researchers estimated that warmer and more-humid conditions across Europe have enabled sandflies and the parasites they carry to spread north into new territories. Their range was wider in the 2010s than in the 2000s. “Rising temperatures create more favourable conditions for sandflies to survive and reproduce,” says van Daalen. “Warmer conditions can also accelerate the life cycle of the parasite within sandflies,” she says (See ‘Inching North’).
The team also found that warmer temperatures have made Europe more suitable for the tick Ixodes ricinus, which can transmit a range of diseases when it bites people. “Tick-borne diseases, such as Lyme disease and tick-borne encephalitis, cause symptoms ranging from flu-like illness to severe neurological and cardiovascular complications, leading to missed work, long-term disability and substantial health-care costs,” says van Daalen.
Across most of the continent, I. ricinus found a more hospitable climate to feed and grow in 2013–22 than it did in 1951–60, as measured by the average number of months per year when temperatures were optimal for the juvenile stage of its life cycle.
Publishing boom
As the world warms, research on how climate change intersects with Europeans’ health has intensified, as seen in the number of papers tracked in the open-access database OpenAlex. The researchers counted hundreds of studieson how climate change and Europeans’ health intersect, published between 1991 and 2022. The majority of those studies focused on how global warming affects health, but some also looked into the greenhouse gases emitted by health-care systems, or how to protect people from climate change that is already happening (See ‘Hot topic’).
Ref 1.
The authors also found that around 2% of the studies published in 2022 on climate health referenced equality, equity or justice. “This highlights a substantial gap in research,” says van Daalen. “To properly respond to the climate-related health impacts, it is important to understand which populations are disproportionately affected and most at risk,” she says.
