A study finds that many published papers are not accessible to low-vision and blind readers.Credit: Getty
Around three out of every four PDF versions of scholarly papers are largely inaccessible to low-vision and blind readers, a study has found1.
Researchers looked at how often around 20,000 studies published between 2014 and 2023 were compliant with 6 accessibility criteria. That includes providing alternative text for figures and headers for tables, as well as adding the tags necessary to make PDF files accessible to low-vision and blind readers, who typically access these files using assistive reading devices.
Only around 3% of the analysed studies met all six criteria, the analysis found, and just under 75% met none of the criteria at all.
“After 2019, there was a very sharp decline across almost all of the criteria that we measured,” says study co-author Anukriti Kumar, an information scientist at the University of Washington in Seattle. The authors attribute that trend to the move towards rapid-publishing methods such as preprints and online-first publishing, and say that it was exacerbated by the COVID-19 pandemic, which demanded quick communication of research findings.
The analysis was presented in October at the 26th International ACM SIGACCESS Conference on Computers and Accessibility in St. John’s, Canada.
Lucy Lu Wang, also an information scientist at the University of Washington and a co-author of the study, published2 a similar analysis of accessibility of more than 11,000 PDFs in 2021. “Things were mostly improving,” she recalls. But with new global open-access policies and changes with how publishers are producing PDFs, accessibility overall has decreased since then,” she says.
“Accessibility often falls to the wayside, because it disproportionately affects a smaller group of people,” Wang says, “or the kind of people who don’t have as much clout.”
Systemic changes required
For the 2021 analysis, the authors interviewed several low-vision and blind scientists, some of whom said that they chose their fields of study in part because of how easily accessible the associated literature was. “People were drawn to fields that had more accessible papers,” Wang says. “The barriers to working in those fields were lower.”
Sheri Wells-Jensen, a linguistics researcher at Bowling Green State University in Ohio, who is fully blind, tells Nature that the hassle of finding accessible papers is such that she sometimes doesn’t even try. “I never expect to be able to go to an open-access journal and just get the PDF and read it with the same level of ease and convenience as other scientists do,” she says. “We’ve got different software that could do some scanning, but you have to be a little bit of a wizard sometimes.”
Wells-Jensen notes that academic journals rarely provide information about accessibility for scientists with visual impairment in the ‘information for authors’ sections of their websites, making it unclear how researchers should prepare their manuscripts for optimal accessibility. Manuscript-submission systems themselves are also often inaccessible, she adds.
Addressing such accessibility shortfalls will require “systemic changes” from authors, publishers and others, Kumar says.
Artificial-intelligence tools have helped Candice Chu to streamline academic writing.Credit: MirageC/Getty
Since starting my PhD in veterinary pathobiology in 2014, I have looked for ways to streamline my literature-review workflow. Every day, I search for papers, import them into my citation manager, read them and take notes. I can then incorporate those references and insights into manuscripts. But the conventional approach of searching for and downloading PDF files is tedious and inefficient, involving multiple mouse clicks, scattered files and a large disconnect between my notes and the source. Ten years later, with the development of digital and artificial intelligence (AI) tools, I have finally landed on a process that can streamline my academic writing. I call it ACCU — the acquisition, collection, crystallization and utilization workflow.
Acquisition: how I find papers
For quick keyword searches, I use Google Scholar. In settings, under ‘search results’, I set the export format to RefMan for compatibility with my citation manager and add my university library under ‘library links’ so that I can retrieve the full text when it is available.
For systematic searches, I use the PubMed database’s advanced search function, coupled with the EasyPubMedicine Chrome browser extension to display the journal ranking, impact factor and citation count under each hit. On the PubMed search-results page, I can click ‘create RSS’ to turn the results into a web feed in an RSS reader, such as Feedly, which will then alert me to any new papers that fit the search criteria. This allows me to access all newly published papers in my field — as well as multiple journals’ most recent publications — anytime, anywhere, even on my phone.
AI literacy is an essential skill, says Candice Chu.Credit: Butch Ireland Photography
To flesh out my research, I use the AI-powered academic search engines Perplexity and Consensus. These can provide answers to well-defined, natural-language questions, such as, ‘Is vitamin D beneficial for osteoporosis?’ They have a better user interface than ChatGPT, with hyperlinks to the original sources so that I can verify the results.
Alternatively, I can use ResearchRabbit, an AI-based tool that allows me to use papers in my literature collection as seeds to find related publications. I can add the newly identified papers to my reference manager with one click.
Collection: how I store my papers
I use the free and open-source reference manager Zotero because it is easy to use and compatible with many of my other digital tools.
Zotero covers all the basic functions of many available commercial tools and, in my experience, provides better solutions. The Zotero Connectors Chrome browser extension allows me to import papers from Google Scholar searches in batches. A right-click can save academic articles (or any web page) to my Zotero library in a ready-to-cite format. My university library provides the OpenURL link resolver, which I use in Zotero’s settings to automatically download full-text articles. I can double-click on Zotero entries to highlight and annotate the PDF.
Advanced users can expand Zotero with plugins. For example, I use Notero to import all the ‘What Is Your Diagnosis’ articles from the journal Veterinary Clinical Pathology into Notion, a productivity and note-taking app, to create a searchable database for veterinary clinical pathologists and trainees.
South Korea has the resources and history to collaborate with low-income countries in a mutually beneficial way.Credit: imaginima/Getty
Science is often viewed, at least in its most idealized form, as a perfect form of public knowledge that is freely available to everyone. In principle, scientists in low-income countries (LICs) can consume and build on this knowledge to produce their own research. However, the processes of training scientists, acquiring and setting up equipment and materials, and organizing resources and personnel are not straightforward or found in textbooks. They involve tacit knowledge that is often embedded in specific organizational and institutional contexts, such as experimental routines and methods passed down through experiences, which present further barriers for many resource-limited nations.
Collaborative potential
International research collaborations can provide opportunities for LICs to overcome resource limitations, increase visibility and build sustainable scientific capacity. First, such cross-border partnerships can enable researchers in low-income regions to access resources and expertise that might not be locally available. Second, given that a substantial amount of the research produced by scientists in LICs is underappreciated — for example by being cited less on average — collaborations with more-research-intensive countries can help to boost the impact of their work. At the same time, collaborating with researchers in high-income countries (HICs) might, in principle, allow researchers in LICs to find more ways of understanding problems, by combining their local contextual knowledge about a subject with the general knowledge available in HICs.
2024 Research Leaders
Finally, and perhaps most importantly, international collaboration can help to pass on informal tacit knowledge about research practices to researchers in LICs. This can include information on organizing research workflows, securing funding, choosing impactful topics and training students, issues that are essential to developing local and sustainable scientific capacity.
However, despite the importance of international collaboration in building research capacity, LICs are still struggling to play a key part in international science. A preprint posted on 16 October1 suggests that researchers in LICs are more likely than those in HICs to be included as middle authors rather than first or last authors in internationally collaborative papers. Such contributions are also more likely to be completely ignored so that those researchers receive no credit at all. For example, in clinical research, contributors from LICs are less likely to be granted authorship than are those in HICs. This is consistent with the trend of more LIC-based research projects being led by HIC institutions for ease of collecting local data. This pattern reflects deeper inequities: LIC scientists rarely lead research agendas or control project funding, limiting their ability to build independent capacity.
Honest broker
Steps are being taken to make collaboration more equitable, but the challenges might be difficult to overcome given the power and resource imbalances that exist between HICs and LICs. Often, these imbalances have deep historical roots, such as colonialism, or are being shaped by modern geopolitical trends, such as the rise of China.
With a strategic position that is not fully engaged in the US–China scientific rivalry, South Korea stands out as a country uniquely positioned to act as an ‘honest broker’ in research collaborations between HICs and LICs. Unlike many prominent powers that often dominate partnerships by controlling resources and decision-making structures, South Korea, free from colonial ties with LICs, has the potential to foster equitable, sustainable research partnerships (See ‘Index live: growing Korean research performance in an uncertain world’).
South Korea’s rapid scientific and technological development since the 1970s has endowed it with both the resources and experiences necessary to assist LICs. Furthermore, its close political relationships with the United States and European nations, along with strong geographical and historical ties to Asian science powerhouses such as China and Japan, give it a strategically advantageous position. With an increasing budget for international research collaborations, South Korea has a great opportunity to establish partnerships that are mutually beneficial, supporting LIC’s research-capacity building without imposing exploitative dynamics that echo colonial relationships, thus helping to advance scientific equity between HICs and LICs.
Previous cases have shown how successfully South Korea has already been acting in this respect in international science. One is the long-standing collaborations between South Korea and Brazil, which began in the 1990s in many fields, including nuclear energy, biotechnology, information technology and clean technologies. The resource disparities between the two nations were addressed by identifying mutually beneficial areas of collaboration — starting with nuclear energy and agricultural technology. More recently, this expanded to biomedical engineering — to tackle common health challenges such as brain diseases — and to astronomy, focusing on developing advanced optical systems for telescopes. The partnership esulted not only in numerous research publications, but also in South Korea’s inclusion as one of the first Asian participants in Brazil’s Science Without Borders programme, which facilitated the inclusion of hundreds of Brazilian students in science, technology, engineering and mathematics fields in South Korea.
Another example of South Korea’s aspiring role in promoting international science, particularly for LICs, was prominent during the height of the COVID-19 pandemic. The Korea Research Institute of Bioscience and Biotechnology and the National Research Foundation of Korea, both based in Daejeon, supported a three-year grant to build and establish an Asia–Pacific hub for Global Research Collaboration for Infectious Disease Preparedness (GloPID-R), focused on information sharing and proactive monitoring to prepare against infectious diseases. Given South Korea’s role as a hub country tasked with accommodating and addressing diverse and sometimes conflicting viewpoints from the Asia–Pacific countries, it is well positioned to lead the complex yet essential coordination of research collaborations in this area in a way that benefits the whole region.
These are just two examples, but they show how South Korea has been initiating and boldly taking opportunities to promote international collaboration, particularly with LICs. Despite this progress, it should remain vigilant against partnerships that resemble previous exploitative relationships and should strive to adopt non-discriminatory practices by including local authors who contribute to such research. Empowering these researchers can result in meaningful and productive collaborations that enhance autonomy for local scientists, boost motivation and improve research quality.
Index live: growing Korean research performance in an uncertain world
South Korea’s role in international research collaboration will be one of the core themes of a conference hosted by the Korea Advanced Institute of Science and Technology (KAIST) in Daejeon, South Korea, on 5 February 2025, and co-organized by Nature Index.
Titled ‘Index live: growing Korean research performance in an uncertain world’, the event will feature keynote presentations from world-renowned scholars in science policy, including Cassidy Sugimoto (Georgia Institute of Technology), John Walsh (Georgia Institute of Technology) and So Young Kim (KAIST), and panel discussions involving leading academics from South Korea and Japan.
Nature Index’s news and supplement content is editorially independent of its publisher, Springer Nature. For more information about Nature Index, see the homepage.
An unusually long and complex research paper has caught the attention of the scientific community, sparking questions about the ideal length of a paper. The study, by computational biologist Manolis Kellis and his colleagues, was published in Nature1 in July. Spanning 35 pages, it contains more than 20,000 words, and has 16 figures — or 61, if those in the Supplementary information are included. It describes changes in the genes, cellular pathways and cell types of people with Alzheimer’s disease across six regions of the brain, and provides a detailed atlas of gene expression.
When Kellis, who runs a computational-biology laboratory at the Massachusetts Institute of Technology (MIT) in Cambridge, shared the paper on X (formerly Twitter), the size of it seemed to divide his peers. Some were complimentary: “It must have taken a lot of effort and resources to get this done, so all in all, it is a great paper,” one response read. Others were concerned about its usefulness. “How can anyone read this article, let alone review and critique the work?” asked another user.
An analysis of paper characteristics across scientific fields2, published in 2023, suggests that this study is an outlier in the medical and health sciences, where papers typically hover around ten pages in length. However, it is not so unusual when compared with papers in subject areas such as mathematics, law or the humanities — all of which often exceed 20 pages.
Kellis’s work raises the question of how accessible research papers should be, and how readers in and beyond academia are expected to consume them. For example, is it better to publish large data sets alongside long and dense papers, to keep the information contained in one place? Or should researchers home in on specific topics and publish their results across several papers?
Alireza Haghighi, a geneticist at Harvard Medical School in Boston, Massachusetts, says that there is value in the former approach, particularly at a time when data sets are becoming increasingly large. “Although focus has traditionally been important in publications, we must acknowledge the complexity of new methods and the huge volumes of data generated today,” says Haghighi. “Not all papers can or should be understood in one hour.”
Does size matter?
Papers that provide broad, detailed overviews and extensive data sets — sometimes called ‘atlases’, in the omics fields of genomics, transcriptomics and proteomics — allow researchers to see the big picture, says Haghighi. They enable readers to “identify connections across different areas, and generate new hypotheses”, he explains, and adds that he sees them as drivers of innovation that can guide large-scale, integrative research initiatives better than a more focused paper might.
Responding to the discussion on X, Kellis said he understands that some people will be overwhelmed by his lab’s paper. He likened the work to “a good book with many chapters and many pages”, and said that “each paragraph, parenthesis, panel, supplementary figure, can hide potential hints and secrets that the authors themselves may have missed”. Kellis also suggested that for those who were overwhelmed by the results, tools such as the ChatGPT Consensus app, which is regarded as an academic search engine, could be useful for summarizing some of the paper’s findings.
2024 Research Leaders
Li-Huei Tsai, a neuroscientist at MIT and a corresponding author on the paper, told Nature Index that she is proud of the work, which has “produced important insights into genomic underpinnings of Alzheimer’s vulnerability and resilience”. Kellis did not respond to Nature Index’s request for comment.
Researchers who spoke to Nature Index flagged a number of issues with big, data-dense articles. Luke Dabin, an epigeneticist at the Indiana University School of Medicine in Indianapolis, is a “huge fan” of big data sets and atlas papers, because they have the potential to be a hotbed for generating hypotheses and can inform the design of future experiments. But Dabin says that such papers can sometimes be difficult to interpret — even by scientists working in the same field — and can have quality-control issues. “The Kellis paper has 475 figure panels and is difficult for me to digest, let alone someone with no training or experience in single-cell omics,” Dabin says. Haghighi agrees that accuracy can become a problem in large papers. “We should appreciate that atlas maps are more prone to inaccuracies due to their scope and complexity,” he says.
Such papers can also be resource-heavy for journal editors to publish. It took almost two years for Kellis’s paper to progress from acceptance to publication, although it might not have been under review the entire time. A spokesperson for Nature noted that “the length of the review process for papers submitted to Nature varies considerably from manuscript to manuscript”, and said that its primary focus is “to ensure that a rigorous peer-review process takes place”. (Nature Index’s news and supplement content is editorially independent of its publisher, Springer Nature.)
On X, Kellis noted that “it was a Herculean task by the reviewers and editors, as it was of course for the authors, to go through every figure, every panel, and every result” as part of the publishing process.
A case for brevity?
Some researchers argue that there is simply not enough time to read such long and dense papers. “The readership on most academic papers is low anyway, so writing a long paper is just inviting it not to be read even more,” says Daniel Price, an astrophysicist at Monash University in Melbourne, Australia, and former editor-in-chief of the journal Publications of the Astronomical Society of Australia, which publishes research on data-heavy topics such as modelling and computational astronomy.
Price says it’s unlikely that anyone has ever read the entirety of one of his monster astrophysics papers3, which clocks in at 82 pages and has 57 figures. “It’s definitely too long,” he says of the paper, admitting that it could easily have been 60 pages instead. The problem with going long, he adds, is that it’s “undisciplined” and compromises the ability to self-edit.
Haghighi says some improvements could be made to long, data-heavy papers. He suggests that publishers standardize the way such papers are formatted and published by introducing new editorial guidelines and implementing “a dynamic, continuous review process” that allows authors to update their work regularly over time, after publication. “I appreciate that this might not be easy,” says Haghighi, but “it would make the review process more effective and consistent and make it easier for the scientific community”.
Formatting guidelines at most major journals tend to favour shorter articles with fewer figures. Nature, for instance, suggests the typical length of biological, clinical and social sciences papers should not normally exceed 8 pages, or 4,300 words, and 5–6 figures. That said, it does not enforce specific limits, and instead leaves this up to the editor’s discretion.
In astrophysics, a field that is characterized by vast data sets that are often analysed by large, international teams, there are some examples of how a research finding can be broken down into more digestible parts. For instance, after the first image of a black hole was captured by the Event Horizon Telescope — a global network of radio telescopes run by a group of more than 300 scientists — the team published 6 papers in a special edition of The Astrophysical Journal Letters. Each paper presented an aspect of the research, looking at methodology, specific features of the black hole and the image itself.
Price thinks paper series such as this are “definitely a better idea” than one long paper, and adds that there is a lot to be said for concise papers. He points to a 2016 paper published by the LIGO Scientific Collaboration4 — a conglomerate of more than 100 institutions collaborating in the search for gravitational waves — after its seminal detection of gravitational waves using instruments in Washington and Louisiana. “It’s eight pages [ten, including references] and it revolutionized astrophysics,” he says.
After graduating from the medical and biological illustration programme at Johns Hopkins University in Baltimore, Maryland, Shiz Aoki fulfilled a long-held dream: she launched her own company. Founded in 2010 in Toronto, Canada, Anatomize Studios works with large clients — pharmaceutical companies, magazines and medical professionals with niche needs and capacious budgets. Yet Aoki would often also field requests from individual researchers. They wanted to create visualizations for papers, presentations or outreach, but struggled to distil their complex science down to something approachable, let alone visually appealing.
“I must have turned away hundreds of scientists and saw them taking to PowerPoint to create content that, sadly, didn’t do justice to these really important scientific discoveries they were making,” Aoki recalls. “I realized that my love of art is not just a passion thing — that science was actually being stalled by a lack of tools and understanding of science communication.”
Colour me better: fixing figures for colour blindness
Fortunately, it’s easier than ever for researchers to create compelling figures and images, even without a background in design. For one thing, there’s BioRender, a web-based app that Aoki co-founded in 2017. Akin to Adobe Illustrator, but for life scientists, BioRender includes both bioscience-specific drawing tools and a library of more than 50,000 scientifically accurate icons. This resource and others like it — including BioIcons, Reactome and Servier Medical Art — show just how far the fields of data visualization and scientific illustration have come in the past few years, and how scientists remain hungry for tools to help them depict and share their work.
Nature contacted graphic designers, scientific and medical illustrators, and journal art directors to glean tips and resources for creating polished visuals. Here’s what they said.
Prioritize illustrations
The design of figures might seem secondary to running experiments and writing them up for publication. But visualizations can help readers to make sense of abstract concepts in a way that words alone cannot.
“The figures you choose are actually really important,” says Kelly Krause, creative director for the Nature family of journals, who is based in New York City. “People make snap judgements based on visuals, and if they don’t look good, they can steer someone away from a paper that otherwise they might like to read.” Think, for instance, of a graphical abstract that can serve as an advertisement for a research article.
The software that powers scientific illustration
So, devote time to your visuals. Decide what information is essential, make an outline of the content, and edit your figures as ruthlessly as you would any manuscript.
“Almost every time I talk to a scientist, they initially give me way more information than I need, because every single detail feels important,” says Kelly Finan, a designer based in Hop Bottom, Pennsylvania. “But I often find that when I then ask them to explain their work, scientists become aware of what’s extraneous and what isn’t.”
Identify your audience
You wouldn’t write a popular-science talk as you would a research paper, and the same goes for visualizations (see ‘Focus on basic design principles’). Is the goal to inform the reader, elicit an emotion or present data in a unique way? The answer can guide not just the content, but also style choices. “In certain fields, there’s an established way of doing things, but in others, there’s room to be more creative while still maintaining accuracy,” says Nobles Green II, the founder of Amplify Biovisuals in Atlanta, Georgia, and president of the Association of Medical Illustrators.
Focus on basic design principles
Familiarity with the basics of design — such as hierarchy, composition, colour and typography — can go a long way when it comes to producing polished figures.
Hierarchy
• Let the graphic match the flow of the language used. Because most languages read from left to right, you might want to design your graphic to ‘start’ at the top left.
• Use numbers, bold and italicized lettering, and different font sizes to guide the reader through the image.
• Use left- or right-justified text. Centre-aligned text is harder for the brain to process.
Composition
• Be consistent with style choices, such as those concerning fonts, the colour palette and iconography.
• Use white space to make your visualization easier to digest.
• Focus your visualization on a single goal; create different graphics for different audiences.
Colour
• Use a colour-palette generator to make visualizations more accessible.
• Supplement colour with different line styles (for example, solid, dotted, dashed) to aid comprehension.
• Use the cyan, magenta, yellow and key (CMYK) colour model for print, and the red, green, blue (RGB) model for images that will remain digital.
Typography
• Opt for a font with a uniform line thickness (also called the stroke weight) such as Courier or Roboto Mono.
• Avoid mixing many different fonts in a single image, although some designers will choose a serif font for the main text and a complementary sans serif for subheadings and labels.
• Ensure that text is readable: use a point size of at least 12 for main text, and 7 for labels.
Similarly, consider the intended audience. Nicolle Fuller, the founder and creative director of SayoStudio, a science-communication firm in Bellevue, Washington, says this helps to set boundaries around the amount and types of information necessary in a visualization. “You can get away with more complexity when you’re making graphics for other scientists,” she explains — for instance, by including membrane proteins on the cell surface that would overcomplicate an image for the lay reader.
Some designers therefore warn against trying to make a single visualization serve too many purposes. Instead, they say, it’s better to design a range of items — an infographic for social media, a visual abstract and a figure for a seminar presentation, for instance — using the same information. Fuller says that considering the audience has helped clients to think creatively about their data, prompting occasional “aha moments”.
Don’t over-design
With academic manuscripts ballooning in size, it can be tempting to let figures do the same. But more information doesn’t necessarily lead to greater comprehension, and many illustrators live by the motto that less is more.
Illustration: get your research the attention it deserves
“There’s a tendency to overly decorate a figure — add a gradient or a shadow to make it look more jazzy — that actually gets in the way,” Krause says. “You wouldn’t expect flowery prose in a scientific paper, so why would you do that to your figures?”
Ashleigh Campsall, a senior graphic designer at the life-sciences magazine The Scientist, says lean graphics tend to look more professional, and the more white space, the better. “Letting everything breathe makes it easy to digest and interpret, and takes away some of the mental work for the reader,” she says.
Think accessibility
As dedication to diversity, equity and inclusion has grown, so too has the academic community’s embrace of inclusive visualization methods. For example, colour palettes should be suitable for people with a colour-vision deficiency or who are colour-blind, but should incorporate redundancy, too. A line graph might use different colours to indicate each treatment, for instance, but you can also use solid, dashed and dotted lines to increase comprehension, as well as more-descriptive captions.
Create ‘alt text’, too — a written description of an image to be read aloud by a screen reader. One guideline is to limit alt text to roughly 280 characters, or about the length of a social-media post. And use that space creatively, Green advises: you’re trying to paint a picture with words.
Use AI sparingly (or not at all)
Image generators powered by artificial intelligence (AI) have made it easier than ever to create seemingly high-quality pictures from scratch. But almost as soon as these tools appeared, horror stories emerged. Several papers have been retracted owing to bizarre, AI-generated visualizations, including two published earlier this year, one showing a rat with overly large testes in Frontiers in Cell Development and Biology and the other containing an anatomically flawed figure with nonsense labels in Medicine.
NatureTech hub
Many publishers now ban AI-generated images from manuscripts, and designers who spoke to Nature say they mostly avoid the technology. Campsall, for example, might pull a stock image into Adobe Illustrator and use its AI generator to extend the background. “But for wholesale image design, the technology is really just not there yet,” she says. (Citing the unstable legal framework surrounding generative-AI-based images, Nature has so far barred their use except in instances in which AI is the research focus.)
But other designers, including Aoki, say there’s room to leverage AI creatively. Just as writers might use a chatbot to brainstorm headlines or check a draft for tone, image generators can be helpful during the mock-up process. BioRender, Aoki says, is beta-testing a handful of AI-powered tools that allow users to input a text description — say, a cell–cell interaction or an experimental timeline — and get a draft figure out.
“The difference here is that the data that we’re training on isn’t just random data from the Internet, it’s our massive library of vetted icons,” says Aoki, adding that humans must still provide the final stamp of approval. “Scientific integrity and accuracy are so important, so we want to make sure we get this right.”
Even Cathy Foley, Australia’s chief scientist, has encountered unhelpful peer-review comments on her work.Credit: Fairfax Media Archives/Getty
Electromagnetics researcher Akhlesh Lakhtakia is head of a leading US department of engineering science at Pennsylvania State University in University Park, the author of more than 840 journal articles and a fellow of 9 learned societies. But in 1988, when he was an assistant professor, a peer reviewer said of a paper he had submitted for publication: “This is rubbish. Obviously the author or authors had no EM [electromagnetics] training nor physical intuition.”
Peer review, which has for centuries been the standard tool to determine an academic paper’s suitability for publication, is known to be flawed1. Now, one of its major weaknesses, sheer bad manners on the part of the reviewer, has been highlighted in a YouTube video from IOP Publishing (IOPP), headquartered in Bristol, UK — a society-owned publisher of more than 90 journals.
Released to mark Peer Review Week, which this year runs from 23 to 27 September, the one-minute film features four scientists who hold placards showing the rude, inappropriate or irrelevant reviews that they received when they were early-career researchers. At the same time, an overlay lists their stellar achievements since. Activities taking place during this year’s Peer Review Week, involving more than 35 organizations around the world, will focus on innovation and technology, including artificial intelligence (AI) and how it can be used to automate administrative tasks in the peer-review process.
A study published in 2019 revealed that six in ten researchers in the international science, technology, engineering and mathematics (STEM) community have received at least one unprofessional review — and of those, seven in ten have received several2.
Such reviews don’t always cause long-term damage. Lakhtakia tells Nature that he felt “outraged for a few weeks” after he was refused the chance to submit a revised manuscript that would refute the reviewer’s criticisms. “Then,” he adds, “I wrote a monograph on the broader research topic that led to my elevation to the fellowship of a major learned society in 1992.”
Physicist Cathy Foley, Australia’s chief scientist, who has published 112 refereed papers in international journals, recalls a reviewer’s unhelpful comments on a manuscript that she and a younger colleague had “put our heart and soul into drafting” in 2009, when she was a research-programme leader.
“It was written in a very personal way that suggested our team was substandard and we are not worthy of being researchers,” she says. “It took a lot of discussion and coaching to help us see beyond the nasty comments and look for the research advice. Focusing on that enabled us to revise the paper and move on.”
But not everyone shrugs off insulting remarks. Their impact on self-confidence, productivity and career trajectories can be significant, says Laura Feetham-Walker, IOPP’s reviewer-engagement manager, who led the video project. Here, she explains why mean-spirited peer-review comments should be challenged, and why the science community needs to discuss this commonplace humiliation of its younger members.
When did you realize unprofessional reviewer comments were an issue?
I first heard academics discussing this problem shortly after I joined IOPP in 2020 as its first reviewer-engagement manager, while I was running training workshops for early-career reviewers. Before that, I had worked at BMJ Group and The Lancet. I see my role as engaging with reviewers and supporting them in submitting excellent, constructive reports.
Akhlesh Lakhtakia, who leads a US department of engineering science, holds a piece of paper showing a rude reviewer comment that he received early in his career.Credit: IOP Publishing
It became clear at the workshops how many researchers have been affected by this issue — including the senior reviewers, who spoke up about the rude comments they’d received early in their careers.
Is there a fine line between useful criticism and rudeness?
Not at all. The two are very different. You can have a critical, even very negative, review that is not at all problematic. In the 2019 study, which received feedback from 1,106 STEM professionals who had been first authors on manuscripts submitted to peer-reviewed journals, the definition was clear-cut: unprofessional peer review is that which is unethical, irrelevant, mean-spirited or cruel and lacking constructive criticism2.
Should academics learn to be a bit more thick-skinned?
Maybe. Lakhtakia advises junior researchers to “dispassionately evaluate criticism and then proceed accordingly”. But people from groups that are historically under-represented in STEM — women, non-binary people and those from ethnic minorities — are most likely to report that their confidence as a scientist has been undermined by rude reviews2. They are also the groups most likely to report long-term setbacks in their productivity and career advancement. It’s easy to see how that might happen: if you’ve already got imposter syndrome, and your confidence is low, a mean comment might really get to you. This matters. The STEM community can’t afford to allow unprofessional peer review to disempower effective researchers or lead to important work going unpublished.
What can be done?
It’s an editor’s job to sift out unprofessional comments in reviews. IOPP policy, in line with guidance from the Committee on Publication Ethics, a non-profit organization in Eastleigh, UK, is for the editor to ‘rescind’ problematic reviews and ask the reviewer to revise and resubmit them. If the reviewer declines, then the editor might make minor amendments to remove any problematic comments. But editors are under time pressure and deal with many peer-review reports every day, so some inappropriate comments slip through the net. We need to define unprofessional reviewing, to make it easier to track and to filter such comments out.
We also need to acknowledge that most reviewers do a brilliant job in difficult circumstances and will welcome support to improve their skill and confidence as a reviewer. IOPP now offers a free peer-review training course, available to everyone. Those who complete it earn a certificate, and early-career reviewers can include it on their CV.
But most of all, scientists need to talk more about rude reviews. That’s why we made the video.
Do anonymous reviews encourage rudeness?
Perhaps, yes. There is some evidence of an ‘online disinhibition effect’ — or a lack of restraint that people feel with they communicate over the Internet — so rudeness might have increased as peer review has moved online, although more research is needed in this area.
What about open peer review? Is that helping to end rude reviews?
It has had a major impact, undoubtedly. At IOPP, in February 2022, we introduced a version of open peer review (OPR), called transparent peer review (TPR), throughout our open-access journals. TPR shows the complete peer-review process, from initial review to final decision, with the reviewer reports published alongside accepted articles. It requires both authors and reviewers to opt in. Anecdotally, senior staff on TPR journals say they have never seen rude or unprofessional comments in TPR reviewer reports. But TPR would have to be mandatory for reviewers to completely eliminate rudeness.
Is there a downside to TPR?
No. But uptake has been modest. Only about half of our authors choose to make their reviewer reports visible. The number of STEM journals that use TPR or OPR is relatively low, so often it isn’t an option for authors and reviewers.
In 2021, we introduced double-anonymous (DA) peer review. We were the first physics publisher to adopt this approach across our entire portfolio. Under DA, both authors and reviewers are anonymous. The main aim is to reduce bias in science publishing with respect to gender, race, country of origin and affiliation, the latter reflecting the little-acknowledged risk of ‘prestige bias’, or the favouring of work by scientists associated with elite institutions.
We were also the first society publisher to combine DA and TPR throughout our open-access journals. The reviewer and author are anonymous throughout the review process, but the reviewers’ names and full reports are published with the final research.
We know DA peer review works. Women and non-binary people who choose to submit through DA are 11% more likely to have their papers accepted. In particular, sub. mitting authors in Australasia (9.8%) and Africa (7.7%) are more likely to have their manuscript accepted if it is anonymized, and only authors in western Europe benefit from their papers not being anonymized. We also know, again anecdotally, that DA reduces the frequency of rude comments.
Why do you think that is?
Remember that reviewers are always ‘seen’ by their editors. Our internal editors rate every review on a scale of 1 to 5. Those who get a 5 achieve IOP trusted reviewer status. A review containing unprofessional comments will almost always get a score of 1, or occasionally 2 if the comments are borderline unprofessional.
Last November, we introduced reviewer feedback, which allows reviewers to request their review’s score direct from the editorial team. More than 23,000 reviewers have already opted to receive this feedback. We also offer an overview of how to critique a scientific manuscript, which includes examples of reports, for papers from eight fields of physics, that were rated 1, 3 or 5.
Reviewers’ comments can also be seen by other reviewers during co-reviewing, whereby a reviewer formally invites a colleague to collaborate with them. More than eight in ten reviewers who have participated in co-review have told us that they find it useful or very useful.
The signs all indicate that these mechanisms work together to improve reviewers’ understanding of how to do good peer review — and that will benefit science at large.
Although ChatGPT and other generative AI chatbots are transformative tools, risks to privacy and content ownership are baked in.Credit: Jaap Arriens/NurPhoto/Getty
Timothée Poisot, a computational ecologist at the University of Montreal in Canada, has made a successful career out of studying the world’s biodiversity. A guiding principle for his research is that it must be useful, Poisot says, as he hopes it will be later this year, when it joins other work being considered at the 16th Conference of the Parties (COP16) to the United Nations Convention on Biological Diversity in Cali, Colombia. “Every piece of science we produce that is looked at by policymakers and stakeholders is both exciting and a little terrifying, since there are real stakes to it,” he says.
But Poisot worries that artificial intelligence (AI) will interfere with the relationship between science and policy in the future. Chatbots such as Microsoft’s Bing, Google’s Gemini and ChatGPT, made by tech firm OpenAI in San Francisco, California, were trained using a corpus of data scraped from the Internet — which probably includes Poisot’s work. But because chatbots don’t often cite the original content in their outputs, authors are stripped of the ability to understand how their work is used and to check the credibility of the AI’s statements. It seems, Poisot says, that unvetted claims produced by chatbots are likely to make their way into consequential meetings such as COP16, where they risk drowning out solid science.
Nature Career Guide: Faculty
“There’s an expectation that the research and synthesis is being done transparently, but if we start outsourcing those processes to an AI, there’s no way to know who did what and where the information is coming from and who should be credited,” he says.
Since ChatGPT’s arrival in November 2022, it seems that there’s no part of the research process that chatbots haven’t touched. Generative AI (genAI) tools can now perform literature searches; write manuscripts, grant applications and peer-review comments; and even produce computer code. Yet, because the tools are trained on huge data sets — that often are not made public — these digital helpers can also clash with ownership, plagiarism and privacy standards in unexpected ways that cannot be addressed under current legal frameworks. And as genAI, overseen mostly by private companies, increasingly enters the public domain, the onus is often on users to ensure that they are using the tools responsibly.
Bot bounty
The technology underlying genAI, which was first developed at public institutions in the 1960s, has now been taken over by private companies, which usually have no incentive to prioritize transparency or open access. As a result, the inner mechanics of genAI chatbots are almost always a black box — a series of algorithms that aren’t fully understood, even by their creators — and attribution of sources is often scrubbed from the output. This makes it nearly impossible to know exactly what has gone into a model’s answer to a prompt. Organizations such as OpenAI have so far asked users to ensure that outputs used in other work do not violate laws, including intellectual-property and copyright regulations, or divulge sensitive information, such as a person’s location, gender, age, ethnicity or contact information. Studies have shown that genAI tools might do both1,2.
Chatbots are powerful in part because they have learnt from nearly all the information on the Internet — obtained through licensing agreements with publishers such as the Associated Press and social-media platforms including Reddit, or through broad trawls of freely accessible content — and they excel at identifying patterns in mountains of data. For example, the GPT-3.5 model, which underlies one version of ChatGPT, was trained on roughly 300 billion words, which it uses to create strings of text on the basis of predictive algorithms.
The approach to AI regulation is likely to differ between the United States and Europe.Credit: Amanda Andrade-Rhoades for The Washington Post/Getty
AI companies are increasingly interested in developing products marketed to academics. Several have released AI-powered search engines. In May, OpenAI announced ChatGPT Edu, a platform that layers extra analytical capabilities onto the company’s popular chatbot and includes the ability to build custom versions of ChatGPT.
Two studies this year have found evidence of widespread genAI use to write both published scientific manuscripts3 and peer-review comments4, even as publishers attempt to place guardrails around the use of AI by either banning it or asking writers to disclose whether and when AI is used. Legal scholars and researchers who spoke to Nature made it clear that, when academics use chatbots in this way, they open themselves up to risks that they might not fully anticipate or understand. “People who are using these models have no idea what they’re really capable of, and I wish they’d take protecting themselves and their data more seriously,” says Ben Zhao, a computer-security researcher at the University of Chicago in Illinois who develops tools to shield creative work, such as art and photography, from being scraped or mimicked by AI.
When contacted for comment, an OpenAI spokesperson said the company was looking into ways to improve the opt-out process. “As a research company, we believe that AI offers huge benefits for academia and the progress of science,” the spokesperson says. “We respect that some content owners, including academics, may not want their publicly available works used to help teach our AI, which is why we offer ways for them to opt out. We’re also exploring what other tools may be useful.”
In fields such as academia, in which research output is linked to professional success and prestige, losing out on attribution not only denies people compensation, but also perpetuates reputational harm. “Removing peoples’ names from their work can be really damaging, especially for early-career scientists or people working in places in the global south,” says Evan Spotte-Smith, a computational chemist at Carnegie Mellon University in Pittsburgh, Pennsylvania, who avoids using AI for ethical and moral reasons. Research has shown that members of groups that are marginalized in science have their work published and cited less frequently than average5, and overall have access to fewer opportunities for advancement. AI stands to further exacerbate these challenges, Spotte-Smith says: failing to attribute someone’s work to them “creates a new form of ‘digital colonialism’, where we’re able to get access to what colleagues are producing without needing to actually engage with them”.
Computational chemist Evan Spotte-Smith avoids using AI tools for ethical reasons.Credit: UC Berkeley Engineering Student Services
Academics today have little recourse in directing how their data are used or having them ‘unlearnt’ by existing AI models6. Research is often published open access, and it is more challenging to litigate the misuse of published papers or books than that of a piece of music or a work of art. Zhao says that most opt-out policies “are at best a hope and a dream”, and many researchers don’t even own the rights to their creative output, having signed them over to institutions or publishers that in turn can enter partnerships with AI companies seeking to use their corpus to train new models and create products that can be marketed back to academics.
Representatives of the publishers Springer Nature, the American Association for the Advancement of Science (which publishes the Science family of journals), PLOS and Elsevier say they have not entered such licensing agreements — although some, including those for the Science journals, Springer Nature and PLOS, noted that the journals do disclose the use of AI in editing and peer review and to check for plagiarism. (Springer Nature publishes Nature, but the journal is editorially independent from its publisher.)
Other publishers, such as Wiley and Oxford University Press, have brokered deals with AI companies. Taylor & Francis, for example, has a US$10-million agreement with Microsoft. The Cambridge University Press (CUP) has not yet entered any partnerships, but is developing policies that will offer an ‘opt-in’ agreement to authors, who will receive remuneration. In a statement to TheBookseller magazine discussing future plans for the CUP — which oversees 45,000 print titles, more than 24,000 e-books and more than 300 research journals — Mandy Hill, the company’s managing director of academic publishing, who is based in Oxford, UK, said that it “will put authors’ interests and desires first, before allowing their work to be licensed for GenAI”.
Some authors are unsettled by the news that their work will be fed into AI algorithms (see ‘How to protect your intellectual property from AI’). “I don’t feel confident that I can predict all the ways AI might impact me or my work, and that feels frustrating and a little frightening,” says Edward Ballister, a cancer biologist at Columbia University in New York City. “I think institutions and publishers have a responsibility to think about what this all means and to be open and communicative about their plans.”
How to protect your intellectual property from AI
New laws will ultimately establish more robust expectations around ownership and transparency of the data used to train generative AI (genAI) models. Meanwhile, there are a few steps that researchers can take to protect their intellectual property (IP) and safeguard sensitive data.
1. Think critically about whether AI is truly needed.
Abstaining from using genAI might feel like missing out on a golden opportunity. But for certain disciplines — particularly those that involve sensitive data, such as medical diagnoses — giving it a miss could be the more ethical option. “Right now, we don’t really have a good way of making AI forget, so there are still a lot of constraints on using these models in health-care settings,” says Uri Gal, an informatician at the University of Sydney in Australia, who studies the ethics of digital technologies.
2. If you do use AI, implement safeguards.
Specialists broadly agree that it’s nearly impossible to completely shield your data from web scrapers, tools that extract data from the Internet. However, there are some steps — such as hosting data locally on a private server or making resources open and available, but only by request — that can add an extra layer of oversight. Several companies, including OpenAI, Microsoft and IBM, allow customers to create their own chatbots, trained on their own data, that can be isolated in this way.
3. When possible, opt out.
The enforceability of opt-out policies that omit data from AI training sets varies widely, but companies such as Slack, Adobe, Quora, Squarespace, Substack and OpenAI all offer options to prevent content from being scraped. However, some platforms make the process more challenging than others or limit the option to certain types of account. If you’re good at coding, you can modify your personal website’s robots.txt file, which tells web crawlers whether they are allowed to visit your page, to keep the tools from scraping your content.
4. If you can, ‘poison’ your data.
Scientists can now detect whether visual products, such as images or graphics, have been included in a training set, and have developed tools that can ‘poison’ data such that AI models trained on them break in unpredictable ways. “We basically teach the models that a cow is something with four wheels and a nice fender,” says Ben Zhao, a computer-security researcher at the University of Chicago in Illinois. Zhao worked on one such tool, called Nightshade, which manipulates the individual pixels of an image so that an AI model associates the corrupted pattern with a different type of image (a dog instead of a cat, for example). Unfortunately, there are not yet similar tools for poisoning writing.
5. Voice your concerns.
Academics often sign their IP over to institutions or publishers, giving them less leverage in deciding how their data are used. But Christopher Cornelison, the director of IP development at Kennesaw State University in Georgia, says it’s worth starting a conversation with your institution or publisher if you have concerns. These entities could be better placed to broker a licensing agreement with an AI company or pursue litigation when infringement seems likely to happen. “We certainly don’t want an adversarial relationship with our faculty, and the expectation is that we’re working towards a common goal,” he says.
Some evidence suggests that publishers are noting scientists’ discomfort and acting accordingly, however. Daniel Weld, chief scientist at the AI search engine Semantic Scholar, based at the University of Washington in Seattle, has noticed that more publishers and individuals are reaching out to retroactively request that papers in the Semantic Scholar corpus not be used to train AI models.
The law weighs in
International policy is only now catching up with the burst of AI technology, and clear answers to foundational questions — such as where AI output falls under existing copyright legislation, who owns that copyright and what AI companies need to consider when they feed data into their models — are probably years away. “We are now in this period where there are very fast technological developments, but the legislation is lagging,” says Christophe Geiger, a legal scholar at Luiss Guido Carli University in Rome. “The challenge is how we establish a legal framework that will not disincentivize progress, but still take care of our human rights.”
Dragoş Tudorache was instrumental in designing the world’s first comprehensive AI legislation, the EU AI Act.Credit: European Parliament
Even as observers settle in for what could be a long wait, Peter Yu, an intellectual-property lawyer and legal scholar at Texas A&M University School of Law in Fort Worth, says that existing US case law suggests that the courts will be more likely to side with AI companies, in part because the United States often prioritizes the development of new technologies. “That helps push technology to a high level in the US when a lot of other countries are still trying to catch up, but it makes it more challenging for creators to pursue suspected infringement.”
The European Union, by contrast, has historically favoured personal protections over the development of new technologies. In May, it approved the world’s first comprehensive AI law, the AI Act. This broadly categorizes uses of AI on the basis of their potential risks to people’s health, safety or fundamental rights, and mandates corresponding safeguards. Some applications, such as using AI to infer sensitive personal details, will be banned. The law will be rolled out over the next two years, coming into full effect in 2026, and applies to models operating in the EU.
The impact of the AI Act on academia is likely to be minimal, because the policy gives broad exemptions for products used in research and development. But Dragoş Tudorache, a member of the European Parliament and one of the two lead negotiators of the AI Act, hopes the law will have trickle-down effects on transparency. Under the act, AI companies producing “general purpose” models, such as chatbots, will be subject to new requirements, including an accounting of how their models are trained and how much energy they use, and will need to offer opt-out policies and enforce them. Any group that violates the act could be fined as much as 7% of its annual profits.
Tudorache sees the act as an acknowledgement of a new reality in which AI is here to stay. “We’ve had many other industrial revolutions in the history of mankind, and they all profoundly affected different sectors of the economy and society at large, but I think none of them have had the deep transformative effect that I think AI is going to have,” he says.
Bioinformatician Sam Payne stumbled on a manuscript in March that included figures that, he says, looked identical to those in a paper he published in 2021.Credit: Getty
When bioinformatician Sam Payne was asked to review a manuscript on a topic relevant to his own work, he agreed — not anticipating just how relevant it would be.
The manuscript, which was sent to Payne in March, was about a study on the effect of cell sample sizes for protein analysis. “I immediately recognized it,” says Payne, who is at Brigham Young University in Provo, Utah. The text, he says, was similar to that of a paper1 he’d authored three years earlier, but the most striking feature was the plots: several were identical down to the last data point. He fired off an e-mail to the journal, BioSystems, which promptly rejected the manuscript.
In July, Payne discovered that the manuscript had been published2 in the journal Proteomics, and he alerted the editors. On 15 August, the journal retracted the paper. An accompanying statement cited “major unattributed overlap between the figures” in it and Payne’s work. In response to questions from Nature, a spokesperson for Wiley, which publishes Proteomics, said, “This paper was simultaneously submitted to multiple journals and included plagiarized images.”
AI is complicating plagiarism. How should scientists respond?
The retraction statement also stated that four of the authors said they “did not participate in the writing and submission of the article and gave no consent for publication”, and that the fifth author did not respond. However, Nature’s news team found links between several of the authors and International Publisher, a paper mill based in Moscow. Neither the authors nor International Publisher responded to Nature’s requests for comment.
The alleged plagiarism of Payne’s paper highlights systemic vulnerabilities in the global research community, says Lisa Rasmussen, editor-in-chief of the journal Accountability in Research. According to one analysis, roughly 70,000 papers with characteristics common to work produced by paper mills were published in 2022 alone.
Despite the scale of the problem, there is no Interpol equivalent for journals, nor an official authority to provide industry-wide alerts about suspicious manuscripts. “It was just a complete lucky break that the person asked to review it was the author,” Rasmussen says. “Obviously our system should not depend on that kind of serendipity.”
Carbon copy
Although some figures in the BioSystems manuscript were direct copies of those in Payne’s paper, others were simply replotted using his data, which are publicly available, he says. He shared the disconcerting experience on X, formerly known as Twitter. “Well, it happened,” he wrote. He was reviewing a manuscript, he wrote in a post, that included “a direct copy of the figures” in one of his own papers.
Source: Ref. 1 and Ref. 2
When, months later, he discovered the Proteomics paper, he posted a follow-up. “Well. It REALLY happened” — the paper that he had been asked to review had been published. Two weeks later, Proteomics retracted the paper, citing plagiarism of images.
Unlike the figures, the main text of the Proteomics paper is similar to that of Payne’s, but not identical. For example, Payne and his colleagues wrote:
“From the large population of 10,000 cells, we subsampled a given number of cells n_sample ∈ [7, 16, 20, 30, 100] and calculated S/Vest.”
The corresponding paragraph of the Proteomics paper features the same numbers and many of the same words:
“The authors calculated S/Vest using sample n = [7, 16, 20, 30, 100] cells from a population of 10,000 cells.”
The use of the third person caught Payne’s eye. He says such oddities led him to think his paper had been paraphrased using artificial intelligence (AI) to create believable but different text.
Paper pushing
In the course of reporting, Nature found links between authors of the Proteomics paper and a paper mill. Two authors, Dmitrii Babaskin and Tatyana Degtyarevskaya, both at the I.M. Sechenov First Moscow State Medical University, had separate articles3,4 retracted from the International Journal of Emerging Technologies in Learning. Both retraction statements, issued in July 2022, use the same language: “The work could be linked to a criminal paper mill selling authorships and articles for publication.”
As evidence, the statements cited the work of Brian Perron — who studies social work at the University of Michigan in Ann Arbor and also works as a misconduct sleuth — and his colleagues, who had found links between both of the retracted papers and International Publisher. Neither Babaskin nor Degtyarevskaya responded to Nature’s requests for comment about the retractions.
Publishers unite to tackle doctored images in research papers
International Publisher’s website advertises a selection of more than 10,000 manuscripts, on topics as diverse as the metallurgy of aluminium-alloy welding and the biological features of quails. Prospective buyers can see the paper’s title, and sometimes its abstracts, as well as the expected ranking in the citation database Scopus of the journal of publication. They then select an author slot, with costs ranging from about US$500 to $3,000. The company promises that titles and abstracts shown online will be “completely changed” for publication. “No one will ever be able to find the manuscript anywhere,” the website declares.
Nevertheless, in 2021, Perron and his colleagues reported on the scientific-fraud watchdog website Retraction Watch that they had identified nearly 200 published papers that probably originated from International Publisher. A number of the published titles “were almost word-for-word” the same as those listed for sale, Perron says. Many of the papers listed in the Retraction Watch report were later retracted. Asked for comment on allegations that it is a paper mill, International Publisher did not respond.
Clearing the catalogue
International Publisher removes paper listings from its online catalogue after papers are purchased. To get around this, Nature examined a database of past International Publisher paper listings, created by Perron, and combed through screenshots of the paper mill’s website taken by the non-profit organization Internet Archive, based in San Francisco, California. The search showed that the titles of multiple articles published by four of the five authors of the Proteomics study matched the titles of papers previously listed for sale by International Publisher.
Biomedical paper retractions have quadrupled in 20 years — why?
These paper listings do not include the full article text, but strong circumstantial evidence connects the paper mill’s listings to published studies. For example, a screenshot of the paper mill’s website taken in September 2021 shows that among the articles for sale was #1584, “The structure of forest vegetation on industrial dumps of different ages.” Degtyarevskaya was an author of a paper published in Ecology and Evolution5in July 2023 with a nearly identical title and matching abstract. In response to an enquiry from the news team, Ecology and Evolution said that it is now investigating the matter.
Although Nature’s news team was unable to locate a sales listing on International Publisher’s website for the Proteomics paper, Perron says that the paper has several hallmarks of paper-mill articles. Nature could not find any other studies published by the authors on the paper’s subject matter, protein analysis. Moreover, the manuscript was submitted to BioSystems while it was still under review at Proteomics. Perron says that submitting a manuscript to more than one journal simultaneously is a classic tactic of researchers trying to publish paper-mill products.
A spokesperson for Wiley did not specify whether the allegedly plagiarized Proteomics paper came from a paper mill, but said: “Our investigation confirmed that systematic manipulation of the publication process was at play.”
Check and check again
In recent years, some publishers and journals have taken extra countermeasures against plagiarism and paper mills. One such effort, developed by the International Association of Scientific, Technical and Medical Publishers (STM), a trade organization in The Hague, the Netherlands, is the STM Integrity Hub, a resource for scientific publishers that includes a ‘paper mill checker tool’ and ‘duplicate submission checker tool’. The latter is in use at more than 150 journals and scans more than 20,000 papers each month. More than 1% are identified as duplicates.
There are no metrics for how often researchers spot plagiarism of their own work, but several researchers responded to Payne’s social-media posts by sharing that they had found themselves in a similar situation.
For Payne, the prospect of paper mills taking advantage of AI is a daunting one. “This, I think, is a pretty good con,” he says. “I think it’s going to happen more.”
Studies that try to replicate the findings of published research are hard to come by: it can be difficult to find funders to support them and journals to publish them. And when these papers do get published, it’s not easy to locate them, because they are rarely linked to the original studies.
A database described in a preprint posted in April1 aims to address these issues by hosting replication studies from the social sciences and making them more traceable and discoverable. It was launched as part of the Framework for Open and Reproducible Research Training (FORTT), a community-driven initiative that teaches principles of open science and reproducibility to researchers.
The initiative follows other efforts to improve the accessibility of replication work in science, such as the Institute for Replication, which hosts a database listing studies published in selected economics and politics journals that academics can choose to replicate.
The team behind the FORTT database hopes that it will draw more attention to replication studies, which it argues is a fundamental part of science. The database can be accessed through the web application Shiny, and will soon be available on the FORTT website.
Nature Index spoke to one of the project’s leaders, Lukas Röseler, a metascience researcher and director of the University of Münster’s Center for Open Science in Germany.
Why did you create this database?
We’re trying to make it easier for researchers to make their replication attempts public, because it’s often difficult to publish them, regardless of their outcome.
We also wanted to make it easier to track replication studies. If you’re building on previous research and want to check whether replication studies have already been done, it’s often difficult to find them, partly because journals tend to not link them to the original work.
We started out with psychology, which has been hit hard by the replication crisis, and have branched out to studies in judgement and decision-making, marketing and medicine. We are now looking into other fields to understand how their researchers conduct replication studies and what replication means in those contexts.
Who might want to use the database?
A mentor of mine wrote a textbook on social psychology and said that he had no easy way of screening his 50 pages of references for replication attempts. Now, he can enter his references into our database and check which studies have been replicated.
The database can also be used to determine the effectiveness of certain procedures by tracking the replication history of studies. Nowadays, for instance, academics are expected to pre-register their studies — publishing their research design, hypotheses and analysis plans before conducting the study — and make their data freely available online. We would like to empirically see whether interventions such as these affect how likely a study is to be replicable.
2024 Research Leaders
How is the database updated?
It is currently an online spreadsheet, which we created by manually adding the original findings, their replication studies and their outcomes. So far, we have more than 3,300 entries — or replication findings — of just under 1,100 original studies. There are often multiple findings in one study; a replication study might include attempts to replicate four different findings, constituting four entries.
There are hundreds of volunteers who are collecting replications and logging studies on the spreadsheet. You can either just enter a study so that it’s findable, or include both the original study and the replication findings.
We are in contact with teams that conduct a lot of replication research, and we regularly issue calls for people to add their studies. This is a crowdsourced effort and a large proportion of it is based on the FORTT replications and reverses project, which is also crowdsourced. It aims to collate replications and ‘reversal effects’ in social science, in which replication attempts have results in the opposite direction compared with the original.
Do you plan to automate this process?
We are absolutely looking into ways to automate this. For instance, we are working on a machine-readable manuscript template, in which people can enter their manuscript and have it automatically read into the database.
We have code that automatically recognizes DOIs and cross checks them with all the original studies in the database to check whether there is a match. We are working on turning this into a search engine, but it’s beyond our capabilities and resources at the moment.
Does your database provide any data on the replications it hosts?
If you go to our website, there is a replication tracker, where you can see the percentage of studies that were able to replicate original findings, and those that failed to do so.
In a version of the database that we will launch in the coming months, users will be able to choose the criteria by which they judge whether a study successfully replicated the original findings. Right now, it’s all based on how strong the effect sizes — a measure of the relationship between two variables — were on both the original study and the replication attempts, but there are many other criteria and metrics of replication success that we are considering.
We’re also planning to launch a peer-reviewed, open-access journal at FORTT to publish replication studies from various disciplines.
This interview has been edited for length and clarity.
Nature Index’s news and supplement content is editorially independent of its publisher, Springer Nature. For more information about Nature Index, see the homepage.
The definition of copyright in an age of artificial-intelligence tools is an open question, both legally and morally.Credit: Getty
No one knows for sure exactly what ChatGPT — the most famous product of artificial intelligence — and similar tools were trained on. But millions of academic papers scraped from the web are among the reams of data that have been fed into large language models (LLMs) that generate text, and similar algorithms that make images (see Nature632, 715–716; 2024). Should the creators of such training data get credit — and if so, how? There is an urgent need for more clarity around the boundaries of acceptable use.
Few LLMs — even those described as ‘open’ — have developers who are upfront about exactly which data were used for training. But information-rich, long-form text, a category that includes many scientific papers, is particularly valuable. According to an investigation by The Washington Post and the Allen Institute for Artificial Intelligence in Seattle, Washington, material from the open-access journal families PLOS and Frontiers features prominently in a data set called C4, which has been used to train LLMs such as Llama, made by the technology giant Meta. It is also widely suspected that, just as copyrighted books have been used to train LLMs, so have non-open-access research papers.
Science and the new age of AI: a Nature special
One fundamental question concerns what is allowed under current laws. The World Intellectual Property Organization (WIPO), based in Geneva, Switzerland, says that it is unclear whether collecting data or using them to create LLM outputs is considered copyright infringement, or whether these activities fall under one of several exemptions, which differ by jurisdiction. Some publishers are seeking clarity in the courts: in an ongoing case, The New York Times has alleged that the tech firms Microsoft and OpenAI — the company that developed ChatGPT — copied its articles to train their LLMs. To avoid the risk of litigation, more AI firms are now, as recommended by WIPO, purchasing licences from copyright holders for training data. Content owners are also using code on their websites that tells tools scraping data for LLMs whether they are allowed to do so.
Things get much fuzzier when material is published under licences that encourage free distribution and reuse, but that can still have certain restrictions. Creative Commons, a non-profit organization in Mountain View, California, that aims to increase sharing of creative works, says that copying material to train an AI should not generally be treated as infringement. But it also acknowledges concerns about the impact of AI on creators, and how to ensure that AI that is trained on ‘the commons’ — the body of freely available material — contributes to the commons in return.
These broader questions of fairness are particularly pressing for artists, writers and coders, whose livelihoods depend on their creative outputs and whose work risks being replaced by the products of generative AI. But they are also highly relevant for researchers. The move towards open-access publishing explicitly favours the free distribution and reuse of scientific work — and this presumably applies to LLMs, too. Learning from scientific papers can make LLMs better, and some researchers might rejoice if improved AI models could help them to gain new insights.
Credit where it is due
But others are worried about principles such as attribution, the currency by which science operates. Fair attribution is a condition of reuse under CC BY, a commonly used open-access copyright license. In jurisdictions such as the European Union and Japan, there are exemptions to copyright rules that cover factors such as attribution — for text and data mining in research using automated analysis of sources to find patterns, for example. Some scientists see LLM data-scraping for proprietary LLMs as going well beyond what these exemptions were intended to achieve.
Has your paper been used to train an AI model? Almost certainly
In any case, attribution is impossible when a large commercial LLM uses millions of sources to generate a given output. But when developers create AI tools for use in science, a method known as retrieval-augmented generation could help. This technique doesn’t apportion credit to the data that trained the LLM, but does allow the model to cite papers that are relevant to its output, says Lucy Lu Wang, an AI researcher at the University of Washington in Seattle.
Giving researchers the ability to opt out of having their work used in LLM training could also ease their worries. Creators have this right under EU law, but it is tough to enforce in practice, says Yaniv Benhamou, who studies digital law and copyright at the University of Geneva. Firms are devising innovative ways to make it easier. Spawning, a start-up company in Minneapolis, Minnesota, has developed tools to allow creators to opt out of data scraping. Some developers are also getting on board: OpenAI’s Media Manager tool, for example, allows creators to specify how their works can be used by machine-learning algorithms.
Greater transparency can also play a part. The EU’s AI Act, which came into force on 1 August, requires developers to publish a summary of the works used to train their AI models. This could bolster creators’ ability to opt out, and might serve as a template for other jurisdictions. But it remains to be seen how this will work in practice.
Meanwhile, research should continue into whether there is a need for more-radical solutions, such as new kinds of licence or changes to copyright law. Generative AI tools are using a data ecosystem built by open-source movements, yet often ignore the accompanying expectations of reciprocity and reasonable use, says Sylvie Delacroix, a digital-law scholar at King’s College London. The tools also risk polluting the Internet with AI-generated content of dubious quality. By failing to redirect users to the human-made sources on which they were built, LLMs could disincentivize original creation. Without putting more power into the hands of creators, the system will come under severe strain. Regulators and companies must act.
