Tag: Publishing

fresh incentives for reporting negative results

[ad_1]

The editor-in-chief of the Journal of Trial & Error, Sarahanne Field wants to publish the messy, null and negative results sitting in researchers’ file drawers.Credit: Sander Martens

Editor-in-chief Sarahanne Field describes herself and her team at the Journal of Trial & Error as wanting to highlight the “ugly side of science — the parts of the process that have gone wrong”.

She clarifies that the editorial board of the journal, which launched in 2020, isn’t interested in papers in which “you did a shitty study and you found nothing. We’re interested in stuff that was done methodologically soundly, but still yielded a result that was unexpected.” These types of result — which do not prove a hypothesis or could yield unexplained outcomes — often simply go unpublished, explains Field, who is also an open-science researcher at the University of Groningen in the Netherlands. Along with Stefan Gaillard, one of the journal’s founders, she hopes to change that.

Calls for researchers to publish failed studies are not new. The ‘file-drawer problem’ — the stacks of unpublished, negative results that most researchers accumulate — was first described in 1979 by psychologist Robert Rosenthal. He argued that this leads to publication bias in the scientific record: the gap of missing unsuccessful results leads to overemphasis on the positive results that do get published.

Careers Collection: Publishing

Over the past 30 years, the proportion of negative results being published has decreased further. A 2012 study showed that, from 1990 to 2007, there was a 22% increase in positive conclusions in papers; by 2007, 85% of papers published had positive results¹. “People fail to report [negative] results, because they know they won’t get published — and when people do attempt to publish them, they get rejected,” says Field. A 2022 survey of researchers in France in chemistry, physics, engineering and environmental sciences showed that, although 81% had produced relevant negative results and 75% were willing to publish them, only 12.5% had the opportunity to do so².

One factor that is leading some researchers to revisit the problem is the growing use of predictive modelling using machine-learning tools in many fields. These tools are trained on large data sets that are often derived from published work, and scientists have found that the absence of negative data in the literature is hampering the process. Without a concerted effort to publish more negative results that artificial intelligence (AI) can be trained on, the promise of the technology could be stifled.

“Machine learning is changing how we think about data,” says chemist Keisuke Takahashi at Hokkaido University in Japan, who has brought the issue to the attention of the catalysis-research community. Scientists in the field have typically relied on a mixture of trial and error and serendipity in their experiments, but there is hope that AI could provide a new route for catalyst discovery. Takahashi and his colleagues mined data from 1,866 previous studies and patents to train a machine-learning model to predict the best catalyst for the reaction between methane and oxygen to form ethane and ethylene, both of which are important chemicals used in industry³. But, he says, “over the years, people have only collected the good data — if they fail, they don’t report it”. This led to a skewed model that, in some cases, enhanced the predicted performance of a material, rather than realistically assessing its properties.

Synthetic organic chemist Felix Strieth-Kalthoff found that published data were too heavily biased toward positive results to effectively train an AI model to optimize chemical reaction yields.Credit: Cindy Huang

Alongside the flawed training of AI models, the huge gap of negative results in the scientific record continues to be a problem across all disciplines. In areas such as psychology and medicine, publication bias is one factor exacerbating the ongoing reproducibility crisis — in which many published studies are impossible to replicate. Without sharing negative studies and data, researchers could be doomed to repeat work that led nowhere. Many scientists are calling for changes in academic culture and practice — be it the creation of repositories that include positive and negative data, new publication formats or conferences aimed at discussing failure. The solutions are varied, but the message is the same: “To convey an accurate picture of the scientific process, then at least one of the components should be communicating all the results, [including] some negative results,” says Gaillard, “and even where you don’t end up with results, where it just goes wrong.”

Science’s messy side

Synthetic organic chemist Felix Strieth-Kalthoff, who is now setting up his own laboratory at the University of Wuppertal, Germany, has encountered positive-result bias when using data-driven approaches to optimize the yields of certain medicinal-chemistry reactions. His PhD work with chemist Frank Glorius at the University of Münster, Germany, involved creating models that could predict which reactants and conditions would maximize yields. Initially, he relied on data sets that he had generated from high-throughput experiments in the lab, which included results from both high- and low-yield reactions, to train his AI model. “Our next logical step was to do that based on the literature,” says Strieth-Kalthoff. This would allow him to curate a much larger data set to be used for training.

But when he incorporated real data from the reactions database Reaxys into the training process, he says, “[it] turned out they don’t really work at all”. Strieth-Kalthoff concluded the errors were due the lack of low-yield reactions⁴; “All of the data that we see in the literature have average yields of 60–80%.” Without learning from the messy ‘failed’ experiments with low yields that were present in the initial real-life data, the AI could not model realistic reaction outcomes.

Although AI has the potential to spot relationships in complex data that a researcher might not see, encountering negative results can give experimentalists a gut feeling, says molecular modeller Berend Smit at the Swiss Federal Institute of Technology Lausanne. The usual failures that every chemist experiences at the bench give them a ‘chemical intuition’ that AI models trained only on successful data lack.

Smit and his team attempted to embed something similar to this human intuition into a model tasked with designing a metal-organic framework (MOF) with the largest known surface area for this type of material. A large surface area allows these porous materials to be used as reaction supports or molecular storage reservoirs. “If the binding [between components] is too strong, it becomes amorphous; if the binding is too weak, it becomes unstable, so you need to find the sweet spot,” Smit says. He showed that training the machine-learning model on both successful and unsuccessful reaction conditions created better predictions and ultimately led to one that successfully optimized the MOF⁵. “When we saw the results, we thought, ‘Wow, this is the chemical intuition we’re talking about!’” he says.

According to Strieth-Kalthoff, AI models are currently limited because “the data that are out there just do not reflect all of our knowledge”. Some researchers have sought statistical solutions to fill the negative-data gap. Techniques include oversampling, which means supplementing data with several copies of existing negative data or creating artificial data points, for example by including reactions with a yield of zero. But, he says, these types of approach can introduce their own biases.

Computer scientist Ella Peltonen helped to organize the first International Workshop on Negative Results in Pervasive Computing in 2022 to give researchers an opportunity to discuss failed experiments.Credit: University of Oulu

Capturing more negative data is now a priority for Takahashi. “We definitely need some sort of infrastructure to share the data freely.” His group has created a website for sharing large amounts of experimental data for catalysis reactions. Other organizations are trying to collect and publish negative data — but Takahashi says that, so far, they lack coordination, so data formats aren’t standardized. In his field, Strieth-Kalthoff says, there are initiatives such as the Open Reaction Database, launched in 2021 to share organic-reaction data and enable training of machine-learning applications. But, he says, “right now, nobody’s using it, [because] there’s no incentive”.

Smit has argued for a modular open-science platform that would directly link to electronic lab notebooks to help to make different data types extractable and reusable. Through this process, publication of negative data in peer-reviewed journals could be skipped, but the information would still be available for researchers to use in AI training. Strieth-Kalthoff agrees with this strategy in theory, but thinks it’s a long way off in practice, because it would require analytical instruments to be coupled to a third-party source to automatically collect data — which instrument manufacturers might not agree to, he says.

Publishing the non-positive

In other disciplines, the emphasis is still on peer-reviewed journals that will publish negative results. Gaillard, a science-studies PhD student at Radboud University in Nijmegen, the Netherlands, co-founded the Journal of Trial & Error after attending talks on how science can be made more open. Gaillard says that, although everyone whom they approached liked the idea of the journal, nobody wanted to submit articles at first. He and the founding editorial team embarked on a campaign involving cold calls and publicity at open-science conferences. “Slowly, we started getting our first submissions, and now we just get people sending things in [unsolicited],” he says. Most years the journal publishes one issue of about 8–14 articles, and it is starting to publish more special issues. It focuses mainly on the life sciences and data-based social sciences.

In 2008, David Alcantara, then a chemistry PhD student at the University of Seville in Spain who was frustrated by the lack of platforms for sharing negative results, set up The All Results journals, which were aimed at disseminating results regardless of the outcome. Of the four disciplines included at launch, only the biology journal is still being published. “Attracting submissions has always posed a challenge,” says Alcantara, now president at the consultancy and training organization the Society for the Improvement of Science in Seville.

But Alcantara thinks there has been a shift in attitudes: “More established journals [are] becoming increasingly open to considering negative results for publication.” Gaillard agrees: “I’ve seen more and more journals, like PLoS ONE, for example, that explicitly mentioned that they also publish negative results.” (Nature welcomes submissions of replication studies and those that include null results, as described in this 2020 editorial.)

Journals might be changing their publication preferences, but there are still significant disincentives that stop researchers from publishing their file-drawer studies. “The current academic system often prioritizes high-impact publications and ground-breaking discoveries for career advancement, grants and tenure,” says Alcantara, noting that negative results are perceived as contributing little to nothing to these endeavours. Plus, there is still a stigma associated with any kind of failure. “People are afraid that this will look negative on their CV,” says Gaillard. Smit describes reporting failed experiments as a no-win situation: “It’s more work for [researchers], and they don’t get anything in return in the short term.” And, jokes Smit, what’s worse is that they could be providing data for an AI tool to take over their role.

Ultimately, most researchers conclude that publishing their failed studies and negative data is just not worth the time and effort — and there’s evidence that they judge others’ negative research more harshly than positive outcomes. In a study published in August, 500 researchers from top economics departments around the world were randomized to two groups and asked to judge a hypothetical research paper. Half of the participants were told that the study had a null conclusion, and the other half were told the results were sizeably significant. The null results were perceived to be 25% less likely to be published, of lower quality and less important than were the statistically significant findings⁶.

Some researchers have had positive experiences sharing their unsuccessful findings. For example, in 2021, psychologist Wendy Ross at the London Metropolitan University published her negative results from testing a hypothesis about human problem-solving in the Journal of Trial & Error⁷, and says the paper was “the best one I have published to date”. She adds, “Understanding the reasons for null results can really test and expand our theoretical understanding.”

Fields forging solutions

The field of psychology has introduced one innovation that could change publication biases — registered reports (RRs). These peer-reviewed reports, first published in 2014, came about largely as a response to psychology’s replication crisis, which began in around 2011. RRs set out the methodology of a study before the results are known, to try to prevent selective reporting of positive results. Daniël Lakens, who studies science-reward structures at Eindhoven University of Technology in the Netherlands, says there is evidence that RRs increase the proportion of negative results in the psychology literature.

In a 2021 study, Lakens analysed the proportion of published RRs whose results eventually support the primary hypothesis. In a random sample of hypothesis-testing studies from the standard psychology literature, 96% of the results were positive. In RRs, this fell to only 44%⁸. Lakens says the study shows “that if you offer this as an option, many more null results enter the scientific literature, and that is a desirable thing”. At least 300 journals, including Nature, are now accepting RRs, and the format is spreading to journals in biology, medicine and some social-science fields.

Yet another approach has emerged from the field of pervasive computing, the study of how computer systems are integrated into physical surroundings and everyday life. About four years ago, members of the community started discussing reproducibility, says computer scientist Ella Peltonen at the University of Oulu in Finland. Peltonen says that researchers realized that, to avoid the repetition of mistakes, there was a need to discuss the practical problems with studies and failed results that don’t get published. So in 2022, Peltonen and her colleagues held the first virtual International Workshop on Negative Results in Pervasive Computing (PerFail), in conjunction with the field’s annual conference, the International Conference on Pervasive Computing and Communications.

Peltonen explains that PerFail speakers first present their negative results and then have the same amount of time for discussion afterwards, during which participants tease out how failed studies can inform future work. “It also encourages the community to showcase that things require effort and trial and error, and there is value in that,” she adds. Now an annual event, the organizers invite students to attend so they can see that failure is a part of research and that “you are not a bad researcher because you fail”, says Peltonen.

In the long run, Alcantara thinks a continued effort to persuade scientists to share all their results needs to be coupled with policies at funding agencies and journals that reward full transparency. “Criteria for grants, promotions and tenure should recognize the value of comprehensive research dissemination, including failures and negative outcomes,” he says. Lakens thinks funders could be key to boosting the RR format, as well. Funders, he adds, should say, “We want the research that we’re funding to appear in the scientific literature, regardless of the significance of the finding.”

There are some positive signs of change about sharing negative data: “Early-career researchers and the next generation of scientists are particularly receptive to the idea,” says Alcantara. Gaillard is also optimistic, given the increased interest in his journal, including submissions for an upcoming special issue on mistakes in the medical domain. “It is slow, of course, but science is a bit slow.”

[ad_2]

Source link

May 8, 2024
Plagiarism in peer-review reports could be the ‘tip of the iceberg’

[ad_1]

Mikołaj Piniewski is a researcher to whom PhD students and collaborators turn when they need to revise or refine a manuscript. The hydrologist, at the Warsaw University of Life Sciences, has a keen eye for problems in text — a skill that came in handy last year when he encountered some suspicious writing in peer-review reports of his own paper.

Last May, when Piniewski was reading the peer-review feedback that he and his co-authors had received for a manuscript they’d submitted to an environmental-science journal, alarm bells started ringing in his head. Comments by two of the three reviewers were vague and lacked substance, so Piniewski decided to run a Google search, looking at specific phrases and quotes the reviewers had used.

To his surprise, he found the comments were identical to those that were already available on the Internet, in multiple open-access review reports from publishers such as MDPI and PLOS. “I was speechless,” says Piniewski. The revelation caused him to go back to another manuscript that he had submitted a few months earlier, and dig out the peer-review reports he received for that. He found more plagiarized text. After e-mailing several collaborators, he assembled a team to dig deeper.

Meet this super-spotter of duplicated images in science papers

The team published the results of its investigation in Scientometrics in February¹, examining dozens of cases of apparent plagiarism in peer-review reports, identifying the use of identical phrases across reports prepared for 19 journals. The team discovered exact quotes duplicated across 50 publications, saying that the findings are just “the tip of the iceberg” when it comes to misconduct in the peer-review system.

Dorothy Bishop, a former neuroscientist at the University of Oxford, UK, who has turned her attention to investigating research misconduct, was “favourably impressed” by the team’s analysis. “I felt the way they approached it was quite useful and might be a guide for other people trying to pin this stuff down,” she says.

Peer review under review

Piniewski and his colleagues conducted three analyses. First, they uploaded five peer-review reports from the two manuscripts that his laboratory had submitted to a rudimentary online plagiarism-detection tool. The reports had 44–100% similarity to previously published online content. Links were provided to the sources in which duplications were found.

The researchers drilled down further. They broke one of the suspicious peer-review reports down to fragments of one to three sentences each and searched for them on Google. In seconds, the search engine returned a number of hits: the exact phrases appeared in 22 open peer-review reports, published between 2021 and 2023.

The final analysis provided the most worrying results. They took a single quote — 43 words long and featuring multiple language errors, including incorrect capitalization — and pasted it into Google. The search revealed that the quote, or variants of it, had been used in 50 peer-review reports.

Predominantly, these reports were from journals published by MDPI, PLOS and Elsevier, and the team found that the amount of duplication increased year-on-year between 2021 and 2023. Whether this is because of an increase in the number of open-access peer-review reports during this time or an indication of a growing problem is unclear — but Piniewski thinks that it could be a little bit of both.

Why would a peer reviewer use plagiarized text in their report? The team says that some might be attempting to save time, whereas others could be motivated by a lack of confidence in their writing ability, for example, if they aren’t fluent in English.

The team notes that there are instances that might not represent misconduct. “A tolerable rephrasing of your own words from a different review? I think that’s fine,” says Piniewski. “But I imagine that most of these cases we found are actually something else.”

The source of the problem

Duplication and manipulation of peer-review reports is not a new phenomenon. “I think it’s now increasingly recognized that the manipulation of the peer-review process, which was recognized around 2010, was probably an indication of paper mills operating at that point,” says Jennifer Byrne, director of biobanking at New South Wales Health in Sydney, Australia, who also studies research integrity in scientific literature.

Paper mills — organizations that churn out fake research papers and sell authorships to turn a profit — have been known to tamper with reviews to push manuscripts through to publication, says Byrne.

The fight against fake-paper factories that churn out sham science

However, when Bishop looked at Piniewski’s case, she could not find any overt evidence of paper-mill activity. Rather, she suspects that journal editors might be involved in cases of peer-review-report duplication and suggests studying the track records of those who’ve allowed inadequate or plagiarized reports to proliferate.

Piniewski’s team is also concerned about the rise of duplications as generative artificial intelligence (AI) becomes easier to access. Although his team didn’t look for signs of AI use, its ability to quickly ingest and rephrase large swathes of text is seen as an emerging issue.

A preprint posted in March² showed evidence of researchers using AI chatbots to assist with peer review, identifying specific adjectives that could be hallmarks of AI-written text in peer-review reports.

Bishop isn’t as concerned as Piniewski about AI-generated reports, saying that it’s easy to distinguish between AI-generated text and legitimate reviewer commentary. “The beautiful thing about peer review,” she says, is that it is “one thing you couldn’t do a credible job with AI”.

Preventing plagiarism

Publishers seem to be taking action. Bethany Baker, a media-relations manager at PLOS, who is based in Cambridge, UK, told Nature Index that the PLOS Publication Ethics team “is investigating the concerns raised in the Scientometrics article about potential plagiarism in peer reviews”.

How big is science’s fake-paper problem?

An Elsevier representative told Nature Index that the publisher “can confirm that this matter has been brought to our attention and we are conducting an investigation”.

In a statement, the MDPI Research Integrity and Publication Ethics Team said that it has been made aware of potential misconduct by reviewers in its journals and is “actively addressing and investigating this issue”. It did not confirm whether this was related to the Scientometrics article.

One proposed solution to the problem is ensuring that all submitted reviews are checked using plagiarism-detection software. In 2022, exploratory work by Adam Day, a data scientist at Sage Publications, based in Thousand Oaks, California, identified duplicated text in peer-review reports that might be suggestive of paper-mill activity. Day offered a similar solution of using anti-plagiarism software, such as Turnitin.

Piniewski expects the problem to get worse in the coming years, but he hasn’t received any unusual peer-review reports since those that originally sparked his research. Still, he says that he’s now even more vigilant. “If something unusual occurs, I will spot it.”

[ad_2]

Source link

May 1, 2024
How reliable is this research? Tool flags papers discussed on PubPeer

[ad_1]

RedacTek’s tool alerts users to PubPeer discussions, and indicates when a study, or the papers that it cites, has been retracted.Credit: deepblue4you/Getty

A free online tool released earlier this month alerts researchers when a paper cites studies that are mentioned on the website PubPeer, a forum scientists often use to raise integrity concerns surrounding published papers.

Studies are usually flagged on PubPeer when readers have suspicions, for example about image manipulation, plagiarism, data fabrication or artificial intelligence (AI)-generated text. PubPeer already offers its own browser plug-in that alerts users when a study that they are reading has been posted on the site. The new tool, a plug-in released on 13 April by RedacTek, based in Oakland, California, goes further — it searches through reference lists for papers that have been flagged. The software pulls information from many sources, including PubPeer’s database; data from the digital-infrastructure organization Crossref, which assigns digital object identifiers to articles; and OpenAlex, a free index of hundreds of millions of scientific documents.

It’s important to track mentions of referenced articles on PubPeer, says Jodi Schneider, an information scientist at the University of Illinois Urbana-Champaign, who has tried out the RedacTek plug-in. “Not every single reference that’s in the bibliography matters, but some of them do,” she adds. “When you see a large number of problems in somebody’s bibliography, that just calls everything into question.”

The aim of the tool is to flag potential problems with studies to researchers early on, to reduce the circulation of poor-quality science, says RedacTek founder Rick Meyler, based in Emeryville, California. Future versions might also use AI to automatically clarify whether the PubPeer comments on a paper are positive or negative, he adds.

Third-generation retractions

As well as flagging PubPeer discussions, the plug-in indicates when a study, or the papers that it cites, has been retracted. There are existing tools that alert academics about retracted citations; some can do this during the writing process, so that researchers are aware of the publication status of studies when constructing bibliographies. But with the new tool, users can opt in to receive notifications about further ‘generations’ of retractions — alerts cover not only the study that they are reading, but also the papers it cites, articles cited by those references and even papers cited by the secondary references.

The software also calculates a ‘retraction association value’ for studies, a metric that measures the extent to which the paper is associated with science that has been withdrawn from the literature. As well as informing individual researchers, the plug-in could help scholarly publishers to keep tabs on their own journals, Meyler says, because it allows users to filter by publication.

In its ‘paper scorecard’, the tool also flags any papers in the three generations of referenced studies in which more than 25% of papers in the bibliography are self-citations — references by authors to their previous works.

Future versions could highlight whether papers cited retracted studies before or after the retraction was issued, notes Meyler, or whether mentions of such studies acknowledge the retraction. That would be useful, says Schneider, who co-authored a 2020 analysis that found that as little as 4% of citations to retracted studies note that the referenced paper has been retracted¹.

Meyler says that RedacTek is currently in talks with scholarly-services firm Cabell’s International in Beaumont, Texas, which maintains pay-to-view lists of suspected predatory journals, which publish articles without proper quality checks for issues such as plagiarism but still collect authors’ fees. The plan is to use these lists to improve the tool so that it can also automatically flag any cited papers that are published in such journals.

[ad_2]

Source link

April 29, 2024
Algorithm ranks peer reviewers by reputation — but critics warn of bias

[ad_1]

An algorithm ranks the reputation of peer reviewers on the basis of how many citations the studies they have reviewed attracted.

The tool, outlined in a study published in February¹, could help to identify which papers could become high impact during peer review, its creators say. They add that, during peer review, authors should put the most weight on the recommendations and feedback from reviewers of previous papers that have been highly cited.

The study authors extracted citation data from 308,243 papers published by journals of the American Physical Society (APS) between 1990 and 2010 that had accumulated more than 5 citations each. Information about the referees of these papers was not available, so the authors used an algorithm to create imaginary reviewers, which rated papers on the basis of an algorithm that was trained on citation data from the APS data set. Using the review scores that these papers received in real life (a score of 1 being poor and 5 being outstanding), the study authors compared how closely the imaginary reviewers’ scores correlated to the actual scores the papers received.

To rank the imaginary reviewers, the study authors tracked the citations accumulated by the papers published between 1990 and 2000 and checked the review scores they were given. Imaginary reviewers that gave high review scores to papers that went on to attract a high number of citations were given a high ranking.

The authors then tested how effective these reputation rankings were in predicting citation numbers of papers refereed by the same imaginary reviewers in the second decade of the data. The study found that the imaginary reviewers’ recommendations on the 2000–10 papers were in line with the actual citation counts of these papers over that time span, says study co-author An Zeng, an environmental scientist at Beijing Normal University. This suggests that the algorithm is good at predicting high-impact papers, he adds.

More eyes on peer reviewers

Previous attempts to quantify and predict the reach of studies have been widely criticized for relying too heavily on citation-based metrics, which, critics say, exacerbate existing biases in academia. A 2021 study² found that non-replicable papers are cited more than replicable studies, possibly because they have more ‘interesting’ results.

Zeng acknowledges the limitations of focusing on citation metrics, but says that it’s important to evaluate the work of peer reviewers. Solid studies are sometimes rejected because of one negative review, he notes, but there’s little attention given to how professional or reliable that reviewer is. “If this algorithm can identify reliable reviewers, it will give less weight to the reviewers who are not so reliable,” says Zeng.

Journal editors often use search tools to identify candidates to peer review papers, but they have to manually decide who to contact. If referee activities were ranked and quantified, this would make it easier for journal editors to choose, Zeng points out.

However, ranking reviewers on their reputation is likely to exacerbate the inequities and biases that exist in peer review, says Anita Bandrowski, an information scientist at the University of California, San Diego.

As previous data have shown, most of the responsibility of the peer-review process in science falls to a small subset of peer reviewers — typically men in senior positions in high-income nations that are geographically closer to most journal editors.

Bandrowski notes that the algorithm might favour those with a long history of reviewing, because they’ve had more time to accumulate citations on their refereed papers. “The oldest reviewers by this metric would be the best reviewers and yet the oldest reviewers are going to be retired or dead,” she says.

Zeng disagrees that his approach will make the selection of peer reviewers more inequitable than it is now. After implementing the reputation ranking, editors might find that some reviewers who are not frequently invited have high reputation scores — in some cases better than those who are inundated with referee requests, he says.

Capturing the nuance

Laura Feetham-Walker, a reviewer-engagement manager at the Institute of Physics Publishing in Bristol, UK, worries that the algorithm might not account for incremental studies, negative findings and replications of previous studies, all of which are crucial for science, albeit often not highly cited.

“Under their system, a reviewer who gave a favourable recommendation on an incremental study — for example, for a journal that does not have novelty as an editorial criterion — would go down in the reviewer reputation ranking, simply because that manuscript would be unlikely to accrue large numbers of citations when published,” she says.

Neither does the ranking account for researchers who have never reviewed before, Feetham-Walker adds, or at least those who have never reviewed for a particular publisher.

“We know that a reviewer’s ability to provide a helpful review is dependent not just on their expertise, but also their availability and interest in the subject matter. We also know that reviewers are human, and their reviewing behaviour can change over time depending on various factors,” Feetham-Walker says. “A nuanced algorithm that took all of this into account, as well as adding new reviewers to enrich the pool, would be of genuine value to publishers.”

[ad_2]

Source link

April 25, 2024
Scientists urged to collect royalties from the ‘magic money tree’

[ad_1]

Credit: Richard Drury/Getty

“Lots of our members call us ‘the magic money tree’,” says Alison Baxter, head of communications for the Authors’ Licensing and Collecting Society (ALCS), based in the United Kingdom. “They don’t really understand where the payments come from,” she says, “but they like getting them.” The ALCS is one of a global group of collecting societies and agencies that compensate authors when their works are copied or shared after publication. This year, the ALCS says, it is due to pay out more than £45 million (US$56 million) and, when money is earned from the use of academic textbooks and research papers, the copyright holders who stand to benefit are often scientists.

ALCS members who claimed for journal or magazine articles this year received around £450 each, on average. Although the sums any individual author is entitled to could be much less than that, the fact remains that researchers who are not members might be missing out on their share.

Collecting societies vary in their exact function, but their common goal is to ensure that authors are remunerated when, for example, a company prints out part of a book to circulate among its staff, or a research paper is printed out and distributed to students. The societies generate income by selling licences that give blanket permission to reproduce copy-righted material, or by gathering payments for the use of specific works.

They then share that money among their members on the basis of which activities generated the funds. Although this is a well-established source of income for many authors and journalists, among researchers there is less awareness of its existence. “My oldest friend is a scientist,” Baxter says, “and it took me a while to convince her that she could claim for her papers by joining the ALCS.”

One reason for a lack of take-up might be cynicism among scientists, and a misapprehension about fraud. “Everyone I have told thinks it’s a scam,” says Nicole Melzack, who is studying for a PhD in energy storage at the University of Southampton, UK, and has been a member of the ALCS since last year. “It’s really hard convincing people that it’s not, but, as long as you own the copyright, which I think most people will for their journal articles, then you have nothing to lose by signing up.”

Careers Collection: Publishing

For those who do, collecting societies can provide a welcome and regular cash flow that requires little or no effort to maintain. Yashar Mousavi, a senior analytical engineer at American Axle & Manufacturing, an automotive engineering firm based in Detroit, Michigan, joined the ALCS as a PhD student at Glasgow Caledonian University, UK, in 2020. “I’ve been paid twice so far, each time between £400 and £600, for papers published in the UK in the journal Chaos, Solitons & Fractals on the topic of fractional calculus and optimization,” he says. “The size of payment depends on many factors, such as the amount of money the ALCS has collected, the number of papers I have shared with them, the percentage of my contribution to the paper, and the journal’s impact factor.”

Even early-career researchers who do not have many publications can benefit, says Melzack. “In 2023, I had one paper published in Frontiers in Energy Research and made £464, and this year I published five papers and got £357,” they say. “It’s great, given the general cost of living and the fact that the academic publishing ecosystem involves so much unpaid labour, so to get something for a paper I’ve written feels validating in some way too.”

Nicole Melzack says that scientists who own the copyright in their publications have nothing to lose by joining a collecting society.Credit: Nicole Melzack

For scientists who publish outside academic journals, the rewards can be even greater. Isabel Thomas, a freelance science writer and children’s book author based in Cambridge, UK, joined the ALCS in 2014. “Since then I’ve had payments every six months, ranging from £77 to £8,000,” she says. “The ALCS also approached a friend of mine who writes practice exam papers and it turned out they were holding almost £30,000 due to her.”

How do collecting societies work?

Collecting societies might seem unusual in the context of academic research, but they are long established in other fields. For example, the Performing Right Society in the United Kingdom and the American Society of Composers, Authors and Publishers in the United States, both founded in 1914, collect fees for music played in public, then distribute the money to the composers and songwriters concerned. Other organizations ensure that artists and photographers are paid when their images are used.

The same principles apply to written work and authors. The ALCS was founded in 1977 by a group of writers who realized that photocopiers were enabling people to reproduce and share works without the creators being compensated. They also set up an accompanying body, the Copyright Licensing Agency (CLA), which collects money that the ALCS then distributes to writers.

The CLA sells and manages collective licences that give organizations the legal right to reproduce copyrighted works (whereas the ALCS handles payments to copyright holders). Baxter says that schools and universities, as well as the UK National Health Service and businesses, all pay the CLA for a licence. “That then means that their staff, students or users are allowed to copy sections of the books they own and share them, both physically and digitally.” The money generated is split between the publishers and authors.

Many other countries have similar collective licensing bodies. The Copyright Agency in Australia, the Indian Reprographic Rights Organisation, CADRA in Argentina and Canada’s Access Copyright all generate revenue through similar processes. The Copyright Clearance Center (CCC) is responsible for similar licences in the United States, but sends payments to publishers for distribution to authors.

Collecting societies also act as advocates and support networks. CADRA, for example, has been particularly successful in attracting researchers, who make up an estimated 40% of its members. Executive director Magdalena Iraizoz, who is based in Buenos Aires, says, “If a scientist has published work, being a member of CADRA not only gives them the benefit of receiving payments for the secondary uses of their works, but also free legal protection against piracy and illegal reproduction.”

How to collect payments

Anyone with publications to their name can join a collecting society and potentially receive payments. Baxter advises that scientists first determine what copyrights they own. “With books, authors aren’t generally asked to sign their copyright away,” she says. “In cases where they do have to, like when publishing in some academic journals, the contract can include a ‘quick clause’ that means the writer can still receive money from us.” These clauses can also apply to work that is published open access.

Authors then need to join the relevant collecting society. This will generally be one based in the country in which their work has been published, although many have reciprocal agreements that allow them to collect income generated overseas that is owed to their members. In its 2022–23 financial year, the Copyright Agency paid out Aus$142 million (US$92 million) to rights holders in Australia and elsewhere. “Most of our direct payments are to Australian writers, artists and publishers,” says a spokesperson for the Copyright Agency. “Most payments from copyright fees we collect for non-Australian works are made via copyright-management organizations similar to us in other countries.”

Science writer Isabel Thomas’s biannual collecting-society payments have ranged from £77 to £8,000.Credit: Elodie Guige

The majority of collecting societies are members of the International Federation of Reproduction Rights Organisations and are listed on its website. Generally, they do not ask individuals for a joining fee and instead take a small percentage from payments they distribute.

Scientists who join the ALCS can register any book with a unique International Standard Book Number (ISBN) product identifier, but only papers or articles published in the past three years in a journal with a UK-based International Standard Serial Number (ISSN) qualify. Baxter recommends that members list anything they think might be eligible. “For journals, we operate on a claim scheme,” she explains. “So, we gather money for a particular ISSN. Then we ask people to tell us what they’ve written and anyone who has contributed to that journal gets a share of the pot.”

CADRA operates in a similar way. “For a scientist to become a member of CADRA, they must have written work published with an Argentinian ISBN or ISSN,” says Iraizoz. “They must then sign the association contract and, once their incorporation is approved, they will be able to be part of the next distribution of rights.” Essentially, researchers list what they own and collecting societies will determine what they’re owed.

[ad_2]

Source link

April 25, 2024
Retractions are part of science, but misconduct isn’t — lessons from a superconductivity lab

[ad_1]

Superconductivity has been demonstrated at extremely low temperatures, but it remains elusive at room temperatures.Credit: Brookhaven National Laboratory/SPL

Research misconduct is hugely detrimental to science and to society. Defined as “fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results” by the US Office of Research Integrity, it violates trust in science and can do great harm to the wider public, scientific institutions and especially co-authors and students who had no part in the wrongdoing. In cases involving public funds, it squanders resources that could have been allocated to other research and it can erode lawmakers’ support for science.

Does the scientific community, as a whole, have appropriate processes for reporting, investigating and communicating about instances of potential misconduct? This question is not new. At Nature, we’re asking it again, after two separate studies that we published were subsequently retracted.

Exclusive: official investigation reveals how superconductivity physicist faked blockbuster results

The studies¹^,² were originally published in October 2020 and March 2023. The first was retracted in September 2022 and the second in November 2023. The corresponding author on both papers was Ranga Dias, a physicist studying superconductivity at the University of Rochester in New York, and a recipient of grants from the US National Science Foundation (NSF).

The papers by Dias and his co-authors claimed to report room-temperature superconductivity under extremely high pressures, each in different materials. Room-temperature superconducting materials are highly sought after. They could, for example, transform the efficiency of electricity transmission, from the smallest to the largest application. But high-pressure experiments are difficult and replicating them is complex.

Nature initiated an investigative process that resulted in the 2020 paper being retracted after members of the community told the journal they were troubled by aspects of the data being reported. Nature also initiated an investigation into the 2023 paper. However, this article was retracted at the request of most of Dias’s co-authors while the investigation was still ongoing.

Many details about this case came to light thanks to continued questions from the research community, including during post-publication peer review. Much credit must also go to the persistence of science journalists, including members of Nature’s news team (which is editorially independent of Nature’s journal team) and those from other publications.

What can journal editors, funding organizations and institutions that employ researchers learn from such cases? We have the same goal: producing and reporting rigorous research of the highest possible standard. And we need to learn some collective lessons — including on the exchange of information.

The University of Rochester conducted three inquiries, which are a preliminary step to making a decision about whether to perform a formal investigation into scientific misconduct. The inquiries were completed between January and October 2022. Each concluded that such an investigation was not warranted.

Superconductivity scandal: the inside story of deception in a rising star’s physics lab

Earlier this month, Nature’s news team uncovered a 124-page report on a subsequent confidential investigation, performed at the NSF’s request. In it, a team of reviewers concluded after a ten-month assessment of evidence that it was more likely than not that Dias had committed data fabrication, falsification and plagiarism. The report is dated 8 February 2024, and the determination is regarding the two Nature papers, a 2021 study³ published in Physical Review Letters and a 2022 study⁴ in Chemical Communications — both of which were also retracted. However, the investigation has not yet officially been made public.

Some researchers have asked why Nature published Dias’s second paper in March 2023, when questions were being asked about the first one. Others have asked why the retraction notices didn’t spell out that there has been misconduct.

It’s important to emphasize that it’s Nature’s editorial policy to consider each submission in its own right. Second, peer review is not designed to identify potential misconduct. The role of a journal in such situations is to correct the scientific literature; it is for the institutions involved to determine whether there has been misconduct, and to do so only after the completion of due process, which involves a systematic evaluation of primary evidence, such as unmodified experimental data.

Access to raw data is fundamental to resolving cases of potential misconduct. It is also something we constantly think about in relation to publishing. Indeed, for certain kinds of data, Nature requires authors to deposit them in external databases before publication. But there must be more the research community — including funders and institutions — can all do to incentivize data sharing.

Another question is whether the matter could have been dealt with more quickly. Nature’s editors have been asking the same question: specifically, could there have been more, or better, communication between journals and institutions once evidence of potential misconduct came to light?

Publish, and be damned…

Last month, the Committee on Publication Ethics (COPE), a non-profit organization that represents editors, publishers and research institutions, updated its guidelines on how publishers and universities could communicate better. The guidelines are full of important advice, including that institutions, not publishers, should perform integrity or misconduct investigations. Investigators require access to primary evidence. As employers and grant-givers, institutions are the appropriate bodies to mandate access to unmodified experimental data, correspondence, notebooks and computers and to interview relevant staff members — all essential parts of an investigation.

But often, journals need to start a process that could lead to retracting a study in the absence of an institutional investigation — or while an investigation, or inquiry, is ongoing⁵. Are cases such as this an opportunity for journals and institutions to discuss establishing channels through which to exchange information, in the interest of expedited outcomes — as part of due process? Nature’s editors would be willing to play a part in such discussions.

Retractions are part of publishing research, and all journals must be committed to retracting papers after due process is completed. Although a paper can be retracted for many reasons, when the cause is potential misconduct, institutions must conduct thorough investigations.

This case is not yet closed. Both the university and the funder need to formally announce the investigation’s results, and what action they intend to take. They should not delay any more than is necessary. When there is credible evidence of potential scientific misconduct, investigations should not be postponed. There is strength in collaborating to solve a problem, and nothing to be ashamed of in preserving the integrity of the scientific record.

[ad_2]

Source link

April 24, 2024
Londoners see what a scientist looks like up close in 50 photographs

[ad_1]

Nature’s Where I Work photo-essay section has profiled more than 200 scientists so far, working in settings that range from Vatican City to the University of the West Indies. Now, 50 of the published images are appearing in an outdoor public exhibition in London.

The selection of portraits, which are also collated online, features working researchers in diverse and important fields. The exhibition is organized in collaboration with Argent, a retail-management company based in London.

Two passers-by pause at one of the display boards. On the side facing the camera is a portrait of biotechnologist Sara Abdou, who explores the genetics behind ornamental-flower colours.Credit: John Sturrock

The images, commissioned especially for the journal, are on display in the King’s Cross area, near to Springer Nature’s corporate offices in the United Kingdom. The free exhibition aims to inspire younger generations to consider a career in science, technology, engineering or mathematics, and to challenge stereotypical preconceptions of what a scientist looks like and does. The portraits will remain on display until June 2024.

The exhibition is dedicated to Karen Kaplan, the senior careers editor who launched Where I Work in 2019, to mark Nature’s 150th anniversary. Karen died in November 2023.

[ad_2]

Source link

April 18, 2024
Researchers want a ‘nutrition label’ for academic-paper facts

[ad_1]

Inspired by the nutrition-facts labels that have been displayed on US food packaging since the 1990s, John Willinsky wants to see academic publishing take a similar approach to help to inform readers on how strictly a paper meets scholarly standards.

A team at the Public Knowledge Project, a non-profit organization run by Willinsky and his colleagues at Simon Fraser University in Burnaby, Canada, has been investigating how such a label might be standardized in academic publishing¹.

Willinsky spoke to Nature Index about what he hopes to achieve with the initiative.

Why should academic papers have publication-facts labels?

I, like many others, have grown concerned about research integrity. Through transparency, we want to show how closely journals and authors are adhering to the scholarly standards of publishing. We want to help readers, including researchers, the media and the public, to decide whether an article is worth reporting on or citing.

The facts that we have selected for the label include publisher and funder names, the journal’s acceptance rate and the number of peer reviewers. The label also shows whether the paper includes a competing-interests statement and an editor list, where the journal is indexed and whether the data have been made publicly available. Averages for other participating journals are listed, for comparison.

It’s important that such information is readily available. When we conducted an exercise with secondary-school students, asking them to find these facts for a single academic article online, many of them took 30 minutes to do so. Some couldn’t find the information. This finding justifies the need for the label: it shouldn’t take half an hour to establish that a journal adheres to scholarly standards.

How did you create the label?

The US nutrition-facts label has been proved to change people’s behaviour, specifically their food-purchasing habits². Given that so much work went into the label’s development, I thought it would be wise to build on its design.

On the basis of our early consultations with researchers, editors, science journalists, primary-school teachers and others, we created a prototype with eight elements that reflect scholarly publishing standards. We’re now gathering feedback, and might decide to change some of the facts, or to add others. Some people, for example, suggested that we include the number of days that the peer-review process took to complete.

Is AI ready to mass-produce lay summaries of research articles?

We’ve built in ways to automatically generate the label, to ensure that the format is standardized across journals and articles and to make the label available in several languages. We have created a third-party verification system, too, to ensure that authors’ identities are not revealed to peer reviewers and vice versa. This relies on authors, reviewers and editors using ORCID, the service that provides unique indicators with which to identify researchers.

The label will be displayed on the article landing page of the journal website and will be included in the article PDF.

How are you trialling the label’s use?

We’ve completed work with ten focus groups involving journal editors and authors in the United States and Latin America. We also interviewed 15 science journalists about what kinds of fact they’d want to see at a glance.

We built the label specifically for journals using the scholarly publishing workflow system Open Journal System (OJS), run by the Public Knowledge Project. By the middle of the year, we hope to launch a pilot programme involving more than 100 journals using the OJS. The goal is to explore the prospects of industry-wide implementation of the label by next year.

How could journals be compelled to display such a label?

Unlike the nutrition-facts label, which was mandated by the US government, the publication-facts label is the result of voluntary concern about research integrity in the publishing industry.

Although many groups, such as the International Association of Scientific, Technical and Medical Publishers and the Committee on Publication Ethics, manage concerns about research integrity by releasing guidelines on best practices and accumulating tools to flag suspicious activity, we feel that they have not addressed the fact that open access is public access. We need to adapt our practices to cater to the needs of different audiences, not just those in academia.

Although we’re initially building the label for OJS journals, it is an open-source plug-in that other publishing platforms will easily be able to adapt. The software is currently listed as being ‘under development’ on GitHub and will be shared there on release.

We want to show the publishing industry that we’ve piloted this in our own environment and that it is readily adaptable. We want to show that, although you could build your own label, for the sake of comprehensibility, it’s better to have a common format.

This interview has been edited for length and clarity.

[ad_2]

Source link

April 17, 2024
US COVID-origins hearing puts scientific journals in the hot seat

[ad_1]

Brad Wenstrup (right), a Republican from Ohio who chairs the Select Subcommittee on the Coronavirus Pandemic, speaks with Raul Ruiz (left), a Democrat from California who is ranking member of the subcommittee.Credit: Al Drago/Bloomberg/Getty

During a public hearing in Washington DC today, Republicans in the US House of Representatives alleged that government scientists unduly influenced the editors of scientific journals and that, in turn, those publications stifled discourse about the origins of the COVID-19 pandemic. Democrats clapped back, lambasting their Republican colleagues for making such accusations without adequate evidence and for sowing distrust of science.

US congressional hearing produces heat but no light on COVID-origins debate

The session is the latest in a series of hearings held by the Select Subcommittee on the Coronavirus Pandemic to explore where the SARS-CoV-2 coronavirus came from, despite a lack of any new scientific evidence. Scientists have for some time been arguing over whether the virus spread naturally, from animals to people, or whether it leaked from a laboratory in Wuhan, China. Some have alleged that in the early days of the pandemic, government scientists Anthony Fauci, former director of the US National Institute of Allergy and Infectious Diseases, and Francis Collins, former director of the US National Institutes of Health (NIH), steered the scientific community, including journals, to dismiss the lab-leak hypothesis.

During the pandemic, “rather than journals being a wealth of information”, they instead “put a chilling effect on scientific research regarding the origins of COVID-19”, Brad Wenstrup, a Republican representative from Ohio who is chair of the subcommittee, said at the hearing. Raul Ruiz, a Democratic representative from California who is the ranking member of the subcommittee, shot back: “Congress should not be meddling in the peer-review process, and it should not be holding hearings to throw around baseless accusations.”

Holden Thorp, editor-in-chief of the Science family of journals in Washington DC, appeared before the committee to deny the suggestion that he had been coerced or censored by government scientists.

The subcommittee also invited Magdalena Skipper, Nature’s editor-in-chief, and Richard Horton, editor-in-chief of the medical journal The Lancet, to appear, but neither was present. Skipper was absent owing to scheduling conflicts, but a spokesperson for Springer Nature says the company is “committed to remaining engaged with the Subcommittee and to assisting in its inquiry”. (Nature’s news team is editorially independent of its journals team and of its publisher, Springer Nature.) The Lancet did not respond to requests for comment.

Academic influence?

This is not the first time that Republicans have accused members of the scientific community of colluding with Fauci and Collins. Evolutionary biologist Kristian Andersen and virologist Robert Garry appeared before the same subcommittee on 11 July last year to deny allegations that the officials prompted them to publish a commentary in Nature Medicine¹ in March 2020 concluding that SARS-CoV-2 showed no signs of genetic engineering. They wrote in the journal that they did not “believe that any type of laboratory-based scenario is plausible” for the virus’s origins.

Holden Thorp became editor-in-chief of the Science family of journals in 2019.Credit: Steve Exum

Some lab-leak proponents have suggested, without evidence, that the pandemic began because the NIH funded risky coronavirus research at a lab in Wuhan, offering a motive for Collins and Fauci to promote a natural origin for COVID-19.

During the latest hearing, Republicans went a step further to suggest that not only did Collins and Fauci influence prominent biologists, but that they also encouraged journals to publish research supporting the natural-origin hypothesis. This accusation is based on e-mails that Wenstrup says the subcommittee obtained showing communication between top journal editors and government scientists. Thorp forcefully denied this line of questioning. “No government officials prompted or participated in the review or editing” of two key papers²^,³ on COVID-19’s origins published in Science, he testified. “Any papers supporting the lab-origin theory would go through the very same processes” of peer review as any other paper, he said.

Thorp otherwise spent much of the 80-minute hearing answering questions about how a scientific manuscript is prepared for publication, what a preprint is and how peer review works. In a tense moment, Wenstrup questioned a social-media post on Thorp’s personal X (formerly Twitter) page, in which he downplayed the lab-leak hypothesis. Thorp called the post “flippant” and apologised.

Communication queries

Correspondence between journal editors and government scientists is to be expected, Deborah Ross, a Democratic representative from North Carolina, said at the hearing. “Government actors querying academia on issues that are academic in nature isn’t malpractice or unlawful — it’s just doing their jobs.”

Anita Desikan, a senior analyst at the Union of Concerned Scientists who is based in Washington DC and focuses on scientific integrity, tells Nature’s news team that it is customary for government agencies to reach out to stakeholders to inform policy decisions. Even if a government scientist suggests an idea for a journal paper, “that doesn’t mean it will be published or receive praise from the scientific community”.

Roger Pielke Jr, a science-policy researcher at the University of Colorado Boulder, who was originally slated to testify before the subcommittee until his invitation was rescinded owing to logistical reasons, disagrees. He thinks that Fauci and Collins still shaped the Nature Medicine COVID-19 origins paper by recommending that specific scientists investigate and by offering advice along the way. Nevertheless, the hearing was a “dud”, Pielke Jr says, because Thorp was the wrong witness. Instead, a more relevant witness would have been a government scientific-integrity officer who is more knowledgeable about what constitutes an ethical breach, he adds.

[ad_2]

Source link

April 16, 2024
Is ChatGPT corrupting peer review? Telltale words hint at AI use

[ad_1]

A study suggests that researchers are using chatbots to assist with peer review.Credit: Rmedia7/Shutterstock

A study that identified buzzword adjectives that could be hallmarks of AI-written text in peer-review reports suggests that researchers are turning to ChatGPT and other artificial intelligence (AI) tools to evaluate others’ work.

The authors of the study¹, posted on the arXiv preprint server on 11 March, examined the extent to which AI chatbots could have modified the peer reviews of conference proceedings submitted to four major computer-science meetings since the release of ChatGPT.

Their analysis suggests that up to 17% of the peer-review reports have been substantially modified by chatbots — although it’s unclear whether researchers used the tools to construct reviews from scratch or just to edit and improve written drafts.

How ChatGPT and other AI tools could disrupt scientific publishing

The idea of chatbots writing referee reports for unpublished work is “very shocking” given that the tools often generate misleading or fabricated information, says Debora Weber-Wulff, a computer scientist at the HTW Berlin–University of Applied Sciences in Germany. “It’s the expectation that a human researcher looks at it,” she adds. “AI systems ‘hallucinate’, and we can’t know when they’re hallucinating and when they’re not.”

The meetings included in the study are the Twelfth International Conference on Learning Representations, due to be held in Vienna next month, 2023’s Annual Conference on Neural Information Processing Systems, held in New Orleans, Louisiana, the 2023 Conference on Robot Learning in Atlanta, Georgia, and the 2023 Conference on Empirical Methods in Natural Language Processing in Singapore.

Nature reached out to the organizers of all four conferences for comment, but none responded.

Buzzword search

Since its release in November 2022, ChatGPT has been used to write a number of scientific papers, in some cases even being listed as an author. Out of more than 1,600 scientists who responded to a 2023 Nature survey, nearly 30% said they had used generative AI to write papers and around 15% said they had used it for their own literature reviews and to write grant applications.

In the arXiv study, a team led by Weixin Liang, a computer scientist at Stanford University in California, developed a technique to search for AI-written text by identifying adjectives that are used more often by AI than by humans.

By comparing the use of adjectives in a total of more than 146,000 peer reviews submitted to the same conferences before and after the release of ChatGPT, the analysis found that the frequency of certain positive adjectives, such as ‘commendable’, ‘innovative’, ‘meticulous’, ‘intricate’, ‘notable’ and ‘versatile’, had increased significantly since the chatbot’s use became mainstream. The study flagged the 100 most disproportionately used adjectives.

Scientific sleuths spot dishonest ChatGPT use in papers

Reviews that gave a lower rating to conference proceedings or were submitted close to the deadline, and those whose authors were least likely to respond to rebuttals from authors, were most likely to contain these adjectives, and therefore most likely to have been written by chatbots at least to some extent, the study found.

“It seems like when people have a lack of time, they tend to use ChatGPT,” says Liang.

The study also examined more than 25,000 peer reviews associated with around 10,000 manuscripts that had been accepted for publication across 15 Nature Portfolio journals between 2019 and 2023, but didn’t find a spike in usage of the same adjectives since the release of ChatGPT.

A spokesperson for Springer Nature said the publisher asks peer reviewers not to upload manuscripts into generative AI tools, noting that these still have “considerable limitations” and that reviews might include sensitive or proprietary information. (Nature’s news team is independent of its publisher.)

Springer Nature is exploring the idea of providing peer reviewers with safe AI tools to guide their evaluation, the spokesperson said.

Transparency issue

The increased prevalence of the buzzwords Liang’s study identified in post-ChatGPT reviews is “really striking”, says Andrew Gray, a bibliometrics support officer at University College London. The work inspired him to analyse the extent to which some of the same adjectives, as well as a selection of adverbs, crop up in peer-reviewed studies published between 2015 and 2023. His findings, described in an arXiv preprint published on 25 March, show a significant increase in the use of certain terms, including ‘commendable’, ‘meticulous’ and ‘intricate’, since ChatGPT surfaced². The study estimates that the authors of at least 60,000 papers published in 2023 — just over 1% of all scholarly studies published that year — used chatbots to some extent.

Gray says it’s possible peer reviewers are using chatbots only for copyediting or translation, but that a lack of transparency from authors makes it difficult to tell. “We have the signs that these things are being used,” he says, “but we don’t really understand how they’re being used.”

‘Obviously ChatGPT’ — how reviewers accused me of scientific fraud

“We do not wish to pass a value judgement or claim that the use of AI tools for reviewing papers is necessarily bad or good,” Liang says. “But we do think that for transparency and accountability, it’s important to estimate how much of that final text might be generated or modified by AI.”

Weber-Wulff doesn’t think tools such as ChatGPT should be used to any extent during peer review, and worries that the use of chatbots might be even higher in cases in which referee reports are not published. (The reviews of papers published by Nature Portfolio journals used in Liang’s study were available online as part of a transparent peer-review scheme.) “Peer review has been corrupted by AI systems,” she says.

Using chatbots for peer review could also have copyright implications, Weber-Wulff adds, because it could involve giving the tools access to confidential, unpublished material. She notes that the approach of using telltale adjectives to detect potential AI activity might work well in English, but could be less effective for other languages.

[ad_2]

Source link

April 10, 2024