[Womeninmedicine] ChatGPT identifies gender disparities in scientific peer review

[Piraino, Beth]

Fascinating

Beth Piraino, MD
Professor of Medicine
Associate Dean of Admissions and Financial Aid
Telephone 412 648 9891
piraino at pitt.edu<mailto:piraino at pitt.edu>

For patients:     By communicating with me through e-mail, you agree to comply with UPMC's e-mail terms of use found at http://www.upmc.com/contact/Pages/terms-of-use.aspx#Email<https://exchange.pitt.edu/owa/redir.aspx?C=702639e3ea5a4299bd5900fd39e96307&URL=http%3a%2f%2fwww.upmc.com%2fcontact%2fPages%2fterms-of-use.aspx>. Should you decide that you do not want to comply with these terms, it is your obligation to indicate to me that you do not agree and then communication with me by e-mail will cease.  For those seen at University Center Renal Clinic, you may prefer to use Health Trak for communications as this is more secure.  Please do not use email to communicate urgent requests or concerns but call the on call person if after hours or weekends (412 647 2345 and ask for the renal fellow on call).

Any unauthorized or improper disclosure, copying, distribution, or use of hte contents of this document is prohibited.  The information in this e-mal message is intended only for the personal and confidential use of the recipient(s) named above.  If you have received this communication in error, please notify me by e-mail response and delete the original message.

[https://exchange.pitt.edu/owa/attachment.ashx?id=RgAAAAAzSCCfXXNUQrq8XC3kwMv8BwDCNBPmnmueQpV%2bd0bapyBmAOTavZlnAADCNBPmnmueQpV%2bd0bapyBmAOTcgO%2bhAAAJ&attcnt=1&attid0=EABLE%2bpbcbbrSKxKyid2EaVz]

From: Womeninmedicine <womeninmedicine-bounces at list.pitt.edu> On Behalf Of Weisz, Ora Anna
Sent: Wednesday, October 4, 2023 12:53 PM
To: womeninmedicine at list.pitt.edu
Subject: [Womeninmedicine] ChatGPT identifies gender disparities in scientific peer review

Interesting paper and use of ChatGPT...

https://elifesciences.org/reviewed-preprints/90230?utm_source=content_alert&utm_medium=email&utm_content=fulltext&utm_campaign=4-October-23-elife-alert<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Felifesciences.org%2Freviewed-preprints%2F90230%3Futm_source%3Dcontent_alert%26utm_medium%3Demail%26utm_content%3Dfulltext%26utm_campaign%3D4-October-23-elife-alert&data=05%7C01%7Cwomeninmedicine%40list.pitt.edu%7C09188080993f4767fa3d08dbc50cfffc%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C638320431843295397%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZIslO7azCb00RRDkvhtDrVjw0cuUGsCrzoOoiCuXCLA%3D&reserved=0>

Abstract
The peer review process is a critical step in ensuring the quality of scientific research. However, its subjectivity has raised concerns. To investigate this issue, I examined over 500 publicly available peer review reports from 200 published neuroscience papers in 2022-2023. OpenAI's generative artificial intelligence ChatGPT was used to analyze language use in these reports. It demonstrated superior performance compared to traditional lexicon- and rule-based language models. As expected, most reviews for these published papers were seen as favorable by ChatGPT (89.8% of reviews), and language use was mostly polite (99.8% of reviews). However, this analysis also demonstrated high levels of variability in how each reviewer scored the same paper, indicating the presence of subjectivity in the peer review process. The results further revealed that female first authors received less polite reviews than their male peers, indicating a gender bias in reviewing. In addition, published papers with a female senior author received more favorable reviews than papers with a male senior author, for which I discuss potential causes. Together, this study highlights the potential of generative artificial intelligence in performing natural language processing of specialized scientific texts. As a proof of concept, I show that ChatGPT can identify areas of concern in scientific peer review, underscoring the importance of transparent peer review in studying equitability in scientific publishing.

_____________________________________________________________________
Ora A. Weisz, PhD | Professor of Medicine, Cell Biology, and Clinical and Translational Science
Vice Chair of Faculty Development, Department of Medicine
Associate Dean for Faculty Development, University of Pittsburgh School of Medicine
Assistant Vice Chancellor for Faculty Excellence, University of Pittsburgh Health Sciences
Renal-Electrolyte Division | 978.1 Scaife Hall | 3550 Terrace St. | Pittsburgh PA 15261
https://www.weiszlab.pitt.edu<https://www.weiszlab.pitt.edu/>
Tel: 412-383-8891 | Email: weisz at pitt.edu<mailto:weisz at pitt.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.pitt.edu/pipermail/womeninmedicine/attachments/20231004/384c33dc/attachment-0001.htm>