An AI Detection Tool Exposed! Easily Identifies AI-Written Papers
-
According to a study published on Monday (November 6) in the scientific journal 'Cell Reports Physical Science,' scientists have developed a machine learning tool that can easily identify chemistry papers written using the chatbot ChatGPT.
Heather Desaire, a chemist at the University of Kansas and co-author of the paper, noted, 'Most text analysis fields aim to create a truly universal detector that can be applied to anything.' However, she explained that her team is focusing on developing a detection tool tailored to specific types of papers to 'pursue accuracy.'
Desaire stated that the research results show efforts to develop AI detectors can be advanced by customizing software for specific types of paper writing. 'If you can quickly and easily build a specialized tool, then creating different tools for different fields isn’t so difficult.'
According to the published data, this specialized detector outperforms two existing AI detectors on the market, helping academic publishers identify papers generated by AI text generators.
Remarkable Accuracy
Desaire and her colleagues first described their ChatGPT detector in June, applying it to Perspective articles in the journal 'Science.' The team used the AI detector to examine 20 features of writing style, including variations in sentence length and the frequency of certain words and punctuation, to determine whether an article was written by a human scientist or generated by ChatGPT.
At the time, the results showed that 'you can use a small set of features to achieve high accuracy.'
In the latest study, the team trained the AI detector on the introduction sections of papers from ten chemistry journals published by the American Chemical Society (ACS).
Researchers used 100 published human-written introductions as reference material and then asked ChatGPT-3.5 to write 200 introductions in the style of ACS journals. Of these, 100 were generated after providing ChatGPT with paper titles, while the other 100 were generated after providing paper abstracts.
The results showed that when testing human-written introductions against AI-generated ones from the same journal, the AI detector could identify 100% of the first 100 introductions written by ChatGPT-3.5 (based on paper titles). For the latter 100 introductions generated by ChatGPT-3.5 (based on paper abstracts), the accuracy was slightly lower at 98%.
Additionally, the tool also works well for text written by ChatGPT-4.
In comparison, two other AI detectors on the market performed poorly. Germany’s ZeroGPT had an accuracy rate of only 35-65% in identifying AI-written introductions, depending on the ChatGPT version used and whether the introduction was generated from a paper title or abstract. OpenAI’s own text classifier tool also performed poorly, with an accuracy rate of about 10-55%.
Debora Weber-Wulff, a computer scientist at the Berlin University of Applied Sciences who studies academic plagiarism, described the authors' work as 'fascinating.'
She noted that many existing tools attempt to determine authorship by searching for predictive text patterns in AI-generated articles rather than observing writing style and stylistic features. Weber-Wulff said, 'I never thought of using stylometry on ChatGPT.'