A study by Copyleaks found that a staggering 60% of the outputs from OpenAI’s GPT-3.5 exhibited signs of plagiarism.
Copyleaks, who develop plagiarism and AI content analysis tools, highlight AI-generated text’s questionable originality and reliability, particularly in light of recent copyright infringement and plagiarism controversies.
The study analyzed 1,045 outputs from GPT-3.5, spanning 26 academic and creative subjects, including but not limited to physics, chemistry, computer science, psychology, law, and the humanities, with each output averaging 412 words in length.
The findings of the Copyleaks report include the following:
- Approximately 59.7% of all GPT-3.5 generated texts were found to contain plagiarized content to some degree.
- 45.7% of outputs contained exact text matches, 27.4% included slight modifications, and 46.5% involved paraphrasing from pre-existing sources.
- Notably, the subject of computer science saw the highest individual output “Similarity Score” at some 100%, highlighting a significant concern in fields heavily reliant on technical and specialized language.
The study’s “Similarity Score” is a proprietary metric designed by Copyleaks to quantify the degree of originality in content. It amalgamates various factors, such as identical text and paraphrasing.
Physics recorded the highest mean Similarity Score at 31.3%, with Psychology not far behind at 27.7% and General Science at 26.7%. On the opposite end of the spectrum, Theater had the lowest mean score at just 0.9%, followed by Humanities at 2.8% and the English Language at 5.4%.
The spread of similarity scores across subjects isn’t particularly surprising. There are near-limitless ways to interpret a Shakespeare play and far fewer to analyze a well-established mathematical theorem, for example.
Alon Yamin, CEO and Co-founder of Copyleaks, said subjects like physics, chemistry, computer science, and psychology warrant closer scrutiny for plagiarism due to their higher scores.
“For example, Physics, Chemistry, Mathematics, and Psychology might require a more in-depth look to identify plagiarized text, while other subjects, including Theater and Humanities, may require less scrutiny,” said Yamin.
However, educators must acknowledge how some subjects naturally lend themselves to high similarity scores.
Yamin also stated, “Furthermore, the data underscores the need for organizations to adopt solutions that detect the presence of AI-generated content and provide the necessary transparency surrounding potential plagiarism within the AI content.”
That’s a good point. If educational organizations allow AI to draft and generate content (and some already are), students could still be exposed to plagiarism.
It must also be said that scores for GPT-4-generated content would have shown lower plagiarism scores.
While the bulk of AI-generated content is probably still created with GPT-3.5 (because it’s free), GPT-4 is undoubtedly more effective at generating original work.
However, this introduces another layer of complexity.
Since GPT-4 is part of the paid version of ChatGPT, accepting or encouraging AI uses in education could discriminate against GPT-3.5 users unless subscriptions are subsidized.
A delicate balance
As generative AI tools become embedded in academic settings, both educators and students are confused about their use.
Content analysis companies like Copyleaks and Turnitin have developed AI detection tools that predict when a string of words is likely AI-generated. However, these have evident weaknesses and risk false positives.
Further, AI detection software has been shown to heavily favor native English writing, as it often contains a higher concentration of diverse vocabulary and idioms to sway AI detectors towards labeling text as ‘human-written.’
Curbing the use of AI technology in academia will not be easy. Generative AI is billed as the ultimate productivity tool, and many argue that if you can use it, you should.
Students often argue that if these tools are pervasive in the real world, they should also be allowed in educational settings.
Plus, as many would attest, education is sometimes about finding inventive shortcuts to get things done.
Can you really expect students to leave generative AI untouched on the table?
