Edward Tian, a senior at Princeton University, has created an app called GPTZero that aims to detect text written by the controversial AI chatbot ChatGPT.
The app, which was developed over a summer break, had 30,000 hits within a week of its launch.
According to Tian, the motivation for creating GPTZero was to address concerns about the use of AI to cheat in exams by evading anti-plagiarism software.
He initially promoted the app in a tweet that went viral, claiming it could “quickly and efficiently” detect AI-generated text. However, users have reported mixed results with the app.
GPTZero, the app developed by Edward Tian, uses a calculation of “perplexity” and “burstiness” to determine whether AI wrote a text.
Perplexity measures the complexity of a text, and burstiness compares the variation of sentences.
The app uses a similar data set to the AI bot that is being tested, which allows the app to gauge the likelihood that AI-generated the text.
Streamlit, the free platform that hosts GPTZero, has provided support to Tian by providing hosting and memory capabilities to keep up with the web traffic.
Edward Tian, the creator of GPTZero, has informed subscribers that a new model of the app has been created with an improved capacity for detecting artificial intelligence in text.
He claims that through testing the new model on a dataset of BBC news articles and AI-generated articles from the same headlines, the model has a false positive rate of less than 2%.
He plans to focus on building, improving, and scaling GPTZero in the coming months.
However, Toby Walsh, a professor of artificial intelligence at the University of New South Wales, expresses skepticism about the app’s ability to have an impact on ChatGPT’s capacity to be used for plagiarism.
He believes if a major company adopts the app, it’s likely to make a significant impact and adds that ChatGPT can be easily rephrased to a more personable style. It will make it harder for GPTZero to detect.
He also points out that it’s an ongoing arms race between technology that identifies synthetic text and apps.
According to Professor Toby Walsh, a specialist in AI, ChatGPT users can evade detection by adding more “randomness” to the text, using synonyms and making grammatical changes.
He also suggests that each app created to detect synthetic text gives artificial intelligence programs greater ability to evade detection.
Additionally, every time a user logs into ChatGPT, the system generates human feedback to improve its filters, both implicitly and explicitly.
Professor Walsh also points out that there’s a deep fundamental technical reason why it is impossible to win the arms race, as every program used to identify synthetic text can be added to the original program to generate synthetic text that can fool them, and it’s getting better every day.
Users of GPTZero have reported mixed results.
According to feedback from some users of GPTZero, the app is effective in detecting text generated entirely by GPT models or generated with semi-human intervention.
However, the users report that it needs to work better with essays written by human writers, and it generates many false positives.
Some users found it useful for professors but also caution that trusting it too much could lead to the exaggeration of false positives.
Another user commented that ChatGPT is so good that it managed to pass the test as human text with small manual editing and a strongly enforced context. And is not convinced that GPTZero can flag human-AI cooperation.
In the end, GPTZero, the app developed by a Princeton senior, aims to detect text written by the AI chatbot ChatGPT.
It uses a calculation of “perplexity” and “burstiness” to determine the likelihood that AI-generated the text. However, according to feedback from some users, the app is not a foolproof solution, and it does have a false positive rate.
Users have reported mixed results with the app, and some experts have also expressed skepticism about its ability to have a significant impact on ChatGPT’s capacity to be used for plagiarism.
It’s important to keep in mind that this is an ongoing arms race between technology that identifies synthetic text and apps.
While GPTZero may help detect AI-generated text, it’s not a silver bullet solution.
