Can AI detect whether text was written by AI?
NoNo, not reliably. Detectors miss much AI text and wrongly flag human writing, so a detector score should never be treated as proof.
First shown: Jan 31, 2023. OpenAI released a classifier for AI-written text that caught only 26% of AI text and wrongly flagged 9% of human text; it withdrew it on July 20, 2023 for low accuracy. (OpenAI)
What it means
AI-text detectors look for statistical patterns typical of machine writing and give a probability that a passage was AI-written. Schools, publishers and employers use them to check work.
The record is poor. OpenAI's own classifier, released in January 2023, caught about 26% of AI-written text and mislabeled about 9% of human writing; OpenAI withdrew it on July 20, 2023, citing its low rate of accuracy. A Stanford-led study found detectors flagged writing by non-native English speakers as AI-generated while correctly identifying native speakers' writing, and that simple prompting could help AI text evade detection.
Small error rates add up. Vanderbilt University switched off Turnitin's AI detector in August 2023, noting that even the 1% false-positive rate Turnitin claimed would mean about 750 of its 75,000 yearly papers wrongly flagged. Asking a chatbot whether it wrote something is worse still: chatbots cannot recognize their own output.
Watermarks that an AI company embeds when it generates text are a different approach and can work, but only for that company's tools and only if the text is not reworded.
The evidence ladder
Every capability on this site is graded on the same five steps, from a single demonstration to real-world consequences.
| Step | Status | When | Evidence |
|---|---|---|---|
| Demonstrated shown at least once | partly | Jan 31, 2023 | OpenAI's classifier caught about 26% of AI-written text and wrongly flagged about 9% of human text on its challenge set. (OpenAI) |
| Reproduced independently repeated | partly | Apr 6, 2023 | Independent researchers tested several detectors and found they flagged non-native English writing as AI-generated and could be evaded with simple prompting. (Liang et al. (arXiv)) |
| Accessible available to ordinary people, or criminals | yes | Aug 16, 2023 | Detectors are built into everyday tools: Turnitin switched its AI detector on for institutions such as Vanderbilt with less than 24 hours' notice. (Vanderbilt University) |
| Reliable works most of the time for a defined task | no | Jul 20, 2023 | OpenAI withdrew its classifier for low accuracy, and Vanderbilt disabled Turnitin's detector, saying it did not believe AI detection software is an effective tool. (OpenAI) |
| Consequences real-world use or harm documented | yes | May 18, 2023 | A Texas A&M-Commerce instructor asked ChatGPT whether students' essays were AI-written and withheld grades; several students were later cleared. (The Register) |
From the record
- A Texas A&M-Commerce instructor used ChatGPT to 'detect' AI essays and withheld seniors' grades; several were later cleared. (The Register)
- OpenAI withdrew its own AI-text classifier because of its low rate of accuracy. (OpenAI)
- Vanderbilt disabled Turnitin's AI detector over false positives, lack of transparency and bias against non-native writers. (Vanderbilt University)
What it means for you
A detector score saying 'AI-written' is a guess, not evidence. Students, especially non-native English writers, can be wrongly accused, and people using AI can slip through.
What to do
- Keep your draftsVersion history, notes and earlier drafts are the best evidence that you wrote something yourself.
- Ask what the evidence isIf accused, ask which tool was used and point to its published error rates and OpenAI's withdrawal of its own detector.
- Teachers: don't rely on a scoreTalk with the student about the work; Vanderbilt and others stopped using detectors for this reason.
- Don't ask a chatbotChatbots cannot tell whether they wrote a passage; their answer is not evidence.
Questions people ask
Are AI detectors accurate?
Not reliably. OpenAI's own detector caught about 26% of AI text and flagged 9% of human text before it was withdrawn in 2023.
Why did OpenAI shut down its AI detector?
OpenAI said on July 20, 2023 that the classifier was no longer available due to its low rate of accuracy.
Are detectors biased against non-native English speakers?
A Stanford-led study found GPT detectors flagged non-native English writing as AI-generated while correctly identifying native speakers' writing.
Can ChatGPT tell if it wrote something?
No. Asking a chatbot whether it wrote a text gives an unreliable answer, as the Texas A&M-Commerce case showed.
Related
Sources
- OpenAI: New AI classifier (with July 2023 withdrawal note)
- Liang et al., GPT detectors are biased against non-native English writers
- Vanderbilt: why we're disabling Turnitin's AI detector
- The Register on Texas A&M-Commerce
Spot an error or something out of date? Tell us; corrections are made and logged. Written with AI assistance from public sources; every fact checked and approved by Colin Balfe. General information, not legal, medical or financial advice.