← Observatory

Do AI detectors work? What the studies say

In Italy «ai detector» is searched on Google about 90,500 times a month, «ai detector italiano» 3,600 and «rilevatore ia» (AI detector, in Italian) 2,400. These are Google Keyword Planner figures for the last year, and behind many of these searches I imagine a precise scene. A teacher has in front of them an assignment that looks too polished, and is looking for a tool that tells them whether the student wrote it or a chatbot did.

There is, however, another number worth looking at. «ai humanizer», the programs that rewrite a text to make it look human and get past detectors, has 8,100 searches a month. On one side there are people trying to find out, on the other people trying to hide. Students are in the middle, and only one question matters. How far can these tools be trusted when a grade, or an accusation, is at stake?

How a detector reasons

A detector does not find a hidden signature in the text. It estimates a probability, comparing the text with the statistical features typical of language models' writing, for example very predictable words and very regular sentences. The result is a hypothesis, not proof.

The makers say so themselves. Turnitin, a service widely used in universities, writes in its own guide that the AI writing score should not be used as the sole basis for adverse actions against a student. That is why, in my view, before looking at scores it is worth looking at what independent studies say.

What independent studies say

The best-known case is OpenAI's. In January 2023 the company behind ChatGPT released its own text classifier. In its tests it identified only 26% of AI-written texts as «likely AI-written», and mistook 9% of human-written texts for AI. On 20 July 2023 OpenAI withdrew it, explaining that its accuracy was too low.

The same year a group of European researchers led by Debora Weber-Wulff tested 14 tools, including Turnitin, on six types of text. None reached 80% accuracy. The picture changes a lot depending on the text.

Accuracy ranges of 14 detectors on five types of text, from 94-96% on texts written by people to 15-31% on AI-generated texts then paraphrased by software

Average accuracy of 14 detectors. The range goes from the strictest to the most favourable of the three calculation methods used in the study. Source, Weber-Wulff and colleagues, International Journal for Educational Integrity (2023).

With texts written by people the tools make few mistakes. With AI-generated texts that are then edited by hand, or run through a paraphrasing program, most slip through. The authors' conclusion is clear-cut, the tools tested are neither accurate nor reliable and should not be used in academic settings.

A year later, in 2024, Mike Perkins and colleagues checked how easy they are to get around. On 805 texts, detectors identified unedited AI texts only 39.5% of the time, and with simple techniques such as a few spelling mistakes or sentences of more varied length accuracy fell to 22.1%.

The tools are improving, though. A study from the Vrije Universiteit Brussel published in June 2026, comparing GPTZero, Pangram, Copyleaks and Turnitin on 160 documents, found rare false positives on texts written entirely by people. Detection got worse on mixed texts and «humanised» ones, and these authors too conclude that the tools can give a first signal but should never be used as sole evidence in high-stakes decisions.

Who is most at risk, people writing in a second language

There is one finding that, in my view, every teacher should know. In 2023 a group at Stanford University had seven detectors analyse essays written by non-native speakers for the TOEFL, the test of English as a foreign language, and essays by US eighth-grade students.

Three tiles with the results on TOEFL essays written by non-native speakers, 61.3% classified on average as AI, 97.8% flagged by at least one detector, 19.8% flagged by all seven

Essays written by people, analysed by seven detectors. Source, Liang and colleagues, Patterns (2023).

On the native speakers' essays the detectors were almost perfect. On the non-native speakers' essays, instead, on average 61.3% were mistaken for AI-generated text, and 97.8% were flagged by at least one of the seven tools. The reason lies in the way they reason. People writing in a language they have not mastered use simpler words and more predictable sentences, the very features detectors associate with AI.

The study concerns English, and I found no equivalent data for Italian. In a class with students who learned the language only a few years ago, though, this result should give pause before trusting a score.

A small error, on many assignments

Turnitin states a false positive rate below 1% at document level, for documents with at least 20% AI writing, and around 4% at the level of individual sentences. These look like small numbers, but they have to be multiplied by the number of assignments.

Vanderbilt University, in the United States, did the maths in August 2023. In 2022 it had received about 75,000 papers, so even a 1% error rate would have meant about 750 students wrongly flagged. That is why it switched off Turnitin's detector, writing that it would keep it off for the foreseeable future.

The consequences can end up in court. On 28 January 2026 the Supreme Court of the State of New York for Nassau County annulled the sanction Adelphi University had imposed on a student over an essay to which Turnitin had given an AI score of 100%, and ordered it removed from the student's record. The court criticised the way the university had reached its decision, without ruling on the reliability of detectors.

What about digital watermarks?

There is another route, the watermark. The model's maker inserts an invisible statistical signal into the text, which can then be checked. Google described its system, SynthID, in Nature in 2024 and released it as open source. The authors themselves write, however, that the watermark weakens when the text is edited or paraphrased, and that it only works on models that adopt it.

European rules point in the same direction. Article 50 of the AI Act requires providers of systems that generate text, images, audio and video to mark the content in a machine-readable format, and in June 2026 the European Commission published the final version of a voluntary code of practice on how to do it. It is a useful step, but it does not solve the case of a student who rewrites a generated text in their own words.

What to do in class

The Italian Ministry of Education and Merit's guidelines on introducing AI in schools, adopted by decree 166 of 9 August 2025, say nothing about detectors. They do, however, ask that human involvement remain central and irreplaceable, especially in situations that affect students. Starting from there, and from the studies above, I suggest a few practical habits.

  • Never use a score as proof. At most it is a reason to talk to the student, not to sanction them.
  • Assess the process, not just the product. Drafts, notes, successive versions and a short conversation about the text say far more than any detector.
  • Set the rules beforehand. If it is clear when AI may be used and how its use must be declared, there is less to argue about afterwards.
  • Be careful with people writing in a second language, because that is where detectors get it wrong most often.
  • Think about students' data. Uploading an assignment to an online service means handing a third party a text that may contain the name, surname and personal information of a minor. If you really need to, remove the personal data first, for example with the Anonimizzatore IA (AI Anonymiser) that we distribute free of charge for macOS and Windows. Data is recognised on your computer and documents stay there, and the result must always be checked. I also wrote about it in the guide on how to anonymise texts.

If you want to receive updates like this one, create your free account on retoria.it (the form is in Italian). If instead you want to work on these topics with your school's teaching staff, starting from real cases, write to us.


Methodological note

I read in full the studies by Weber-Wulff and colleagues (in the July 2023 preprint, whose abstract matches the published one), Liang and colleagues, Perkins and colleagues (March 2024 preprint, the published version refers to six detectors and confirms the 17.4-point drop) and Van Vlasselaer and colleagues, as well as the official pages of OpenAI, Turnitin and Vanderbilt, the ruling in the Adelphi case and the Ministry's guidelines. Some pages block automated reading, and in those cases I consulted the copy kept by the Internet Archive.

In the first chart accuracy is the average of the 14 tools for each type of text. The study calculates it in three different ways, and the range goes from the lowest value to the highest. Unedited AI texts combine the study's two types of generated text.

I do not report the accuracy figures claimed by detector makers, because they have not been independently verified. I found no decisions by Italian courts on this topic. Search volumes come from Google Keyword Planner for Italy, from October 2025 to September 2026. This article is not legal advice.

Sources