Burstiness and perplexity: what AI detectors actually measure

Perplexity is how predictable each word looks to a language model. Burstiness is how much that predictability swings across a document. Low on both means the draft reads like a machine: a model picks the likeliest word every time, at the same rhythm. Neither number is an accuracy score, and no serious detector now uses either one alone.

Most explanations stop at the definition. This one shows the arithmetic, then the measured numbers, then the point where the number stops telling you anything useful.

What burstiness actually measures

Burstiness is a variance statistic wearing a marketing name. Split a document into sentences, score each sentence for how surprised a language model was by its word choices, then measure how far those scores spread. Wide spread, human. Flat spread, machine. That's the entire idea, and it fits in one line.

Vendors compute the spread differently. Some take the standard deviation of the per-sentence scores, some divide by the mean to get a coefficient of variation, some average the result over windows of 100 to 500 words. The window size changes the answer, which is the first sign that this is a heuristic rather than a measurement.

You can approximate burstiness without a language model, because sentence length is the cheapest stand-in for it. It isn't the same quantity, and vendors who score perplexity directly will say so. For a fast read on your own draft, length gets you most of the way, and the recurring patterns that flatten sentence rhythm are easier to see once you've counted them.

There isn't an absolute scale for perplexity, but generally, a perplexity above 85 is more likely than not from a human source.

That line comes from GPTZero's own account of its first detection model, published in early 2023 and still live on their site. Note the hedge inside the sentence: no absolute scale. Everything after that hedge is a judgement call about where to draw a line.

How to measure it on your own draft

You need four numbers and one division. Sentence count, the word count of each sentence, the mean, and the standard deviation. Divide the standard deviation by the mean and you have the coefficient of variation, which is the version worth comparing between drafts.

The division matters. A writer who averages 40 words per sentence will post a large standard deviation purely because their sentences are long, not because the rhythm varies. The coefficient cancels that out, so a terse writer and a sprawling one land on the same scale.

• Split the text on terminal punctuation and drop fragments and anything over 120 words.
• Count words per sentence, punctuation excluded.
• Take the mean and the population standard deviation, not the sample version.
• Divide, then compare the result to the table below rather than to a figure someone quoted at you.

What the numbers look like in real prose

I ran the calculation on three public-domain novels, three windows each, 1,500 words per window. These are the numbers, unrounded beyond one decimal. They are dull, and that's the point.

• Pride and prejudice, opening: 44 sentences, mean 34.3 words, standard deviation 23.1, coefficient 0.675.
• Pride and prejudice, middle: 62 sentences, mean 24.5 words, standard deviation 18.0, coefficient 0.735.
• Pride and prejudice, late: 52 sentences, mean 28.8 words, standard deviation 23.5, coefficient 0.814.
• Moby-Dick, opening: 90 sentences, mean 14.5 words, standard deviation 19.5, coefficient 1.345.
• Moby-Dick, middle: 37 sentences, mean 41.8 words, standard deviation 25.2, coefficient 0.603.
• Moby-Dick, late: 62 sentences, mean 24.5 words, standard deviation 14.8, coefficient 0.607.
• Frankenstein, opening: 62 sentences, mean 24.2 words, standard deviation 17.0, coefficient 0.702.
• Frankenstein, middle: 46 sentences, mean 33.2 words, standard deviation 22.5, coefficient 0.679.
• Frankenstein, late: 53 sentences, mean 28.5 words, standard deviation 19.3, coefficient 0.676.

Nine windows, coefficients from 0.603 to 1.345. Mean sentence length ran from 14.5 to 41.8 words. Three canonical English novels disagree about how long a sentence should be by nearly a factor of three, and not one of those windows sits anywhere near a flat line.

Method notes so you can repeat it: Project Gutenberg plain-text editions, sentences split on terminal punctuation, fragments under three words and sentences over 120 words dropped, population standard deviation. The texts are free, so the numbers are checkable rather than quoted at you.

Which gives a rough reading guide. Under 0.25, the rhythm is doing something repetitive. Between 0.25 and 0.5, the draft has probably been smoothed by an editor, a tool, or a habit. Above 0.5, you're in the range every one of those nine windows occupied. It's a guide and not a verdict, because the coefficient counts length while detectors count more than length, and short samples break the statistic before they break anything else.

Why does the biggest vendor no longer lead with them?

Because it changed its model. GPTZero's original January 2023 detector was this perplexity and burstiness calculation, and the company says so plainly. The same page carries an update: as of autumn 2023 the company no longer uses those two metrics for detection, having migrated to a deep-learning architecture.

The metrics didn't vanish. GPTZero describes them as one of seven indicators inside the upgraded model, and says the same statistical approach is what ships behind ZeroGPT, Copyleaks, Originality and Writer.ai. Cheap detectors still let two statistics decide. Expensive ones treat those statistics as one vote among several.

So every page explaining AI detection as perplexity plus burstiness, full stop, is describing the 2023 model. Several of those pages were published this year. The two-signal story survives because it's teachable, not because it's current.

What perplexity gets wrong about careful writing

Perplexity measures fit to one model's beliefs about language. It doesn't measure truth, effort, originality, or whether a person typed the words. A sentence a model would have predicted gets a low score even when a human wrote it slowly and meant every word of it.

The clearest evidence is a 2023 study in the journal Patterns. Liang and colleagues ran seven detectors over 91 human-written TOEFL essays and 88 essays by US eighth-graders. The detectors averaged a 61.22% false positive rate on the non-native writers and near-perfect accuracy on the eighth-graders.

The mechanism is the metric working exactly as designed. Careful, simple, high-frequency English is predictable English, and predictable English scores low. The detector reads a second-language writer's caution as machine flatness, which is why non-native writers get flagged far more often than the tool's marketing implies.

The same shape of error hits anyone whose prose is conventional by trade: legal writing, regulatory filings, scientific abstracts, and any draft that went through a grammar tool which smooths every rough edge on the way out.

False positives aren't a rare edge case you can wave off. They're the metric's normal failure mode, they land hardest on writers with the least power to appeal, and the published false positive rates are worse than the marketing pages suggest.

There's a second, less comfortable finding. A 2023 paper by Sadasivan and colleagues showed that recursively paraphrasing model output breaks a wide range of detectors, including watermarking schemes, zero-shot classifiers and retrieval-based ones. If one editing pass can move a verdict that far, the number was never measuring authorship.

Can you raise burstiness without wrecking the sentence?

Yes, and the demonstration is the reason to be suspicious of the number. Here are two six-sentence passages I wrote for this piece, scored the same way as the novels.

• Flat: 15, 14, 13, 14, 12, 13 words. Mean 13.5, standard deviation 1.0, coefficient 0.071.
• Varied: 6, 34, 2, 35, 1 words. Mean 15.6, standard deviation 15.5, coefficient 0.995.

The second passage scores like a human and reads like a stunt. One-word sentences, a 35-word sprawl, no paragraph shape worth defending. It clears the threshold and fails as writing, which tells you what the threshold is actually worth.

The goal was never the coefficient. If your prose is flat because your thinking is flat, splicing in a two-word sentence is cosmetic, and detectors that measure structural variation rather than raw length won't be fooled by it. Deleting the sentences that say nothing lifts variation as a side effect, and that deletion pass is the version that survives a second reading.

Which numbers should you ignore?

Start with any whole-document percentage handed to you without a distribution. A 1,000-word file can hold three machine-flat paragraphs and five ordinary ones and still return a mid-range score that tells nobody anything actionable, which is the first thing to check when reading a detector result.

Ignore scores on short text. Perplexity needs hundreds of words before it means anything, and burstiness needs several sentences before it has a spread at all.

Ignore the implied precision too. Perplexity is relative to whichever reference model computed it, so a low score against an older model and a high score against a newer one can describe the same paragraph.

And ignore the framing that a detector can tell you whether a person wrote something. It estimates a probability from surface statistics. That's a real signal with real uses, and it isn't authorship. Spotting the tells yourself is slower and more honest.

A five-line audit for your own draft

Take the piece you're worried about, drop it into a text file, and do five things in order. It takes about four minutes.

1. Count the sentences and the words per sentence.
2. Get the mean and the standard deviation.
3. Divide for the coefficient and compare it to the nine windows above.
4. Read your longest and your shortest sentence out loud, back to back. If they sound like the same person, the number is fine and the worry is somewhere else.
5. If the coefficient sits under 0.3, don't splice. Find the paragraph where three consecutive sentences open the same way and cut one of them.

The coefficient is a mirror, and mirrors are useful right up until you start dressing for them. Write the sentence you'd defend to a reader. If a detector dislikes it, that's the detector's problem, and a 61.22% false positive rate already proved the problem exists.

Frequently asked questions