How consistent are AI humanizer results across multiple tries?

AI humanizer results aren't consistent across multiple tries. Run one passage through the same tool five times and you get five different rewrites, and the detector scores attached to them can swing 5 to 15 points. That's expected, not broken, because both the rewriting model and the detector sample their output. Measure the range, not the best score.

Most people test a humanizer once, see a green score, and move on. That single run tells you almost nothing, because the next person who pastes the same paragraph will get a different rewrite. The question worth asking is narrower: does this tool reproduce its own work, and how far does it drift when it doesn't?

How consistent are AI humanizer results across multiple tries?

Consistency means run-to-run agreement. You freeze one passage, run it several times with the same model, the same mode and the same settings, then compare the outputs and the scores that come back. You're not measuring quality. You're measuring whether the tool's answer belongs to your text or to the moment you happen to hit send.

In its October 2026 cycle, the AI Humanizer Benchmark ran 11 tools over an identical set of 33 source texts, scored every output with 7 commercial detectors, and published the raw data behind the ranking. Consistency is scored separately and carries 10% of a tool's composite, and the published consistency column ranges from 76% to 98%.

For your own five runs, treat a range of 10 points or less as tight, 10 to 30 points as checkable, and anything past 30 points as a tool that can't reproduce itself.

• The lowest score across your five runs.
• The highest score across your five runs.
• The gap between them, in points. That gap is the number to compare between tools.

A tool that scores 8% on its best run and 34% on its worst hasn't given you an 8% result. It's given you an 8 to 34 percent result, and you can't know which one Monday's document will get.

Why does the same text come back different every time?

Humanizers are language models wearing a rewrite instruction, so they inherit the sampling behaviour of every model behind an API. When the model chooses its next word, it draws from a probability distribution instead of reading off a fixed answer. Nudge one draw and every sentence after it drifts.

Thinking Machines Lab measured this in September 2025, and the write-up is still the clearest: sending one identical prompt at temperature 0 a thousand times produced 80 unique completions, with the most common one appearing 78 times. The outputs stayed byte-identical for the first 102 tokens and only then split apart.

The cause isn't the temperature dial you can see. Hosted models batch requests from every user on the machine, and batch size changes how the arithmetic is grouped, so your result depends on how busy the server was when you hit send.

Independent testing of five hosted models at temperature 0 found accuracy moving by up to 15% across ten runs, with the gap between a model's best and worst run reaching 70%. The authors' conclusion was blunt: none of the models delivers repeatable output, let alone identical strings.

A separate 2025 study on code generation found the same thing from a different angle: at temperature 0, between 18% and 44% of problems had no matching output across five attempts, depending on the benchmark.

Humanizers then add a layer of their own. Each tool brings its own prompt, its own word list and its own idea of what sounds human, and some deliberately randomise the rewrite so two customers never get the same page. A reroll button is an open admission that the first answer was one of many.

What makes detector scores wobble on top of that?

The rewrite drifting is only half the story. The detector on the other side isn't a ruler with fixed markings, and three things move its number.

• It can sample too. Detectors that score text in segments and average the parts will round differently when the segmentation shifts, which is how one paragraph lands at 71% and then 68% on consecutive scans.
• It measures a different quantity from the tool beside it. A detector built on perplexity and one weighting behavioural signals don't disagree about a fact, because they aren't answering the same question. One four-tool test on identical text returned 12%, 34%, 67% and 88%.
• It changes under you. Detector vendors retrain without announcing it. In a release note dated October 14 2025, Turnitin shipped a model tuned for higher recall and told institutions that existing submissions had to be resubmitted to see the new score.

That note also says the improved detection may produce higher AI scores on new submissions. The file didn't change, the verdict did. One detector's own FAQ admits a version of the same limit, noting that accuracy improves as more text is submitted, which is why short passages swing hardest.

Put those together and a 300-word email scored twice can move several points without you touching a word. Small texts give a detector less signal to average out, so run-to-run noise grows exactly where professional writing lives.

A five-run consistency test you can finish in ten minutes

You don't need a lab. You need one passage you never edit, one detector you never switch, and five runs.

1. Freeze a 300-word passage you actually wrote, so you're measuring the tool rather than the sample.
2. Run it five times in one sitting with identical settings, and don't reroll until you get a version you like.
3. Score every run with the same detector and write the five numbers down in the order they came back.
4. Subtract the lowest from the highest. That gap is your range.
5. Repeat the whole set a week later. A tool that was tight in September can loosen after a model update, and you'll only see it if the text stays the same.

Keep the passage on file. Re-scoring a frozen benchmark beats testing fresh prose every month, because a moving sample hides the drift you're trying to find. Our guide to testing an AI humanizer walks through building that fixed set.

Two habits keep the numbers honest. Never compare a range from one tool against a single score from another, and never trust a vendor's comparison table over your own five runs.

What a wide range actually tells you about a tool

A wide range is information, not automatically a defect.

Rewriting is generative, so some spread is honest. What matters is severity. Under 10 points is tight enough to plan around, 10 to 30 points means you check the output before you use it, and past 30 points the tool isn't giving you a result at all. It's giving you a lottery ticket.

The second thing a range tells you is where the ceiling sits. A tool with a 4-point spread that always lands at 60% is consistent and useless. Consistency answers whether you can trust the score. It says nothing about whether the score is any good.

A narrow range on a short passage isn't a win either. Plenty of tools look stable on 120 words and fall apart the moment you paste a full page, which is the same length effect that shows up in the research.

Which humanizers hold up best in public benchmarks?

The open AI Humanizer Benchmark is the rare public source that reports consistency as its own column instead of burying it in a marketing page. Each monthly cycle commits a hash before it runs, scores every tool on identical texts with seven detectors, and lets anyone recompute the leaderboard from the published data.

In the October 2026 cycle the top of that column read: Undetectable AI 98%, UndetectedGPT 97%, WriteHuman 96%, HIX Bypass 94%. StealthGPT sat lowest at 76%, a 22-point spread across a field where every tool advertises the same promise.

Read those columns together, never one at a time. The same cycle scored WriteHuman at 98% on bypass and 53% on meaning preservation, and Undetectable AI at 73% on meaning with a 2% bypass rate. A tool can be perfectly repeatable and still fail the job.

If you're picking between tools rather than auditing one you already pay for, our humanizer comparison ranks the same field on output quality instead of stability, which is the column to read second.

Consistency is a floor, not the goal

A reproducible tool that flattens your voice is still a bad trade. If every run comes back in the same generic register, you've automated the part you were supposed to protect, which is the cost measured in what humanizing does to your voice.

The floor matters more than the average because it decides your workflow. You can live with a tool that drifts 30 points if you read every output. You can't live with one that drifts 30 points while you publish on autopilot.

What survives a model update is dull and cheap: a frozen benchmark set, a monthly rescore, and the edits you'd make by hand anyway. That's the pattern in the research on whether humanizers work.

Every piece of this stack is a moving target. Detectors retrain, humanizers swap models, and neither announces it loudly enough for you to notice before your numbers move. Whether detectors catch humanized text is a question with a different answer every quarter.

What to do when two runs disagree and you have to publish

Run it a third time. Two runs is an anecdote. Three gives you a middle value to fall back on, and it costs you a minute.

Then stop optimizing the score and read the sentence you're least sure about. If a rewrite moved a fact, a name or a number, the consistency question is settled for that run no matter what the detector reports.

If the piece is already published, remember that a score isn't a verdict. It's one classifier's estimate on one afternoon, produced by a tool whose own documentation tells institutions to treat it as a starting point.

Keep the receipt, too. Save the passage, the five scores and the date. When the number moves next month, the only way to tell whether the tool changed or your writing did is to have the old run on file.

Related reading: Does humanize AI actually work on Turnitin? · Which AI detector do college professors trust?

Frequently asked questions