# Public humanizer benchmark, 2026-11-06

reright ran this benchmark. Anyone can rerun it: the inputs, each tool's outputs, our outputs, the scorers and the scores are all here or linked.

## Protocol (fixed before any reright output existed)

This section was committed before reright rewrote one benchmark text. Results are appended below it later and the protocol not changed.

### Inputs and competitor outputs

- AI Humanizer Benchmark (AIHB), Hugging Face `ai-humanizer-benchmark/ai-humanizer-benchmark`, revision `35cf47ea697819c30d2d7845d37389db318e0363`, CC BY 4.0. AIHB is run by the creators of UndetectedGPT, one of the tools they rank. They use its data. They have not reviewed or approved this work.
- Main set: cycle 2026-11. That is 33 AI-written texts (188 to 349 words, 7 writing categories, 6 source models), and the results of the 10 tools AIHB processed on 2026-11-01 with default settings.
- Second set: cycles 2026-09 and 2026-10, 33 texts each, 11 tools each. These add WriteHuman and AI Humanize, not in the November cycle. Reported separately and also pooled with November (99 texts) for tools common to all three cycles.
- We change no aihb text. The only thing we add is reright's output for same input.

### reright run

- Code: production `main` at `45e0a05`, installed as Modal app `reright-voice` version v8. We log `modal app history` before and after the run. If the version changes during the run, we discard the run and restart.
- Path: the production Go pipeline, the same branch `Queue.run` runs. `voice.UseLong` chooses `RewriteLong` or `Rewrite`. Including masking, the number and name guard and the meaning guard. When the guard fails, user text is returned. The Go harness (`harness/`) calls production `voice/api.py` (`build_app`) via a local bridge whose backend connects to the deployed `Rewriter`. The only difference from production is the transport between the Go server and API, and the request signing key.
- No voice adapter: the default style every account gets before training it's own voice. A synthetic user key, no user data.
- Deadline: `AgentDeadline`(3m) per text as in production. At most 4 texts in flight, so real users not slowed down.
- Kind for every AIHB category now fixed: business_email -->`email`; academic_essay, application_essay, news_article -->`document`; blog_post, marketing_copy, discussion_board --> `post`.
- Two whole runs. Run 1 is the answer. Run 2 shows what the numbers drift between runs and is never used to choose a better run.
- A text that's returns guard unchanged is reright's output in all metrics. It's share is reported as its own column.

### Scorers (the same for every tool and for the unmodified inputs)

Detectors are all open, and they are commercial licences. None used inside reright.

| detector | model | license | direction |
|---|---|---|---|
| desklib | `desklib/ai-text-detector-v1.01` | MIT | p(AI) |
| fakespot | `fakespot-ai/roberta-base-ai-text-detection-v1` | Apache 2.0 | p(AI) |
| Binoculars | observer `tiiuae/falcon-7b`, performer `tiiuae/falcon-7b-instruct` | Apache 2.0 | lower = more AI |
| Fast-DetectGPT | `EleutherAI/gpt-neo-2.7B` as sampling and scoring model | MIT | higher = more AI |

RADAR was excluded from our internal tests as weights aren't licensed for commercial use reright's own AI-text filter was excluded as reright used it

Overlap to disclose: reright picks among 4 samples partly by surprisal under a small reference model (SmolLM2-360M). Binoculars and Fast-DetectGPT are also likelihood based, so reright is partly tuned toward what those two measure. desklib and fakespot are trained classifiers and share nothing with reright.

Cuts: each detector has a cut which flags at most 5% of a set of human texts of the same length, the human set consists of two sets.
1. The 70 human controls from our internal test set: 15 essays from `qwedsacf/ivypanda-essays`, 40 blog and academic chunks from the HAP-E human split (`browndw/human-ai-parallel-corpus`, from COCA) and 15 reports from `ccdv/govreport-summarization`. We publish dataset pointers and row ids, not the texts, because the underlying texts are not our to redistribute.
2. The 54 US federal public domain passages from the GoHumanize Open Humanizer test set (Zenodo 10.5281/zenodo.22843083, CC BY 4.0). We release the texts.
Each human text is chopped off to complete paragraphs, from the beginning, until it has at least 250 words, capped at 350. That's the same as AIHB lengths. Given 124 texts, the cut marks at most 6 of those. Results show the flag rate of each detector for both halves of the human set, and the percentage of human texts flagged by any detector.

Long texts: texts are chopped in windows of at most 480 tokens (512 for Binoculars and Fast-DetectGPT) and the document score is the token-weighted mean. Most AIHB texts are one window.

Meaning:
- Facts: `factalign.py` from `zwang-devv/humanizer-fact-audit` (MIT), unchanged. It is the checker from "Humanized but Still True?" (arXiv 2611.00359). The metric is the share of eligible texts (those with at least one number, name or quote of three or more words) with at least one unit changed or missing. We also report the same share with quotes left out. Before scoring reright, we reran the checker on the AIHB October outputs. Its unit judgments matched the paper's released data on all 363 documents.
- NLI: `FacebookAI/roberta-large-mnli` (MIT). That is a different model from the DeBERTa NLI reright uses as a guard. For each sentence of the input, we take the entailment probability with the output as premise. The output is cut into windows of 400 tokens and the best window counts. The metric is the share of input sentences entailed (p > 0.5), averaged over texts. We do the same in the other direction (output sentences entailed by the input) to catch added content.
- Judge: `openai/gpt-oss-20b`, temp 0, with prompt in `code/judge.py`. It scores the preservation of meaning on a scale from 1 to 5. We give the average and percentage of texts scoring 4 or higher.

### What gets reported, whatever the outcome

For each tool, reright and the unmodified inputs: the flag rate per detector, the share flagged by none of the four, judge mean and share of 4 or more, NLI both ways, fact change rate with and without quotes, reright's passthrough rate, and run 2 next to run 1. Every comparison with reright carries a paired bootstrap 95% interval over the texts (2,000 draws, seed 1). We do not rank tools whose intervals overlap.

Dates: competitor outputs are from AIHB's run on 2026-11-01 (and 2026-09 and 2026-10 for the second set). reright's outputs are from the date of our run. Detectors were run on all outputs on the same day with the same code.

AIHB also scores every tool on seven commercial detectors (GPTZero, Originality.ai, Copyleaks, Pangram, ZeroGPT, QuillBot, Grammarly). We have no scores from those for reright, so we compare nothing on them.

## Results (run on 2026-11-06)

### Main set: November 2026, 33 texts, reright run 1

| tool | changed a number, name or quote | handed back unchanged | judge, 1 to 5 | passes all four detectors |
|---|---|---|---|---|
| reright | 14% (3 of 21) | 15% | 4.33 | 39% |
| CleverHumanizer | 38% | 0% | 3.12 | 45% |
| GPTinf | 43% | 0% | 2.45 | 27% |
| HIX Bypass | 38% | 0% | 4.39 | 30% |
| ReHumanize | 38% | 0% | 4.67 | 9% |
| Rephrasy | 43% | 0% | 2.48 | 70% |
| SmartHumanizer | 38% | 0% | 4.12 | 45% |
| StealthGPT | 43% | 0% | 3.00 | 3% |
| SuperHumanizer | 33% | 0% | 3.36 | 58% |
| Undetectable AI | 38% | 0% | 4.09 | 3% |
| UndetectedGPT | 38% | 0% | 4.12 | 48% |
| Unmodified AI input | 0% | - | - | 3% |

Fact shares are over the 21 texts with at least one number, name or quote. The full tables, with every detector, NLI both ways, the intervals and run 2, are in `results/tables.md`.

### What the numbers say

- Facts. reright changed a number, name or quote less often than any other tool, in every cycle. Over all 99 texts it changed one in 15% of the 67 texts that have them, and the 9 tools in all three cycles changed one in 30% to 46%. Every paired interval against reright excludes zero. Part of this comes from the guard: when a rewrite fails the check, reright hands the text back as it was. That happened for 21% of all texts, and 20 of those 21 texts had a number, name or quote. On the 47 texts reright did rewrite, it changed a fact in 21% of them, and the other tools changed one in 26% to 43% of the same texts (`results/posthoc.md`).
- Quotes are reright's weak spot. In run 1 it changed a fact in 10 texts, and in 7 of them the only change was a reworded quotation. The guard checks numbers and names, not quotes. Leaving quotes out, reright changed a fact in 5% of texts (3 of 65).
- Meaning. reright's judge mean is 4.48 over 99 texts (4.60 in run 2). ReHumanize scores higher (4.75, interval +0.07 to +0.46), and 95% of its outputs fail at least one detector. HIX Bypass, SmartHumanizer, UndetectedGPT and Undetectable AI score 0.3 to 0.6 lower, and those intervals exclude zero. The rest score at least 1.0 lower. The order holds when the judge's low-effort retries are left out.
- Detectors. reright is in the middle. Over 99 texts, 30% of its outputs pass all four detectors (31% in run 2). HIX Bypass (64%) and SuperHumanizer (56%) pass more often, and the intervals exclude zero. Texts the guard hands back are unmodified AI text and almost always fail, so this number includes them. Human texts are the ceiling: 16% of the 124 controls fail at least one detector.
- Runs. The two runs differ by a few points on each metric, inside the intervals. On the November set, run 2 scored better on facts and the judge (5% and 4.70); we report run 1 as fixed in the protocol.

### Human controls

At the cuts, each detector flags 6 of the 124 controls (4.8%). The federal passages are flagged more often than the internal ones (any detector: 24% against 10%), so the cuts are not equally strict on every kind of human text. Cut values are in `results/tables.md`.

### Changes from the protocol

- Version. Production moved from v8 (`45e0a05`) to v9 (`aa7c6b0`) at 09:41, after the protocol commit and before any benchmark text was sent. v9 only changes GPU memory handling in `voice/`, and the Go pipeline is the same. We tested v9, the version users had. `logs/run1.app_*.txt` and `logs/run2.app_*.txt` show v9 before and after both runs.
- NLI windows. With 400-token premise windows, a text scored against itself got an entailment share near 0, because MNLI models are trained on short premises. Before any reright output existed, we changed the premise to windows of 4 consecutive sentences (stride 2). The same text against itself then scores 1.0.
- A lost first attempt. The first run 1 rewrote all 99 texts, but the output file was not written because the results folder was missing. Nobody saw those outputs. Its log is `logs/reright-run1-unsaved.log`. We ran run 1 again with the same code and version.
- Short controls. 25 human controls, all federal passages, have fewer than 250 words and were kept whole.
- Region. The scorer first asked for A100 GPUs in the US region and was queued for lack of capacity, so the region was dropped. The GPU type stayed the same.
- The approved protocol text has three wording slips from editing. "They use its data" should read "We use its data". RADAR was used in our internal tests and is left out only here. "Commercial licences" means the four detectors are licensed for commercial use.
- Added after seeing the results: `code/posthoc.py`, which counts near-unchanged outputs for every tool, recomputes the judge without low-effort retries and compares facts on the texts reright rewrote.

### Overlap with reright's own checks

reright's guard checks numbers and names, which is close to what the fact checker scores. That is the feature being measured, but it means reright is built to do well on that metric. Its sample picking also uses surprisal, which is related to what Binoculars and Fast-DetectGPT measure.

## Rerun on the updated production version (runs 3 and 4, 2026-11-06)

On the afternoon of 6 November production changed: some parts of a text are now kept by code instead of the model. Greetings and sign-offs, quotations of three words or more, template slots and title labels stay as written. Texts with headings, bold labels or a list are rewritten paragraph by paragraph. A spelled-out number must stay, and a capital the model drops from a name is put back. The record of that change is `docs/experiments/2026-11-06-rules/`. We then ran the 99 texts twice more with the same protocol, harness, scorers and cuts.

- Code: `main` at `6b20450`, Modal app `reright-voice` version v14, deployed with `VOICE_RULES=facts`. The server ran with `RERIGHT_VOICE_RULES=all`, as production does. `logs/run3.app_*.txt` and `logs/run4.app_*.txt` show v14 before and after both runs.
- Run 3 is the answer for this version, the way run 1 was for the morning one. Run 4 shows the drift. The site's tables are built from run 3 (`BENCH_REF=reright-run3` for `code/analyze.py`, `code/posthoc.py` and `code/site_build.py`; without it they still compare against run 1).

| over 99 texts | run 1 | run 2 | run 3 | run 4 |
|---|---|---|---|---|
| changed a number, name or quote (67 texts with one) | 15% | 15% | 3% | 3% |
| handed back unchanged | 21% | 20% | 4% | 8% |
| passes all four detectors | 30% | 31% | 49% | 51% |
| judge mean | 4.48 | 4.60 | 4.23 | 4.36 |

- Facts. Run 3 changed a fact in 2 texts. It dropped a "first", and it wrote "$30 billion" as "$30-billion", which the checker counts as a change. Every other tool still changed facts more often, and every interval excludes zero.
- Meaning. The judge mean went down, mostly because reright now rewrites about 20 texts it used to hand back, and a text handed back scores 5. Those texts score 4.62 in run 3. On the texts both versions rewrote, run 3 is 0.22 below run 1 [-0.48, +0.05] and run 4 is 0.08 below [-0.35, +0.15].
- Detectors. Against run 3, HIX Bypass passes all four more often and the interval excludes zero. For SuperHumanizer the interval now includes zero.

## Rerun on production v15 (runs 5 and 6, 2026-11-06)

That evening production changed again. A number, date or name the model changed is put back when nothing else around it changed, numbers keep the input's format, "first" after a determiner must stay, the form of address must stay, and masked quotations keep their quotation marks. The record is `docs/experiments/2026-11-06-rules2/`. We ran the 99 texts twice more with the same protocol, harness, scorers and cuts.

- Code: `main` at `7422752`, Modal app `reright-voice` version v15, deployed with `VOICE_RULES=facts,repair,ordinals,address`. The server ran with `RERIGHT_VOICE_RULES=all,repair,ordinals,address,quotemarks,learn`, as production does; the learned locks do nothing for the benchmark's user, who has no edits. `logs/run5.app_*.txt` and `logs/run6.app_*.txt` show v15 before and after both runs.
- Run 5 is the answer for this version, and the site's tables are built from it (`BENCH_REF=reright-run5`). Run 6 shows the drift.

| over 99 texts | run 1 | run 3 | run 5 | run 6 |
|---|---|---|---|---|
| changed a number, name or quote (67 texts with one) | 15% | 3% | 1% | 1% |
| handed back unchanged | 21% | 4% | 4% | 4% |
| passes all four detectors | 30% | 49% | 51% | 53% |
| judge mean | 4.48 | 4.23 | 4.41 | 4.23 |

- Facts. Run 5 changed a fact in 1 text: it dropped "first" from "first-time triathletes", which is not after a determiner and so not protected. Every other tool still changed facts more often, and every interval excludes zero.
- Meaning. Run 5 is 0.07 below run 1 [-0.15, +0.29 for run 1 minus run 5], and 0.09 above it on the texts both rewrote. Seven tools score lower with intervals that exclude zero; Undetectable AI is within the margin, and ReHumanize scores higher.
- Detectors. HIX Bypass passes all four more often, and the interval excludes zero.

## Rerun it

1. `code/fetch.sh <empty_dir>` downloads the pinned AIHB tables, the GoHumanize test set and the fact checker.
2. `python3 -I code/prepare.py <dir>/aihb data/` rebuilds `data/inputs.jsonl` and `data/aihb_outputs.jsonl`.
3. reright outputs: start `harness/bridge.py`, then run `harness/` (`go run`) on `data/inputs.jsonl`. This needs Modal access to the deployed Rewriter, so outside reright you can only check our outputs in `results/reright-run*.jsonl`.
4. `modal run code/score_modal.py` and `modal run code/judge_modal.py` score any output file. `code/facts.py` runs the fact checker.
5. `python3 -I code/analyze.py .` and `python3 -I code/posthoc.py .` print every table.

The 54 federal controls are in `data/humans_public.jsonl` in full. The other 70 are given as dataset pointers with a sha256 of the cut text, because those texts are not ours to share. With the federal part alone you can recompute cuts, but they will differ from ours.

## Files

- `data/inputs.jsonl`, `data/aihb_outputs.jsonl`: AIHB texts and outputs, reformatted (one line per text, fields selected). Data from AI Humanizer Benchmark (https://aihumanizerbenchmark.com), cycles 2026-09, 2026-10 and 2026-11, CC BY 4.0.
- `data/humans_public.jsonl`: the human controls.
- `data/pricing-2026-11-06.json`: prices and features of each tool from its public pages, with the URL and a short quote for every value.
- `results/`: reright outputs, every score, judge reply and fact unit, `tables.md`, `posthoc.md` and `summary.json`.
- `logs/`: run logs and the deployed version before and after each run.

Questions or a takedown request: reright@interpt.co.