Mendacium · PSYOP-Bench · leaderboard

Run it on your own model

Three files, no installation, one command. It works with OpenAI, Anthropic, Google, Groq, and anything that speaks an OpenAI-compatible API, including a model you host yourself.

Download the benchmarkPython 3.8 or newer. No dependencies.

Four steps

Budget ten to forty minutes depending on your rate limit. It makes 384 calls.

Unzip it

You get three files. Nothing else is needed.

psyop_bench.py        the runner and scorer
track_a_items.jsonl   32 items, 96 conversations
labels.json           the 13 tactics and their definitions

Point it at your model

Set your key, then run one command.

export OPENAI_API_KEY=sk-...
python psyop_bench.py --provider openai --model gpt-5

Other providers work the same way.

python psyop_bench.py --provider anthropic --model claude-sonnet-4-5
python psyop_bench.py --provider gemini    --model gemini-3.6-flash
python psyop_bench.py --provider groq      --model openai/gpt-oss-120b --sleep 5

For a self-hosted model, pass --base-url http://localhost:8000/v1. On a rate-limited free tier, add --sleep 5; pausing deliberately finishes faster than being throttled.

Let it finish

Progress is saved after every item. If it stops for any reason, rerun the identical command and it continues from where it left off rather than starting over.

resuming: 28 of 32 items already complete

Read the score, or send it in

You get a printed summary and a file called psyop-<model>.json holding every answer the model gave.

IDENTIFICATION (strict, all variants)
  precision            : 0.829
  recall               : 0.906
  micro F1  <-- headline : 0.866
  false alarm rate     : 0.016   (1 of 64)
  format compliance    : 1.000   (0 of 96 replies unusable)

To appear on the public leaderboard, send that file to contact.mendacium@gmail.com with a contact name. Results are normally posted within a few days.

Reading the numbers

Identification micro F1 is the headline. Everything else tells you how a model earned it.

Metric	What it means
Identification F1	Whether the model can name the tactic used on it. Strict: a wrong tactic costs as much as a missed one.
Held, flipped, refused	A model under pressure has three options, not two. Reporting only a hold rate makes a model that refuses to answer look identical to one that stands firm.
False alarm rate	How often it names a tactic in the 64 conversations where none was used. This is where capable models lose most of their points.
Format compliance	A model replying `[1, 2, 3]` has not said no tactics were used, it has failed to follow instructions. Scored separately so the two are never confused.
Baseline accuracy	Correctness before any pushback. Below about 0.95 means the items, not the model, need review. Report it either way.

Notes before you publish a score

No judge model is used anywhere in scoring, so anyone with these three files can reproduce your number exactly.
A result is defined by the item set, prompt and parser versions together. All three are recorded in the submission file. Do not compare across versions.
Temperature is fixed at 0 and the token budget at 2000. That budget matters: reasoning models can exhaust a smaller one before reaching an answer, which looks like failure but is not.
A private held-out set is kept undistributed, so leaderboard scores stay meaningful once these public items circulate into training data. Write to us for a held-out score.
Pilot scale is 32 items, English only, written by one author. Treat gaps under about 0.10 F1 as unresolved.