Every model release trumpets benchmark scores, but few people check them — partly because running benchmarks looks fiddly. It isn’t, any more. The EleutherAI lm-evaluation-harness is the open-source standard behind the Hugging Face Open LLM Leaderboard: one CLI, 300+ standardised tasks, reproducible numbers. In this guide you’ll install it, run a real evaluation on an open model, interpret the results, and learn why your scores might not match the model card.
This is for anyone choosing between open models, or checking whether a fine-tune actually improved anything. You need a GPU for reasonable speed, but small models run on modest cards.
Prerequisites
- Python: 3.10 or newer.
- GPU: NVIDIA with 8 GB+ VRAM recommended. CPU evaluation works for tiny models but is slow.
- Disk: a few GB for model weights plus benchmark datasets (downloaded automatically).
- Hugging Face access: for gated models like Llama, log in with
huggingface-cli login.
Step 1: Install the harness
pip install lm-eval
lm_eval --help | head -20One version note: lm-eval 0.4 and newer use subcommands — lm_eval run to evaluate, lm_eval ls tasks to list tasks. Older tutorials show the bare lm_eval --model ... form; if your install rejects subcommands, you’re on an old version and the bare form is correct. Check with pip show lm-eval.
Step 2: List tasks and pick a suite
lm_eval ls tasks | head -40Hundreds of tasks appear. For a first evaluation, a small, informative suite covers the basics:
hellaswag— commonsense reasoning (10k questions, quick).arc_easy/arc_challenge— grade-school science questions.truthfulqa_mc2— does the model prefer true answers over plausible-sounding falsehoods.gsm8k— grade-school maths word problems, tests step-by-step reasoning.mmlu— the famous one: 57 subjects, ~14k multiple-choice questions. Slow; save it for the full run.
Step 3: Run a smoke test
Start tiny — 50 samples, one fast task — to validate your whole setup before committing to a full run:
lm_eval run \
--model hf \
--model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct,dtype=bfloat16 \
--tasks hellaswag \
--num_fewshot 10 \
--batch_size auto \
--limit 50 \
--output_path results/smokeExpected output ends with a results table:
|Tasks|Version|Filter|n-shot|Metric| |Value| |Stderr|
|hellaswag|1|none|10|acc|↑|0.5123|±|0.0051|acc is accuracy; acc_norm (length-normalised) is the fairer number for multiple-choice. The ± column is the standard error — if two models differ by less than ~2 standard errors, the gap is noise.
A word on that normalisation, because it confuses everyone once: raw acc picks whichever answer choice the model assigns the highest total probability, which biases toward short answers (fewer tokens, less probability mass to spread). acc_norm divides by answer length first, removing the bias. Leaderboards standardised on acc_norm for MMLU-family tasks years ago, so if you’re comparing against published numbers, make sure you’re reading the same column — mixing acc and acc_norm across sources is one of the most common ways benchmark comparisons go wrong.
Step 4: Run a real comparison
The harness shines when comparing models apples-to-apples. Evaluate a base model and its instruct variant on the same suite:
# Base model
lm_eval run --model hf \
--model_args pretrained=Qwen/Qwen2.5-7B,dtype=bfloat16 \
--tasks hellaswag,arc_challenge,truthfulqa_mc2 \
--num_fewshot 10 --batch_size auto \
--output_path results/base
# Instruct variant — note the chat template flag
lm_eval run --model hf \
--model_args pretrained=Qwen/Qwen2.5-7B-Instruct,dtype=bfloat16 \
--tasks hellaswag,arc_challenge,truthfulqa_mc2 \
--num_fewshot 10 --batch_size auto \
--apply_chat_template \
--output_path results/instructResults land as JSON in the output directories, including per-sample details with --log_samples. Diff the two and you have a publishable comparison. Keep the task versions and few-shot counts identical between runs — the harness versions tasks precisely so results stay comparable over time.
Step 5: Read the results file
Each run writes a timestamped JSON file. The interesting parts:
python3 - <<'EOF'
import json, glob
path = sorted(glob.glob("results/instruct/*.json"))[-1]
data = json.load(open(path))
for task, metrics in data["results"].items():
acc = metrics.get("acc,none")
err = metrics.get("acc_stderr,none")
print(f"{task:20s} acc={acc:.4f} ± {err:.4f}")
print("task versions:", data["versions"])
print("few-shot:", data["n-shot"])
EOFAlways report the task version, few-shot count, and whether you used the chat template alongside the score. A bare "72.4 on MMLU" is meaningless without them — which is why half the numbers quoted online can't be reproduced.
Which benchmarks answer which question
Not every benchmark suits every decision. A rough map: choosing a general chat model — MMLU plus MT-Bench-style human preference data; choosing a coding model — HumanEval and MBPP, and run them with the model's chat template since that's how you'll use it; checking a fine-tune for regressions — the same small suite before and after, because absolute scores matter less than the delta; testing reasoning — GSM8K and ARC-Challenge, where small models separate clearly from large ones. TruthfulQA is worth including whenever the model will answer factual questions unsupervised — it's the closest thing to a "does it make things up" probe in the standard set.
And know what benchmarks don't tell you: instruction-following quality, tone, refusal behaviour, and anything about your specific domain. A model can top MMLU and still be useless at your task. Benchmarks are a filter, not a verdict — they rule out weak models cheaply, but the final call always needs evaluation on your own data.
Where things go wrong
Scores far below the model card. The most common cause is prompt formatting: for instruct/chat models you must pass --apply_chat_template, otherwise the model sees a malformed prompt and scores collapse. Second most common: wrong --num_fewshot.
Out of memory. Use a smaller model, add ,load_in_4bit=True to --model_args, or evaluate against a vLLM server with --model vllm instead of loading weights into the harness.
MMLU takes forever. It's ~14k questions. That's normal — run it overnight, or use --limit for iteration and the full task only for final numbers.
"Task not found". Task names change between harness versions (mmlu vs mmlu_flan_n_shot etc.). Use lm_eval ls tasks on your installed version rather than copying names from old blog posts.
Wrap-up
You can now produce benchmark numbers you'd actually defend: versioned tasks, fixed few-shot counts, standard errors, and a documented prompt format. Use this whenever a model choice matters — picking a base model, validating a fine-tune, or deciding whether a quantised model kept its accuracy (our quantisation guide pairs naturally with it). Benchmarks don't measure everything, but they beat vibes every time.
Featured image: Greenstrat via Wikimedia Commons (Public domain)
