my-hermes-skills

Does an expert persona in the system prompt help or hurt current models? An A/B test and literature review

Abstract

Background. Many prompts and skills start with "You are a senior engineer" or "You are an expert in X". Earlier studies suggest such personas do not improve accuracy and can lower it, but current models may behave differently.

Objective. To test whether a generic expert persona, or a scoped brief without an identity claim, changes accuracy, cost or speed on current models, and to summarise what research and vendors now recommend.

Methods. A local A/B test ran 446 valid runs through omp -p, comparing three system-prompt arms (none, expert, scoped) on Claude Fable 5.2 and GPT-6 Luna at medium effort, on MMLU-Pro questions and planted-defect coding tasks[1, 2]. Paired MCQ answers were compared with an exact two-sided sign test. Two studies and a press report[3, 4, 5], vendor prompting docs[6, 7, 8] and two secondary syntheses[9, 10] were reviewed.

Results. With 420 paired MCQ answers per model, the expert persona changed accuracy by +1.0 pp on Fable (p = 0.34) and +0.2 pp on Luna (p = 1.00) (measured). Coding tasks scored 97-100% in every arm. Persona arms produced 10-25% more output tokens on coding runs. Prior work found no gain across 162 roles[5] and an accuracy loss on MMLU (68.0% vs 71.6%)[3].

Conclusions. A generic "You are a world-class expert" persona neither helped nor hurt either model in 446 runs: every accuracy difference was within noise (sign-test p ≥ 0.34). It is not a hindrance on these models, but it buys nothing and can be dropped from skills; roles that state scope, priorities and format remain worth keeping.

1. Introduction

Persona lines are a habit carried over from earlier models. Zheng et al. tested 162 roles on 4 model families and found that "adding personas in system prompts does not improve model performance"[5]. Hu, Rostami and Thomason (USC, March 2026 preprint) found that expert personas improve alignment but damage accuracy[3, 4]. Vendors still recommend a role, framed as scope[6, 7], and Anthropic advises reviewing prompts written for earlier models[8]. Whether a persona still matters on the models used daily here, Claude Fable 5.2 and the GPT-6 family, decides whether skills should keep these lines. GPT-6 Astra could not be tested, because no configured provider serves it; GPT-6 Luna stands in for the GPT-6 family.

2. Methods

2.1 Sources and search strategy

Web research on 2026-11-14 covered 2 peer-reviewed or preprint studies[3, 5], press coverage of one[4], Anthropic and OpenAI prompting docs[6, 7, 8], and 2 secondary syntheses[9, 10]. Search terms and inclusion criteria were not recorded. MCQ items come from the MMLU-Pro dataset[2].

2.2 Evidence classification

2.3 Experimental setup

Three arms were appended to omp's default system prompt; the exact texts are in benchmarks/persona-ab/bench.py[1].

Table 1. Experimental arms.

ArmAppended text
noneNothing. This is the control.
expert"You are a world-class senior software engineer with 20+ years of experience... you never miss a bug" for code. "You are a world-class expert in {subject}... You always get the right answer" for exam questions, matched to each question's subject.
scopedNo identity claim. A brief instead: the context, priorities in order, what to compare against, and the output format.

Models: Claude Fable 5.2 and GPT-6 Luna, both at medium effort, run through omp -p. Tasks: MMLU-Pro questions with 10 options[2], and the planted-defect review and test-fixing coding tasks from the 2026-10-14 harness (3 tasks, 9 runs per arm). The MCQ set was run in two phases: phase 1 (140 answers per arm) and phase 2 (280 new questions for none and expert). omp's advisor was off for every run (omp-overlay.yml). Every run was checked to have been answered only by the requested model. One run retried after a network error. How the 446 runs map onto the MCQ answer counts (answers per run, number of reps) was not recorded.

2.4 Analysis

3. Results

3.1 Accuracy (RQ1)

Table 2. Accuracy by arm (measured). MCQ = MMLU-Pro questions with 10 options[2]. Coding = the planted-defect review and test-fixing tasks from the 2026-10-14 harness.

ModelTasknoneexpertscopedRuns per arm
Fable 5.2MCQ93.1%94.0%93.6%420 / 420 / 140 answers
Fable 5.2Coding (3 tasks)100%*100%100%9 runs
GPT-6 LunaMCQ82.6%82.9%80.7%420 / 420 / 140 answers
GPT-6 LunaCoding (3 tasks)100%97.2%100%9 runs

*The automatic grader scored one Fable "none" review at 3/4. The answer did find the unlocked get, but wrote "get reads..." without the parentheses the regex expects; it was manually adjudicated to 4/4. Luna's one expert-arm miss (it did not report the unlocked get) is real.

Table 3. Paired MCQ comparison against the control (measured; p derived, exact two-sided sign test).

ModelArmAnswersControlArmΔControl right onlyArm right onlyp
Fable 5.2expert42093.1%94.0%+1.0 pp370.34
Fable 5.2scoped14092.9%93.6%+0.7 pp231.00
GPT-6 Lunaexpert42082.6%82.9%+0.2 pp13141.00
GPT-6 Lunascoped14080.7%80.7%+0.0 pp331.00

3.2 Cost and speed (RQ2)

Table 4. Cost and speed (measured); medians for wall time, means for output tokens and cost.

ModelTaskArmWall timeOutput tokensCost per run
Fable 5.2MCQnone5.4 s354$0.076
Fable 5.2MCQexpert5.8 s418$0.097
Fable 5.2codingnone19.3 s1,461$0.179
Fable 5.2codingexpert22.1 s1,723$0.179
Fable 5.2codingscoped23.5 s1,833$0.196
GPT-6 Lunacodingnone36.6 s2,824$0.004
GPT-6 Lunacodingexpert41.1 s3,278$0.005

Total spend: $22.38. Fable cost $21.82 and Luna $0.56.

3.3 Research and vendor guidance (RQ3)

4. Discussion

4.1 Principal findings

RQ1: the expert persona had no measurable effect on accuracy for either model (Fable +1.0 pp, p = 0.34; Luna +0.2 pp, p = 1.00; 420 paired answers each). The phase-1 swing for Luna, which reversed in phase 2, shows how small samples move, which is why the pooled result is the one to trust; the per-subject swings are likewise noise. The coding tasks hit the ceiling, so the coding result is "no harm seen", not "no effect". The scoped brief did not change accuracy either, which is expected: the research places its value in alignment (format, priorities and project fit), and these tests only measure correctness.

RQ2: personas made answers slightly longer with no accuracy gain. The Fable MCQ cost gap is a caching artifact, not a persona cost.

4.2 Comparison with prior work

This does not reproduce the USC result: Hu et al. found expert personas lowered MMLU accuracy (68.0% vs 71.6%)[3, 4], while here neither model lost accuracy. The likely reason is model strength: both models here score 83-94% on harder MMLU-Pro questions, and strong models seem to ignore an identity line that carries no information. ASSUMPTION: this explanation is inferred and was not tested. The null result agrees with Zheng et al.[5] and with vendor guidance that treats a role as scope rather than as a performance lever[6, 7].

4.3 Practical implications

Table 5. Recommendation for skills and prompts.

PatternKeep?Why
"You are a world-class / senior / expert X" with nothing elseDropNo accuracy effect in 446 local runs or in Zheng et al.[5]. USC found harm on weaker models[3]. It also adds output length.
A role used as scope ("You review PRs for a Rust CLI used by on-call SREs")KeepAnthropic still recommends it[6]. It carries real context.
Concrete priorities, constraints and output formatKeepThis is where the research finds gains: alignment and format[3]. It costs nothing in accuracy (no change here).
Persona for tone, voice, safety or refusal policyKeepUSC measured alignment gains, e.g. +17.7 pp jailbreak refusals[3].
"CRITICAL", "MUST", "never miss" wordingSoftenCurrent models over-trigger on it[6].

5. Limitations

6. Conclusion

On Claude Fable 5.2 and GPT-6 Luna, a generic expert persona made no measurable difference to accuracy in 446 runs, added some output length, and so can be dropped from skills and prompts. Roles that carry real scope, priorities, format or tone remain useful, in line with vendor guidance and with research showing alignment rather than accuracy benefits.

Data and code availability

Harness, fixtures and raw data: benchmarks/persona-ab/ (bench.py, analyze.py, results-2026-11-14.jsonl)[1].

References

  1. [1] Local harness, fixtures and 446 raw runs (bench.py, analyze.py, results-2026-11-14.jsonl): https://github.com/LaansDole/my-hermes-skills/tree/main/benchmarks/persona-ab back to text 1 2 3 4
  2. [2] TIGER-Lab, MMLU-Pro dataset (Hugging Face): https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro back to text 1 2 3 4
  3. [3] Hu, Rostami, Thomason, "Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM" (arXiv, Mar 2026): https://arxiv.org/abs/2603.18507 back to text 1 2 3 4 5 6 7 8 9 10
  4. [4] The Register, "Telling an AI model that it's an expert makes it worse" (Mar 24, 2026): https://www.theregister.com/software/2026/03/24/telling-an-ai-model-that-its-an-expert-makes-it-worse/5226049 back to text 1 2 3 4 5 6
  5. [5] Zheng et al., "When 'A Helpful Assistant' Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of LLMs" (EMNLP Findings 2024): https://arxiv.org/abs/2311.10054 back to text 1 2 3 4 5 6 7 8
  6. [6] Anthropic, Prompting best practices (Claude Platform Docs): https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices back to text 1 2 3 4 5 6 7 8
  7. [7] OpenAI, Prompting guide (API docs): https://developers.openai.com/api/docs/guides/prompting back to text 1 2 3 4 5 6
  8. [8] Anthropic, Prompting Claude Fable 5.2: https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-2 back to text 1 2 3 4 5
  9. [9] Learn Prompting, "Revisiting Roles in Prompting" (Schulhoff, Aug 2024): https://learnprompting.org/docs/intermediate/revisiting_roles back to text 1 2 3 4
  10. [10] PromptForge, "Best Practices for Prompting Frontier AI Systems" (Bagalman, May 2026): https://github.com/mbagalman/PromptForge/blob/main/guides/prompting-best-practices-2026.md back to text 1 2 3 4