my-hermes-skills

Which effort level should omp's small-model agents use? Claude Haiku 5.5 vs GPT-6 Luna effort-level benchmarks

Abstract

Background. omp delegates lookups, setup and reviews to small-model subagents, which were configured at the :max effort level. An earlier speed comparison of the two models at :max left open whether lower effort settings would serve these agents better.

Objective. To measure how effort level (medium, high, max) changes speed, quality and cost for Claude Haiku 5.5 and GPT-6 Luna on short omp agent tasks, and to compare with public long-task benchmarks.

Methods. A local harness ran 4 planted-defect and repository tasks x 2 models x 3 effort levels x 2 runs = 48 runs through omp -p --mode json, graded automatically[1]. Results were compared with VulcanBench Frontier v4 effort ladders[2, 3, 4] and the Artificial Analysis Intelligence Index[5, 6].

Results. Haiku:medium and Haiku:high scored 97% and 100% with median times of 18 s and 27 s, against 335 s and $0.203 per 8 runs for Haiku:max at the same 100% score (measured). Luna's median time stayed at 57-80 s across effort levels. On VulcanBench long tasks Haiku solved 22/23 at both xhigh and max, with xhigh cheaper ($0.84 vs $1.54 per task)[3]; Luna solved at most 12/23[4].

Conclusions. Haiku 5.5 is the stronger model, but :max is the wrong setting for omp's small-model agents: Haiku at medium or high scored as well as Haiku:max on these tasks, ran 10-20x faster, and cost a tenth as much. Luna's speed barely changes with effort level, but it gives up quality on harder work.

1. Introduction

omp assigns models to its subagents (scout, init, reviewer, sonic) through task.agentModelOverrides. Haiku 5.5 and GPT-6 Luna were both set to :max, the highest effort level, on the assumption that more reasoning gives better results. An earlier report compared the two models at :max for speed in omp[7]. This study varies the effort level instead, on short agent tasks typical of subagent work, and checks the result against public benchmarks on long software tasks.

2. Methods

2.1 Sources and search strategy

Measured data come from the local harness[1]. Public benchmarks: VulcanBench Frontier v4 pages for Haiku 5.5 versus frontier models[2], the Haiku 5.5 effort ladder[3] and the GPT-6 Luna effort ladder[4]; Artificial Analysis's comparison page[5] (fetched 2026-10-14; figures since updated) and its Haiku 5.5 launch article[6]. How the public sources were found was not recorded.

2.2 Evidence classification

2.3 Experimental setup

The harness ran each task through omp -p --mode json in a fresh session with a restricted tool set: 4 tasks x 2 models x 3 effort levels x 2 runs = 48 runs[1]. Six ran at a time. All 48 exited cleanly, and every run's session log confirms which model actually answered. Luna ran through the Command Code provider and Haiku through Anthropic. Hardware was not recorded.

Table 1. Task fixtures.

TaskWhat it measuresToolsHow it was scored
hardReview of a TTL cache plus a rate limiter with 4 planted defects: a deadlock from a non-reentrant lock, an unlocked get, an off-by-one in the limit, and a TTL refresh that turns the fixed window into a sliding oneread onlydefects found / 4
bughuntReview of orders.py with 5 planted bugs: last page skipped, mutable default argument, SQL injection, float money math, swallowed exceptionread onlybugs found / 5
fixEdit textutil.py until 7 unit tests pass (6 fail at the start)read, edit, bashtests passing / 7; 0 if the tests were edited
exploreAnswer 3 factual questions about this reporead onlyfacts correct / 3

2.4 Analysis

Grading was automatic: a regex match per planted defect, plus a real unittest run for the fix task. Misses were read through by hand. Score = mean fraction of defects, tests or facts found per setting. Times are medians, because six runs were affected by concurrency stalls (Section 5); with only two runs per cell, the per-task "median" is the average of two values. Cost per setting is the sum over its 8 runs; how cost was metered was not recorded.

3. Results

3.1 Local runs by effort level (RQ1)

Table 2. Overall results per setting, 8 runs each (measured).

SettingMedian timeAvg output tokensAvg turnsScoreCost for 8 runs
haiku-5-5:medium18 s1.7k2.697%$0.013
haiku-5-5:high27 s3.2k3.0100%$0.021
haiku-5-5:max335 s36.6k4.8100%$0.203
gpt-6-luna:medium58 s1.9k8.097%~$0
gpt-6-luna:high57 s1.8k7.8100%~$0
gpt-6-luna:max80 s2.5k8.6100%~$0

Table 3. Median time per task in seconds (measured; two runs per cell, so noisy).

TaskHaiku medHaiku highHaiku maxLuna medLuna highLuna max
hard2026861555786
bughunt18307925691109
fix24298138844602
explore9605763462715

3.2 Public benchmarks (RQ2)

Table 4. VulcanBench Frontier v4, 23 real software tasks per setting (third-party benchmark)[2, 3, 4].

EffortHaiku scoreHaiku solvedHaiku min/taskLuna scoreLuna solvedLuna min/task
Low84.413/2322.940.80/235.6
Medium84.013/2324.045.70/233.6
High88.017/2326.857.42/2315.6
Extra-high91.822/2319.078.811/2352.6
Max92.222/2327.781.412/2360.5

Artificial Analysis puts Haiku 5.5 (max) above GPT-6 Luna (max) on its Intelligence Index, 43 vs 38[5, 6]. Haiku also generates faster, but uses about 3.2x more output tokens per task[6], so a full answer takes about 3.5x longer[5]. Going from Haiku xhigh to max adds about 2 index points for about 1.8x the tokens[6].

4. Discussion

4.1 Principal findings

RQ1: Haiku:max spends most of its time thinking, not finding more; Haiku at medium and high is the fastest option overall, and Luna's many small tool calls mean even its quick runs need a minute. Effort level hardly changes Luna's speed, which fits the public data: Luna's extra effort shows up as more quality on hard tasks, not more speed. Haiku 5.5 is the stronger model in both the local runs and the public benchmarks.

4.2 Comparison with prior work

The local runs and VulcanBench differ on where extra effort stops paying. On short agent tasks like these, which take 2-8 turns, the cost of Haiku:max is pure thinking time, and the extra effort has nothing hard enough to pay off on. On VulcanBench's long tasks the extra reasoning does pay for itself up to xhigh, by solving more and avoiding dead ends[3]. In both sources, max is past the point of diminishing returns. The earlier :max-only speed report[7] measured the two models at a single effort level; this study adds the effort dimension.

4.3 Practical implications

RQ3: Table 5 gives the change applied in ~/.omp/agent/config.yml (task.agentModelOverrides); the previous config is backed up. For long, multi-file tasks, use Haiku at xhigh rather than max: it's the best option on VulcanBench[3].

Table 5. omp subagent model settings before and after this study.

omp agentBeforeNowWhy
scouthaiku-5-5:maxhaiku-5-5:mediumLookups and file finding. Medium answered explore questions in ~9 s.
inithaiku-5-5:maxhaiku-5-5:mediumProject setup is short and well-structured.
reviewerhaiku-5-5:maxhaiku-5-5:highMedium missed the deadlock once and high never did. 27 s median.
sonicgpt-6-luna:maxunchangedEffort barely affects Luna's speed, and max protects quality.

5. Limitations

6. Conclusion

For omp's small-model agents, Haiku 5.5 at medium or high matches Haiku:max on quality while running 10-20x faster at a tenth of the cost, so scout and init moved to medium and reviewer to high. Luna's effort level barely affects its speed and was left at max. For long, multi-file work, Haiku at xhigh is the better setting than max.

Data and code availability

Harness, fixtures and the 48 raw results: benchmarks/haiku-vs-luna-omp/ in this repository; rerun with python3 bench.py (see its README)[1].

References

  1. [1] Local runs: harness, fixtures and the 48 raw results (rerun with python3 bench.py, see its README): https://github.com/LaansDole/my-hermes-skills/tree/main/benchmarks/haiku-vs-luna-omp back to text 1 2 3 4 5
  2. [2] VulcanBench, Frontier v4: Claude Haiku 5.5 vs frontier (Oct 2026): https://vulcanbench.com/benchmarks/swe-v4-haiku55-vs-frontier.html back to text 1 2 3 4 5
  3. [3] VulcanBench, Frontier v4: Claude Haiku 5.5 effort ladder (Oct 2026): https://vulcanbench.com/benchmarks/swe-v4-haiku55-v324.html back to text 1 2 3 4 5 6 7 8 9 10
  4. [4] VulcanBench, Frontier v4: GPT-6 Luna effort ladder (Sep 2026): https://vulcanbench.com/benchmarks/swe-v4-gpt6-luna-v316.html back to text 1 2 3 4 5 6 7
  5. [5] Artificial Analysis, Claude Haiku 5.5 vs GPT-6 Luna comparison (fetched 2026-10-14; figures since updated): https://artificialanalysis.ai/models/comparisons/claude-haiku-5-5-vs-gpt-6-luna back to text 1 2 3 4 5 6
  6. [6] Artificial Analysis, Anthropic has released Claude Haiku 5.5 (Oct 7, 2026): https://artificialanalysis.ai/articles/claude-haiku-5-5 back to text 1 2 3 4 5 6
  7. [7] Earlier report, Haiku 5.5:max vs GPT-6 Luna:max speed in omp (2026-10-14): 2026-10-14-haiku-5-5-vs-gpt-6-luna-omp-speed.md back to text 1 2