Blog

Evaluating SID-2 + Captain for Agentic RAG

Edgar BabajanyanCTO & Co-Founder

SID-2 is a model trained to keep searching until it has what it needs to answer. We ran it on top of Captain, next to Haiku 5.5, Opus 5.5 and a single Captain query, on three public benchmarks.

7 min read

TL;DR

SID-2 recently launched a model trained to iteratively search until it has the data it needs to answer questions. They claim frontier accuracy while maintaining high efficiency. We tested it on top of Captain to see how it compares to purpose built agents assembled on Anthropic models, how their research improves retrieval, and what part of this industry it fits in.

We compared SID-2, Claude Haiku 5.5 and Claude Opus 5.5 against a single Captain query across three public benchmarks. On Wix help-center questions, SID-2 scored 0.796 nDCG@10 against 0.687 for the best single query, and was close to Opus 5.5 (0.789) at about 1/18 of its model cost. On scientific abstracts and Wikipedia tables, Captain out of the box was as good as or better than the agents, taking 0.6 to 0.7 seconds instead of 4 to 17 seconds.

SID-2 Swarm, using up to three agents, showed no clear gain over one SID-2 agent on the 40 questions per text dataset we tested.

Results

BenchmarkSID, querying Captain: SID-2medium effortSID, querying Captain: SID-2default effort
Search quality (nDCG@10): WixQA300 questions0.796Not run
Search quality (nDCG@10): SciFact300 claims0.825Not run
Search quality (nDCG@10): FeTaQA500 questions0.9250.930
Latency (median seconds): WixQA300 questions6.6Not run
Latency (median seconds): SciFact300 claims9.5Not run
Latency (median seconds): FeTaQA500 questions8.46.3
BenchmarkAnthropic, querying Captain: Opus 5.5Anthropic, querying Captain: Opus 5.5reranked toolAnthropic, querying Captain: Haiku 5.5
Search quality (nDCG@10): WixQA300 questions0.7890.7810.717
Search quality (nDCG@10): SciFact300 claims0.7990.8010.797
Search quality (nDCG@10): FeTaQA500 questions0.937Not run0.921
Latency (median seconds): WixQA300 questions14.515.14.7
Latency (median seconds): SciFact300 claims16.417.15.8
Latency (median seconds): FeTaQA500 questions6.0Not run4.3
BenchmarkCaptain 1-shot: vectorCaptain 1-shot: hybridCaptain 1-shot: hybridreranked 50Captain 1-shot: hybridreranked 100Captain 1-shot: keyword
Search quality (nDCG@10): WixQA300 questions0.6870.6190.6380.6260.328
Search quality (nDCG@10): SciFact300 claims0.8880.8320.8080.8040.639
Search quality (nDCG@10): FeTaQA500 questionsNot run0.8840.936Not runNot run
Latency (median seconds): WixQA300 questions0.60.60.70.80.4
Latency (median seconds): SciFact300 claims0.60.60.70.80.4
Latency (median seconds): FeTaQA500 questionsNot run0.50.7Not runNot run

SID and Anthropic models search the same Captain collection. Their scores are Captain plus an agent loop. nDCG@10, higher is better. Latency is median seconds per question, end to end, lower is better. One run per setting. The tinted cell is the best of its row. A dash is a setup we did not run on that benchmark. Methodology

What we compared

Every setup searches the same Captain collection. The key differences were in how the models navigated the data.

  • Captain 1-shot (direct API): one request to Captain's query API per question, with the top 10 documents back as the answer. We ran five configurations: hybrid, hybrid reranked at 50 and at 100 candidates, vector only and keyword only.[3][4]
  • SID-2 querying Captain: SID-2 at medium reasoning effort runs its search loop with two Captain-backed tools, vector search and keyword search, plus a read tool. Up to 10 turns, then it reports a ranked list.[1][5]
  • SID-2 Swarm querying Captain: the multi-agent version, capped at 3 agents, with effort fixed at high by SID.[2]
  • Haiku 5.5 and Opus 5.5 querying Captain: the same loop, prompt, tools and turn limit as SID-2, at medium effort.[5]

For the text benchmarks, we used SID's published Python reference implementation and SDK for the agent loop. We changed only the search backend, the first sentence of the system prompt and the example searches, as their documentation allows. Every request is listed under References at the end of this post.

The three datasets

  • WixQA: 6,221 Wix help-center articles, with questions written the way customers ask them. We scored 300 questions.
  • SciFact: 5,183 scientific abstracts, where each question is a claim to verify. All 300 test claims.
  • FeTaQA (via the TARGET benchmark): 4,222 CSV files with one Wikipedia table each, and 500 questions that each need one specific table.

WixQA and SciFact both appear in SID's own SID-2 technical report, which gave us a check on our setup. Our SID-2 scores landed at 0.796 (WixQA) and 0.825 (SciFact) against their published 0.859 and 0.894. Their per-dataset samples were 20 and 43 questions, which they put at roughly ±0.10 to 0.15, so we're inside that range, though below their point estimates on both.

Help-center articles

WixQA questions are written the way customers talk, rather than the way help-center articles are written. A single vector query found the right article in the top 10 for 88% of questions, but often did not rank it first.

The agent loop helped most on support questions.

SID-2 placed the right article first 58% of the time, compared with 44% for the best single query. Against Captain 1-shot with reranking, SID-2 was ahead by 0.158 nDCG@10 (95% interval 0.123 to 0.192), and Opus 5.5 by 0.151. SID-2's model cost was $0.0024 per question, compared with $0.043 for Opus.

Science abstracts

SciFact claims read more like the abstracts that support them. A single vector query scored 0.888 nDCG@10, compared with 0.825 for SID-2, 0.799 for Opus 5.5 and 0.797 for Haiku 5.5. On this dataset, the agent loop added time and model cost without improving the score.

There is a caveat on the Opus result. Opus 5.5's biology safety filter declined about half the claims before searching. We re-ran each declined question in full on Opus 5, then Opus 4.8, at the same settings. All three declined 12 of 300, which scored zero. Across the 288 answered questions, the Opus setup scored 0.833. Its time and cost include declined attempts. Haiku 5.5 declined six claims, which also scored zero.

Spreadsheets

On the FeTaQA tables, reranking made the difference. It took a single Captain query from 0.884 to 0.936 nDCG@10, close to Opus 5.5 at 0.937. The differences between agents and the reranked query were within the measured uncertainty: SID-2's difference was −0.005, with a 95% interval of −0.019 to +0.009.

This was our first round and used a simpler agent harness rather than SID's reference implementation. SID-2 used the API's default effort in the main run, and we re-ran it at medium effort separately.

SID-2 Swarm

We ran the swarm[2] on the first 40 questions of each text dataset, and compared it with one SID-2 agent and the best single query on those same 40 questions. The scores are in the Swarm view of the results figure.

The swarm scored 0.757 against 0.810 for one SID-2 agent on WixQA, and 0.885 against 0.881 on SciFact. It took about three times as long.

That is no clear gain on the questions we tested, not a verdict on swarms. SID's report shows its largest swarm gains on multi-hop questions and on corpora such as BrowseComp-Plus and arXiv, which we did not test. The swarm API does not report token usage, so we cannot calculate its cost per question.

What we found

An agent loop is not automatically better than a single search. On two of the three datasets we tested, one Captain query was as good as or better than the agents. On support questions, where customer wording differs from the article, SID-2 improved nDCG@10 by 0.109 over the best single query for roughly a quarter of a cent in model cost per question.

We would start with one Captain query and measure it on a few hundred questions from the actual use case. Where that misses, test an agent loop. SID-2 is a useful candidate: it was close to Opus 5.5 on WixQA at about 1/18 of the model cost, and on SciFact at about 1/11. It did not decline any questions.

Reranking also depends on the corpus. It added about five nDCG points on the spreadsheet set, but scored five to eight points below plain vector search on the two text sets. For SID-2, the model accounted for about 1.4 to 2.5 seconds per question. Searches and the 100-requests-per-minute workspace limit also affected end-to-end speed.

Request settings

Every request the benchmarks sent

Every request the benchmarks sent, trimmed to the fields that differ between setups. Haiku 5.5 and Opus 5.5 ran the loop in[1]at medium effort with up to 16,000 output tokens per turn, with SID's tool examples written into each tool's description, since Claude has no equivalent field.

[1]SID-2 request

SID's recommended request, every turn. The system prompt is SID's word for word after its first sentence, which names the corpus. Tool schemas are SID's reference ones, and snippets come from sid-sdk 0.2.1. At most 10 turns, and a run with no report scores zero.

{
  "model": "sid-2",
  "reasoning": {"effort": "medium"},
  "instructions": "<SID's recommended system prompt, first sentence names the corpus>",
  "tools": ["search", "text_search", "read", "report_helpful_ids"],
  "sid": {"tool_examples": ["..."], "reward": "ndcg"},
  "store": false,
  "include": ["reasoning.encrypted_content"],
  "max_output_tokens": 4096
}
[2]SID-2 Swarm request

Sent over SID's WebSocket API with SID's last-turn message. Agent reports are fused with reciprocal rank fusion (k = 60), as SID's docs prescribe.

{
  "type": "response.create",
  "model": "SID-2-Swarm",
  "instructions": "<same system prompt as SID-2>",
  "tools": ["search", "text_search", "read", "report_helpful_ids"],
  "sid": {"tool_examples": ["... plus fork, message, list, wait"], "reward": "ndcg"},
  "multi_agent": {"enabled": true, "max_agents": 3, "agent_max_turns": 50},
  "max_output_tokens": 4096
}
[3]Captain 1-shot query

Hybrid as shown. Vector and keyword are the same request with semantic_ratio 1.0 and 0.0.

{
  "query": "<question>",
  "limit": 10,
  "max_chunks_per_document": 1,
  "semantic_ratio": 0.6,
  "rerank": {"enabled": false}
}
[4]Captain 1-shot query, reranked

Reranked at 50 candidates as shown, and at 100 with candidate_limit 100.

{
  "query": "<question>",
  "limit": 10,
  "max_chunks_per_document": 1,
  "semantic_ratio": 0.6,
  "rerank": {"enabled": true, "candidate_limit": 50}
}
[5]The agents' search tools

Each tool call is one Captain query. The agent picks the limit, default 5. In the reranked-tool variant, search sends semantic_ratio 0.6 with rerank enabled at 50 candidates.

search:      {"query": "<agent's query>", "limit": 5, "max_chunks_per_document": 1,
              "semantic_ratio": 1.0, "rerank": {"enabled": false}}
text_search: {"query": "<agent's query>", "limit": 5, "max_chunks_per_document": 1,
              "semantic_ratio": 0.0, "rerank": {"enabled": false}}

Edgar Babajanyan

CTO & Co-Founder · Captain

Edgar is co-founder and CTO of Captain, building self-tuning file search for AI agents.