Edge Routing for RAGEdge Routing for RAG

For developers connecting to our API from outside the United States, we realized the initial handshake with our servers was adding significant latency to every query. Requests to Captain’s API now connect to a local server near you, traveling over a private network backbone to reach us. This bypasses multiple long-distance round trips during the initial handshake.

Up to 39% lower latency on international queries.

To unlock these gains, we mapped out the journey of international customer queries. We built an edge routing mechanism that works seamlessly with the hostnames and configurations our customers already use. The improvement was immediate, particularly for 'cold' connections, which is how most agent-based tool calls interact with Captain.

We measured these gains using median response times from 100 samples per major international geography, using a fresh connection for every request to simulate real-world usage.

When a customer starts an interaction like a RAG process or database retrieval, they initiate a cold connection to Captain. When the tool call fires, the client connects, sends a request, receives a response, and disconnects. Before any search or data processing happens, the connection requires several round trips to negotiate encryption and establish the session. Each round trip travels the full distance between the user and our server twice.

From locations like Singapore, that latency adds up. A query that took a few hundred milliseconds of search work spent significantly more time on the handshake process, which carries no data. We realized that while our retrieval was fast, connection overhead was the primary bottleneck.

Our new edge routing allows the client to complete these initial round trips with the nearest edge location. The request then travels to Captain over our private backbone on an already warm connection. This preserves our existing search infrastructure and data residency while bringing the connection point much closer to our customers.

We measured v3 Query end-to-end from seven cities, before and after the change, with 100 samples per city and a fresh connection for every request. Each figure is the median total time from sending the request to receiving the full response.

With this improvement, the farther you are from us geographically, the more consistent the experience will be now. Even clients closer to us will see latency improvements (albeit smaller) because requests are still routed much more efficiently.

Edge Routing is live and enabled for every API request, on every plan. No need to update API code.