UpdateFaster global search
captain

Blog

Navigating Jev for Regulated RAG

Edgar BabajanyanCTO & Co-Founder

Captain lets customers in privacy-regulated industries mask data before it goes into embeddings. Moving from an NER model to Jev-style models turned custom PII categories from a per-customer project into a plain-language description.

4 min read

TL;DR

Captain is a self-tuning file search engine for agents. As part of our service, we let customers in privacy-regulated industries mask data before it goes into embeddings. Our original PII masking relied on self-hosted Microsoft Presidio, an NER model that identified only 12 categories of sensitive data.

The NER bottleneck and Jev

We originally built our core indexing pipeline to run masking directly after document parsing, using self-hosted Microsoft Presidio combined with custom regex recognizers. This handled standard built-in entities like names, emails, phones, card numbers and government IDs reliably well. However, when customers requested industry-specific custom categories, we held back from shipping them. Verifying extensibility for each customer took a long time, and we needed to be sure it would generalize. Zero-shot, it typically performed at only 29.4% accuracy.

Switching to Jev-like models changed that problem, delivering near-perfect generalization while improving our baseline capabilities. We found we could accept categories as plain-language descriptions passed at index time, so customers can customize masking however they’d like. To maintain compliance, we built on the popular Kev model and self-hosted it for our customers, delivering nearly the same experience as Jev.

Comparing the old and new redaction engines

We started with 2,602 labeled documents holding 12,845 pieces of personal data (8,658 in built-in categories and 4,187 in extended categories), plus 4,837 look-alike negatives that should stay unredacted. The table below covers the built-in categories.

Benchmark on 2,602 documents with no custom masking instructions. Built-in category redaction rate: Presidio 71.2%, Kev-4B 90.2%, Jev 92.9%. Unmasked leaks: 1,859, 560 and 435 of 8,658. Non-PII accuracy: 94.1%, 96.0%, 97.8%. Category classification accuracy: 81.4%, 91.3%, 94.0%.

Even our built-in categories improved.

On names, SSNs and the other fields our tuned NER model was trained on, Jev redacted 21.7 percentage points more than Presidio did. We expected to trade some accuracy on known categories for flexibility, but the opposite happened. Kev was close behind at 19 points better.

How we use Jev for custom, promptable PII categories

Jev works by taking yes/no booleans (nouls) or multiple-choice questions (choices) and returning a probability. It doesn’t natively point to characters, which we need for masking.

We solved this by proposing candidate spans, starting from the first token of a chunk. That gives us a list of PII candidates, and we ask Jev two questions about each one, in context:

  1. Is this sensitive data?
  2. If yes, which of the PII categories is it?
A night-shift handover note with patient names, bed numbers, record numbers and a phone number masked, next to the custom fields PATIENT_RECORD and BED_NUMBER that defined them.

Where Jev still fails

Jev still doesn’t score 100%, though it gets us much closer. Its data privacy isn’t yet where it needs to be for regulated industries, so we still run our self-hosted NER model and our self-hosted Kev model. They give us stronger auditability of where sensitive data goes.

Edgar Babajanyan

CTO & Co-Founder · Captain

Edgar is co-founder and CTO of Captain, building self-tuning file search for AI agents.