Detection pipeline

1. Windows

Text is cut into windows of 256 model tokens with a 48-token overlap, on line boundaries where possible, using the entity model’s own tokenizer so that no window exceeds the model’s sequence limit. Up to four windows are processed at once. Spans are reported with byte offsets into the original text.

2. Rules and the entity model, in parallel

Each window goes to the regex service and the model service at the same time.

The regex service runs the format rules: checksum validators for TCKN, VKN, IBAN, cards, IMEI and ICCID, numbering-plan checks for phone numbers, structural checks for IP addresses, coordinates and plates, and label-gated rules for identifiers that only count when a caption such as “Müşteri No” or “Passport” is nearby. It also runs the custom rules published for the tenant, read from the database on every request. The detection classes page lists every rule.

The entity model tags people, organisations and locations in free text, with a confidence per span.

Results are merged: identical spans are deduplicated with the higher confidence, and a rule span takes precedence over a model span for the same text.

3. The normaliser

A versioned, deterministic pass with no model in it. It splits spans that cross table cells or lines, unions overlapping spans, strips titles and punctuation, merges a name split across cells, composes a street address out of a location plus its unit nouns and numbers, retypes a location to a place of birth or a nationality when the surrounding words say so, and recognises machine-readable passport zones by their check digits. It also drops junk: abbreviations, protocol constants, defined terms and common nouns that the model mistook for names, using a Turkish lexicon. Rule spans pass through untouched.

4. The role layer

When the text contains a date or a digit run, the role model is asked one question: which of these is a date of birth? It adds a date_of_birth span only where the engine has none, trimmed to the date itself. It never removes or retypes an existing span.

5. The verifier

Every span the entity model produced, person, organisation, location or address, is re-scored by the verifier in context: the span’s own line, the table header above it when the line is tabular, and the neighbouring lines for prose. The verifier returns a probability that the span is junk; above the release’s fixed threshold the span is dropped. Rule spans and role spans are not scored and cannot be dropped. The scope of what the verifier may drop is a deployment setting.

6. The type judge (optional, off by default)

With the role model it can also arbitrate between person and organisation for a span when the entity model was unsure. It changes the type only, never the boundary or whether the span is masked.

7. Clustering and rendering

Contained spans are removed. Spans are clustered by class and normalised value, with case, whitespace and Turkish diacritics folded, so one entity is one cluster. A bare surname joins the full name that contains it. Each cluster gets one token, and replacement is rendered over the original bytes, so nothing outside the spans changes. Documents are rewritten in place.

8. Context store and audit

In vault mode with a context, tokens are written to the vault. The run and every span, with type, offsets, confidence, source and verifier verdict, are written to the audit store.

Determinism and failure

Rules and the normaliser are deterministic. Thresholds are constants of the release. Model files are pinned. The cache key includes the release version, the tenant, the custom-rule revision and the context, so a result is reused only when all of those match. If any layer fails to answer in full, the request fails with 503; a window that would exceed the model’s sequence limit fails with 422. Degraded results are never cached.