LiteLLM guardrail

How it works

LiteLLM stays your gateway: it holds the provider credentials, routes the model call and serves your applications. On every request it calls Salus twice. Pre-call, Salus tokenises the prompt and records which tokens were used in this call. Post-call, Salus restores those tokens in the answer and tokenises any new sensitive value the model produced, with restore disabled for it.

The endpoint is POST /beta/litellm_basic_guardrail_api, authenticated with a Salus API key that has the litellm_guardrail permission. Both Authorization: Bearer and x-api-key are accepted on this route.

Gateway configuration

guardrails:
  - guardrail_name: salus-pii
    litellm_params:
      guardrail: generic_guardrail_api
      mode: [pre_call, post_call]
      api_base: http://salus-gateway-service:8088/beta/litellm_basic_guardrail_api
      api_key: os.environ/SALUS_API_KEY
      default_on: true
      fail_on_error: true
      unreachable_fallback: fail_closed
      streaming_transform_mode: incremental_diff
      streaming_end_of_stream_only: false
      streaming_sampling_rate: 5

fail_on_error and unreachable_fallback: fail_closed make the gateway stop the request when Salus is unavailable. streaming_transform_mode: incremental_diff is required for streamed answers. LiteLLM 1.99.1 or newer is supported; the release is validated against 1.100.1.

If default_on is false, a client opts in per request with guardrails: ["salus-pii"].

Conversations

Clients send x-litellm-trace-id on each request of a conversation. LiteLLM passes it to Salus as litellm_trace_id, and Salus uses it as the vault context, so turn three sees the same tokens as turn one. Without a trace id the call id is used, which scopes tokens to a single request. A request with text but no identity at all is rejected with 422.

from openai import OpenAI
client = OpenAI(base_url="https://litellm.example.internal", api_key=LITELLM_KEY)
r = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Müşteriye yanıt taslağı hazırla: Ayşe Demir, TCKN 12345678901"}],
    extra_headers={"x-litellm-trace-id": "ticket-48213"},
)

The application sees the restored answer. The provider saw PERSON_kxmrqa and NATIONAL_ID_wbqtrz.

Streaming

For a streamed answer LiteLLM calls Salus repeatedly with the accumulated text. Salus keeps one buffer per call and choice, screens a rolling window, commits restored text in steps, and returns stream_holdback_chars: how many trailing characters the gateway must hold back because they might be the start of a token. The stream stays live; the hold-back is bounded by the length of one token.

Because that buffer lives in the engine’s memory, run a single replica of the main service for guardrail traffic.

What is blocked

Requests that carry images, tools or functions, and responses that carry tool calls, are answered with BLOCKED and a reason: those payloads cannot be tokenised, so they are not allowed through. The Responses API in streaming mode is not supported.

Direct test

curl "$SALUS_BASE_URL/beta/litellm_basic_guardrail_api" \
  -H "Authorization: Bearer $SALUS_API_KEY" -H 'content-type: application/json' \
  -d '{"texts":["mail [email protected]"],"input_type":"request","litellm_call_id":"call-1","litellm_trace_id":"conversation-1"}'
# {"action":"GUARDRAIL_INTERVENED","texts":["mail EMAIL_dlwqzb"]}

input_type is request for pre-call and response for post-call. Any other value, or an empty texts, returns {"action":"NONE"}.