Vigil · Sahasa.AI

01 / Evaluations

Golden datasets, judged answers, and agents followed to the end of the task.

Build a dataset from real traffic or write one. Score every response with an LLM-as-judge you can inspect and override. For agents, we follow the whole run: did it call the right tools, in the right order, and finish the task?

Datasets and experiments versioned alongside your prompts
Human annotation queues when the judge isn't enough
Regression on every prompt or model change

vigil · experiment run support-agent v14 vs v15

# 240 cases · judge: claude · seed 7
task_success      v14 0.81   v15 0.88   Δ +0.07  p<0.01
tool_order_ok     v14 0.93   v15 0.96   Δ +0.03
hallucinated_ref  v14 0.04   v15 0.05   Δ +0.01  not significant
pii_leak          v14 0      v15 0
cost_per_case     v14 $0.021 v15 $0.034 Δ +62%
# verdict: ship v15 behind a 20% flag; watch cost

02 / Red teaming

Prompt injection, jailbreaks, data leakage. As a repeatable suite, not a one-off exercise.

Three phases: probe, escalate, confirm. Attacks are generated against your actual system prompt and tools, so the results are about your feature, not a generic model.

Runs on every release, like any other regression
Findings include the exact prompt that got through
Maps to OWASP LLM Top 10

RED TEAM · INDIRECT PROMPT INJECTIONmeasured

Instructions in a pasted invoice changed the agent's tool call

A crafted line in the "notes" field of an uploaded invoice caused the refund agent to call issue_refund with an attacker-supplied account. Confirmed in 6 of 10 attempts.

feature: invoice-assistant · phase 3 confirm · OWASP LLM01

DRIFT · PRODUCTIONmeasured

Refusal rate on billing questions doubled after provider model update

Week-over-week refusal rate moved from 2.1% to 4.6% with no prompt change. Provider changelog shows a model revision on the same day.

03 / In production

Traces, drift, safety and cost, per feature.

Instrument with our SDK, OpenTelemetry, or the adapters for LangChain, LangGraph, CrewAI, Bedrock and OpenAI. Vigil watches quality move over time, checks every response for PII and toxicity, and shows what each feature costs.

A/B tests on prompts, models and versions
Alerts when a metric leaves its band
Prompt playground with the same judges as CI

Feature	Quality (7d)	Safety	Cost / 1k	Status
support-agent	0.88 ▲	0 PII · 0 tox	$34	healthy
invoice-assistant	0.79 ▼	1 injection	$19	watch
search-summary	0.91 ─	0 PII · 0 tox	$6	healthy
onboarding-copilot	—	—	—	not instrumented

Your app has a chatbot in it now. Ordinary tests can't see inside it.

Golden datasets, judged answers, and agents followed to the end of the task.

Prompt injection, jailbreaks, data leakage. As a repeatable suite, not a one-off exercise.

Instructions in a pasted invoice changed the agent's tool call

Refusal rate on billing questions doubled after provider model update

Traces, drift, safety and cost, per feature.

Bring one repository. We'll show you what comes back.