Docs
Product

PII redaction

Foam scrubs sensitive data in two places: directly in your code during installation, and — optionally — in the collector with NER.

In-code scrubbing ships with every installation. Collector-side NER redaction is optional and off by default — contact perla@foam.ai to enable it for your telemetry.

1 · In your code, at installation (always on)

During installation Foam reads each service and detects the call sites that emit sensitive values — an email in a log line, a patient name in a span attribute, an SSN in console output. The install PRs rewrite those call sites so the value is scrubbed at the source: the raw value never leaves your app. Anything Foam detects is scrubbed by default, with no configuration needed.

foam-install · PR: scrub sensitive fieldsdetected
services/checkout/handlers.ts
log.info(`checkout by ${user.email}`);
+log.info(`checkout by ${scrub.email(user.email)}`);
services/patients/record.ts
span.setAttribute('patient.name', patient.name);
+span.setAttribute('patient.name', scrub.phi(patient.name));
workers/claims/submit.ts
console.log('claim ssn', claim.ssn);
+console.log('claim ssn', scrub.ssn(claim.ssn));
Merged during installation — the raw values never leave your app.

2 · In the collector, with NER (optional)

In-code scrubbing only catches what is visible statically. As a second net, the Foam collector can run every incoming log and span through a NER analyzer and rewrite detected entities before anything is stored. This layer is opt-in — reach out to perla@foam.ai to enable it.

collector · POST /analyzelive
10:42:07 POST /v1/checkout 200 · 96ms user=sarah.chen@acme.com<EMAIL_ADDRESS>
10:42:08 patient.record updated name="James Miller"<PERSON> mrn=84-2210-77<MRN>
10:42:09 support.ticket created phone=(415) 555-0163<PHONE_NUMBER>
10:42:11 auth.login retried ip=73.92.14.208<IP_ADDRESS>
10:42:12 claim.submitted ssn=536-90-4143<US_SSN>
10:42:14 payment.method added card=4242 4242 4242 4242<CREDIT_CARD>
7 entities rewritten before storage — the raw values are never written to Foam DB.

Telemetry flows through the collector as a live tail. Values the analyzer recognizes as PII or PHI are rewritten to typed entity markers before anything is stored.

NER data flow

Your appemits traces + logs (OTLP)FOAM COLLECTORNER call: POST /analyzePresidio analyzer 2.2.362spaCy en_core_web_lg NER + pattern recognizersPII found?nostored unchangedyes — rewrite“call Maria at 555-867-5309”↓“call <PERSON> at <PHONE_NUMBER>”Foam DBfoam_otel_dbFoam featuresfoam.ai/chat · foam.ai/issues · foam.ai/monitors

Redaction is opt-in per environment and per service, and fail-closed: if a value can’t be analyzed, it is masked as [REDACTED] rather than stored raw. Nothing is ever dropped.

How NER detection works

The collector runs Microsoft Presidio as a sidecar. Two mechanisms work together: deterministic pattern recognizers with checksums (emails, phone numbers, credit cards, SSNs, IBANs, IPs) and statistical NER (spaCy en_core_web_lg) for names. Domain identifiers like medical record numbers are covered by configurable patterns that only trigger near context words — MRN 4829102 is redacted, order 4829102 is not. Redacted items carry a foam.presidio.redactions marker for auditing.

Benchmarks

Measured on our own telemetry (500 adjudicated documents) and a 14-language test round:

  • Latency: 12 / 81 / 92 ms (p50 / p95 / p99) per document; ~0 ms for unscanned traffic.
  • Pattern entities: IPs 100%, phones 97.4% recall; emails, cards, and IBANs redact across all 14 tested languages, including right-to-left scripts, with zero text corruption.
  • Names: 33.5% recall on production-domain text — the NER model is English-trained. Its published reference is ~0.85 F1 on OntoNotes 5 English news (see spaCy’s published benchmarks and the en_core_web_lg model card); the gap is domain and language shift, which is why we benchmark on our own data using the methodology from Presidio’s evaluation framework (presidio-research).