PII redaction¶
PiiRedactor.ts is the single source of truth. Pino runs it as the redact.censor callback so every record is redacted before any transport writes.
| Source | src/Scrapers/Pipeline/Types/PiiRedactor.ts |
|---|---|
What gets redacted vs what survives¶
| Category | Example before → after | Why we keep the survivor |
|---|---|---|
| Account / card / Israeli ID / phone | 12-170-[REDACTED-DIGITS-6] → ***6789 | Last-4 lets us correlate across phases without showing the full id |
| Cardholder / customer name | דני משהו → <name:8> (length tag) | Length distinguishes spoof attempts |
| Merchant description | סופר-פארם רמת גן → <merchant:14> | Length helps reproduce the bug shape |
| Transaction amount | -247.50 → -*** | Sign preserved for credit/debit distinction |
| Auth tokens / cookies / OTP codes | eyJhbGc..., 123456 → [REDACTED], [OTP] | Discriminates token-shaped from non-token strings |
| URLs | host + path preserved; PII query keys redacted | Lets us correlate to the bank endpoint without leaking ids |
| HTML snapshots | text nodes + value attributes scrubbed in place | Layout preserved for debugging |
| Anything unrecognized | [REDACTED] (default-deny) | Fail closed when in doubt |
The censor function¶
PiiRedactor.censor(record) walks the entire object graph and applies category-aware substitutions:
const redacted = PII_REDACTOR.censor({
event: 'balance-resolve.fetch.success',
bankAccountUniqueId: '12345678', // → '***5678'
authorization: 'Bearer eyJhbGc...', // → '[REDACTED]'
amount: -247.50, // → '-***'
message: 'fetched OK', // unchanged — no PII shape
});
Where redaction runs¶
| Sink | Redacted by | Format on disk |
|---|---|---|
pipeline.log (Pino) | redact.censor callback at log time | JSON lines |
network/*.json (NetworkDiscovery captures) | PiiRedactor pre-write filter | JSON |
screenshots/*.png (SafeScreenshot) | NOT redacted — raster | PNG |
*.html snapshots (FixtureCapture, opt-in DUMP_FIXTURES_DIR) | redactHtml text + value attribute scrubs | HTML |
safeScreenshot API¶
The canonical capture function is safeScreenshot(page, options) in src/Scrapers/Pipeline/Mediator/Browser/SafeScreenshot.ts. It accepts an IScreenshotOptions (path + optional fullPage), writes the raw PNG to that path, and swallows any capture error so a failed shot stays diagnostic-only. It performs no redaction — PNGs are raster and cannot be reliably scrubbed (see the table above). It does not write HTML; DOM snapshots are a separate channel owned by FixtureCapture / SnapshotInterceptor.
Capture is gated upstream, not inside safeScreenshot. The target path comes from TraceConfig.getScreenshotDir, which is empty unless the opt-in FORENSIC_TRACE=true flag is set (TraceConfig.getRunFolder). With forensic capture off — the default — BasePhase.takePhaseScreenshot receives an empty path and returns early, so no PNG is ever written. There is no per-phase or CI allowlist: the single FORENSIC_TRACE switch governs the whole run folder (pipeline.log, network/*.json, screenshots/*.png) together.
safeScreenshot does not decide whether output is public or private and does not divert files into a private/ directory. The public CI artifact path block is the routing control: it uploads only pipeline.log and network/*.json. On failure, upload-private-diagnostics.sh uploads the full forensic run folder, including screenshots, to the access-controlled OCI diagnostics store. If screenshot capture itself fails, only a path-scrubbed reason is logged; absolute paths and leading relative path tokens are replaced before logging.
Forensic capture (FORENSIC_TRACE) — opt-in only¶
FORENSIC_TRACE | Effect |
|---|---|
unset / false / any other value | getRunFolder returns ''. No run folder, no pipeline.log, no network/*.json, no screenshots. Default. |
true (trimmed, case-insensitive) | The pipeline writes the full run folder under RUNS_ROOT. On a failed CI job the whole bundle (pipeline.log, network/*.json, screenshots/*.png) uploads to the access-controlled private store only — never to a public GitHub artifact (raster pixels can carry rendered PII). |
FORENSIC_TRACE is decoupled from LOG_LEVEL (pino verbosity only) and CI (OTP / Telegram). Set it explicitly when triaging a failure:
Disabling redaction¶
PII_REDACTION=off disables runtime redaction. Intended for real-bank E2E tests only (where the maintainer needs to compare actual vs expected values during development). Unit tests always run with redaction default-on so PiiRedactor.test.ts assertions hold.
Never set this in production code or CI.
Commit-time enforcement (PII-Log canary)¶
ESLint AST selectors reject pull requests that try to bypass the runtime layer. Banned patterns:
LOG.<level>(\...${piiIdentifier}...`)` — direct PII interpolation into template literalsLOG.<level>({ result, ... })/{ accounts }/{ transactions }— passing whole payloadsconsole.log(any) — bypasses Pino entirely
Source: the pii-template-literal, pii-error-message, and pii-payload-key canary fixtures + the PII-Log rule in lint-and-validate.ts.
If your new code emits a log that triggers the canary at commit time, the right fix is to extract the safe identifier first: