Israeli Bank Scrapers

Scrape transactions from 19 Israeli banks and credit card companies with built-in Cloudflare WAF bypass and end-to-end PII redaction.

Maintained fork of eshaham/israeli-bank-scrapers, rewritten on a typed phase-based pipeline using Camoufox (Firefox anti-detect), Playwright, and TypeScript 6.0 strict mode.

npm install @sergienko4/israeli-bank-scrapers
Copy

Architecture at a glance

flowchart LR
    subgraph BB["Browser banks (13 pipeline)"]
      direction LR
      INIT --> HOME --> PRELOGIN["PRE-LOGIN (opt-in)"]
      PRELOGIN --> LOGIN
      LOGIN --> OTP["OTP-TRIGGER / OTP-FILL (opt-in)"]
      OTP --> AUTH["AUTH-DISCOVERY"]
      AUTH --> BIND["BIND-API-MEDIATOR"]
      BIND --> SCRAPE["API-DIRECT-SCRAPE
(direct API calls — no nav / DOM walk)"]
      SCRAPE --> TERM["TERMINATE"]
    end

    subgraph API["API-direct banks (3) — OneZero · Pepper · PayBox"]
      direction LR
      CALL["API-DIRECT-CALL
(login + OTP via JSON API)"] --> SCR["API-DIRECT-SCRAPE
(.final emits balanceResolution)"]
    end

    BB -.->|"unified result shape"| RESULT(["IScraperScrapingResult"])
    API -.->|"unified result shape"| RESULT
Copy

Every pipeline bank scrapes direct API after login. On the browser banks (13), once AUTH-DISCOVERY proves the session, BIND-API-MEDIATOR binds an authenticated ApiMediator to the live page and API-DIRECT-SCRAPE walks a typed hard-model shape of REST/GraphQL calls — no post-auth page navigation, no DOM scraping. The api-direct banks (3) reach the same API-DIRECT-SCRAPE through API-DIRECT-CALL (headless JSON login) instead of the browser LOGIN → AUTH-DISCOVERY → BIND-API-MEDIATOR prefix. The retired generic ACCOUNT-RESOLVE → DASHBOARD → SCRAPE → BALANCE-RESOLVE chain is dormant (no bank triggers it). See Architecture for the contract that keeps the wiring honest.

📚 Full documentation: User Guide & Architecture (mkdocs) · API Reference (TypeDoc) · Changelog · Contributing

Table of Contents

• Quick Start
• Features
• Prerequisites
• Usage
  • Sample Output
• Supported Institutions
• OTP (Two-Factor Authentication)
  • Phone input contract
• Error Types
• Configuration
• Testing
• Logging & Bug Reports
  • What gets redacted, and what survives
  • Filing a bug report
  • How redaction stays correct over time
• Architecture
• Advanced Usage
• Contributing
• Version history
• Contributors
• Links
• Known Projects
• License

Quick Start

Three steps to get transactions from Bank Hapoalim:

import { CompanyTypes, createScraper } from '@sergienko4/israeli-bank-scrapers';

const scraper = createScraper({
  companyId: CompanyTypes.Hapoalim,
  startDate: new Date('2024-01-01'),
});

const result = await scraper.scrape({
  userCode: '1234567',
  password: 'mypassword',
});

if (result.success) {
  result.accounts?.forEach(acc => {
    console.log(`${acc.accountNumber}: ${acc.txns.length} txns, balance ${acc.balance}`);
  });
}
Copy

Replace userCode and password with real credentials. See Supported Institutions for other banks and credential fields, and OTP for banks that require an otpCodeRetriever callback.

Features

• WAF bypass on the first attempt — Camoufox (Firefox anti-detect) instead of Puppeteer, no stealth shims.
• Zero CSS selectors in interaction code — visible Hebrew text + a 7-strategy SelectorResolver. Site UI redesigns rarely break logins.
• Auto-detect + auto-fill OTP — either via a callback you provide or a stable long-term token (returned for API banks).
• End-to-end PII redaction — every log line, captured network body, and DOM snapshot goes through one redactor before it touches disk. Share traces publicly without leaking customer data.
• Direct-API scraping after login — once the browser proves the session, data is fetched through a typed hard-model shape of REST/GraphQL calls (BIND-API-MEDIATOR → API-DIRECT-SCRAPE), not DOM navigation. Every pipeline bank (browser and api-direct alike) reads its accounts, balances, and transactions straight from the bank's own API.
• Phase-based pipeline — isolated, single-responsibility phases; browser banks add a WAF-bypassing login front-end, api-direct banks run login as a JSON-API flow. One result shape across both paths.
• Single-phase balance ownership — the hard-model API-DIRECT-SCRAPE shape emits balanceResolution directly from each bank's balance endpoint; legitimate empty-month results pass through without a false "scrape failed" signal.
• Dual ESM + CJS — works with both import and require().
• 3 test surfaces — unit (Jest), pipeline coverage (with thresholds), mock + real E2E orchestrators.

Prerequisites

Environment variables (all optional):

Per-phase navigation timeout is controlled by the defaultTimeout scraper option (not an env var) — see Configuration → Scraper options.

Usage

import { CompanyTypes, createScraper } from '@sergienko4/israeli-bank-scrapers';

const scraper = createScraper({
  companyId: CompanyTypes.Amex,
  startDate: new Date('2024-01-01'),
});

const result = await scraper.scrape({
  id: '123456789',
  card6Digits: '123456',
  password: 'mypassword',
});

if (result.success) {
  for (const account of result.accounts!) {
    console.log(`${account.accountNumber}: ${account.txns.length} transactions`);
  }
} else {
  console.error(result.errorType, result.errorMessage);
  if (result.errorDetails) {
    console.error('Suggestions:', result.errorDetails.suggestions);
  }
}
Copy

Sample Output

{
  "success": true,
  "accounts": [
    {
      "accountNumber": "****1234",
      "balance": 0,
      "txns": [
        {
          "date": "2024-01-15",
          "description": "<merchant:12>",
          "originalAmount": -***,
          "chargedAmount": -***
        }
      ]
    }
  ]
}
Copy

This snippet shows the redacted shape produced by the redaction layer. Real balances, amounts, and merchant strings are masked; account numbers are tail-only. The balance field is populated by BALANCE-RESOLVE.final (browser banks) or API-DIRECT-SCRAPE.final (api-direct banks) — one source of truth across both paths.

Supported Institutions

19 institutions across three categories. CompanyTypes.<Name> is the discriminator passed to createScraper; credential field names are validated at runtime against the tables below.

Migration notice (wide-net policy): 3 banks are still on the legacy scraper path — Behatsdaa, Beyahad Bishvilha, Mizrahi Bank. They keep working through the same createScraper(...) entry point, but new features and bug fixes target the Pipeline architecture. They are scheduled for migration; the broader src/Scrapers/Base/, src/Common/, and legacy bank folders are also on the migration path and will be folded into src/Scrapers/Pipeline/ over time. Public API behavior is preserved.

OTP (Two-Factor Authentication)

createScraper({
  companyId: CompanyTypes.Beinleumi,
  startDate,
  otpCodeRetriever: async phoneHint => await getCodeFromUser(phoneHint),
});
Copy

await scraper.scrape({
  email,
  password,
  phoneNumber: '972000000000',
  otpCodeRetriever: async () => '123456',
});
// result.persistentOtpToken — save to skip SMS next run
Copy

Phone input contract

Every API bank that accepts a phoneNumber credential reads digits-only international form (no +, no dashes). For example: '972000000000'. The pipeline edge rewrites this into each bank's wire format on the fly:

You don't have to know which format each bank wants — pass the international digits and the mediator normalises before login.

Error Types

Configuration

All configuration goes through ScraperOptions at construction time and ScraperCredentials at scrape time. Nothing else reads from disk or env.

Testing

Three test surfaces, all copy-paste runnable:

# Run everything except E2E
npm run test:unit
Copy

npm run test:pipeline
Copy

npm run test:e2e:mock
Copy

# All in-scope banks, parallel workers respecting per-bank sequencing
npm run test:e2e:real

# One bank only — useful when isolating a regression
npm run test:e2e:real:single -- --testPathPatterns=Amex
Copy

npm run lint                # eslint --max-warnings 0 + architecture + canaries + format:check
npm run lint:biome          # Biome rules (CI-equivalent)
npm run lint:dead-code      # detect-dead-code.ts — flags unused exports
npm run lint:architecture src/Scrapers/Pipeline   # cross-layer import validator
npm run type-check          # tsc --noEmit
Copy

Logging & Bug Reports

The package auto-redacts PII before any line is written — terminal, log files, captured network bodies, captured DOM snapshots. You can share pipeline.log, network/*.json, or screenshots/*.html publicly without exposing your customers' data.

What gets redacted, and what survives

The "stable hints" (***NNNN, <merchant:N>, +***/-***, array-size markers) are deliberate — they preserve enough for us to correlate failures across phases without ever showing raw PII.

Filing a bug report

Attach all three if available:

1. pipeline.log — full Pino transcript of the run.
2. network/*.json — captured HTTP bodies (already redacted at write time).
3. screenshots/*.html — DOM snapshots per phase (already redacted).

Skip screenshots/*.png (raster images are not OCR-redacted today — they may contain unredacted PII rendered by the bank's UI). If a PNG is essential to the report, blur or crop before attaching.

How redaction stays correct over time

Two independent enforcement layers keep raw PII out of the log surface even as the codebase evolves:

• Runtime layer — PiiRedactor.ts is the single source of truth. Pino runs it as the redact.censor callback so every record is redacted before any transport. NetworkDiscovery and FixtureCapture route their byte streams through the same redactor before persisting.
• Commit-time layer — ESLint AST selectors (T09 / T16) and an architecture-validator regex (PII-Log rule) reject pull requests that try to bypass the runtime by interpolating PII identifiers into LOG.* template literals or passing full payload objects under result|accounts|transactions|... keys.

If you spot a pattern that leaks past both layers, please open an issue — that's a load-bearing bug, not cosmetic.

Architecture

Pipeline of typed phases. Each phase owns its mediator zone, its own well-known-selectors dictionary, and its own retry policy. Phases never reach into one another's state — communication happens via slim Option<T> fields on the pipeline context.

Browser banks — direct-API after login (hard model):
  INIT → HOME → [PRE-LOGIN] → LOGIN → [OTP-TRIGGER → OTP-FILL]
       → AUTH-DISCOVERY → BIND-API-MEDIATOR → API-DIRECT-SCRAPE → TERMINATE

API-direct banks (fully headless, 2 phases):
  API-DIRECT-CALL → API-DIRECT-SCRAPE
Copy

• [PRE-LOGIN] is opt-in — card banks with a separate "show login" toggle: Amex, Isracard, Max, VisaCal.
• [OTP-TRIGGER → OTP-FILL] is opt-in. Beinleumi-group banks (Beinleumi, Massad, Otsar Hahayal, Pagi) and Hapoalim use them. Hapoalim uses OTP-FILL only, conditionally.
• AUTH-DISCOVERY separates the credential exchange from the data handoff so post-auth signal capture (cookies, ids, tokens) is observable, redactable, and testable in isolation.
• BIND-API-MEDIATOR binds an authenticated ApiMediator to the live login page so the hard-model scrape dispatches REST/GraphQL calls over the same WAF-passing browser session (cookies / discovered tokens ride along). It reads the login-inclusive capture pool once to prime any bearer / session token, then hands off.
• API-DIRECT-SCRAPE replaces the retired generic ACCOUNT-RESOLVE → DASHBOARD → SCRAPE → BALANCE-RESOLVE chain for every pipeline bank — same PRE → ACTION → POST → FINAL lifecycle, but the action is a bank-supplied hard-model shape (a typed list of REST/GraphQL calls) instead of a DOM walk. .final emits ctx.balanceResolution directly. No post-auth page navigation.
• API-DIRECT-CALL replaces LOGIN [+ OTP-TRIGGER + OTP-FILL] for banks with a programmatic auth endpoint (OneZero, Pepper, PayBox). OTP, when required, is fetched via the same otpCodeRetriever callback during this phase.

Advanced Usage

import { Camoufox } from '@hieutran094/camoufox-js';

const browser = await Camoufox({ headless: true });
const results = await Promise.all(
  banks.map(async ({ companyId, credentials }) => {
    const ctx = await browser.newContext();
    const scraper = createScraper({ companyId, startDate, browserContext: ctx });
    const result = await scraper.scrape(credentials);
    await ctx.close();
    return result;
  }),
);
await browser.close();
Copy

createScraper({
  companyId: CompanyTypes.Leumi,
  startDate,
  defaultTimeout: 60000,
  navigationRetryCount: 2,
});
Copy

- npm install israeli-bank-scrapers
+ npm install @sergienko4/israeli-bank-scrapers
Copy

Contributing

Found a bug? Have an improvement? See CONTRIBUTING.md for the contribution workflow, branch strategy, and testing requirements.

Before opening a PR, run:

npm run test:unit         # all unit tests
npm run lint              # eslint + architecture + canaries + format
npm run test:pipeline     # coverage gates (97/95/97/98)
Copy

The pre-commit hook runs the same gates plus mock-E2E and bank tests. PRs are squash-merged; release-please cuts the next version automatically.

Version history

Contributors

Thanks to the original israeli-bank-scrapers contributors whose work inspired this fork:

sergienko4 Eugune 💻 📖 ⚠️ 🚧 🚇	Elad Shaham 💻	sebikaplun 💻	esakal 💻	ezzatq 💻	kfirarad 💻	baruchiro 💻
matanelgabsi 💻	dratler 💻	dudiventura 💻	gczobel 💻	orzarchi 💻	erezd 💻	erikash 💻
daniel-hauser 💻

Links

• API Documentation (TypeDoc)
• Changelog
• Contributing
• Code of Conduct
• Security Policy

Known Projects

• israeli-bank-scrapers-to-actual-budget — Sync to Actual Budget
• Caspion — Auto-send to budget apps
• Moneyman — Save via GitHub Actions
• Firefly III Importer — Import to Firefly III

License

MIT. Maintained by @sergienko4. Based on eshaham/israeli-bank-scrapers.

_{This README follows the Standard README specification.}