Bol Rank Tracker live · measurement day 21 sep

Where our products rank on bol.com — organic & sponsored, per search term and category · measurement period 9 jun – 21 sep (105 measurement days) · sorted by monthly revenue

📖 Data & architecture — how this dashboard is fed (for handover)

Pipeline

PIM/Users/jim/pim/pim.db · active EANs + brand→ bol-rank-pull.mjsdaily 08:40 · launchd com.gandalf.bol-rank→ bol Retailer API/insights/product-ranks · per EAN→ CSV history~/.gandalf/bol-rank-history.csv · canonical source→ this HTML/Users/jim/AI/AI-work/bol-rank-tracker/index.html

CSV→ dash-bolrankpayload in ~/.gandalf/metrics.sqlite report_cache→ Master DashboardStores › Search (refresh.mjs whitelist KEEP_EXACT!)

Scripts: dashboard/bol-rank-pull.mjs (fetch + history + payload) and dashboard/bol-rank-render.mjs (this HTML). Revenue sorting and brand come from ~/.gandalf/dashboard-data.json → product_net.rows (monthly revenue per EAN, written by refresh.mjs) and PIM products.brand.

API — bol Retailer Insights (live verified)

Endpoint	GET https://api.bol.com/retailer/insights/product-ranks?ean=…&period=DAY&number-of-periods=1&date=YYYY-MM-DD
Auth	client-credentials: POST https://login.bol.com/token with Authorization: Basic $BOL_CLIENT_BASE64 (stored in ~/.gandalf/.env) → Bearer token (~5 min valid)
Headers	Accept: application/vnd.retailer.v10+json and Accept-Language: nl-NL — without that last one this endpoint returns HTTP 406 (undici/Node-fetch default is *)
Response	ranks[] with per listing rank, impressions, wasSponsored, and searchTerm or categoryId. We aggregate per (term|category): organic = min(non-sponsored ranks), sponsored = min(sponsored), impressions summed.
Category names	GET /products/categories (1 req/min!) → taxonomy flattened, cached in ~/.gandalf/.bol-cat-names.json (TTL 30 d)

Gotchas (all handled in the pull)

• Rate limit 10/s → 120 ms pacing between EAN calls.
• 404 = EAN doesn't exist in the bol catalog (e.g. loose sleeves) — not an error, counts as a coverage finding.
• Data lag: "yesterday" only becomes available during the morning → the daily job runs at 08:40; a run that's too early returns 0 rows and gets backfilled later by self-healing.
• Only EANs with impressions return data — no rows = invisible on bol (a real finding, not a measurement error).
• number-of-periods > 1 aggregates without a date tag → unusable; always period=DAY + explicit date. Backfill possible up to ~60 days back (--backfill N).
• Fail guard: >30% of the EAN calls fail → the day is not saved (no partial snapshots); 3 retries; missing measurement days in the last 7 days are self-healingly backfilled on every daily run.

CSV schema (canonical history)

Column	Meaning
date	the MEASURED day (not the run day) — daily measures yesterday
ean · product	product (name from product-names.json, fallback PIM)
rank_type · rank_key	search + search term, or category + categoryId
organic_rank · sponsored_rank	best organic / sponsored position that day (empty = none)
impressions	sum of impressions on that listing that day

Run commands

node bol-rank-pull.mjs	daily: measures yesterday + self-healing + HTML + dash-bolrank
node bol-rank-pull.mjs --backfill 30	fetch the last N days (max 60) — fills history/trends
node bol-rank-pull.mjs --render-only	only HTML + payload from existing CSV (no API calls)
node bol-rank-pull.mjs --limit 2	smoke test with 2 EANs, writes nothing

dash-bolrank payload (for reuse in another dashboard)

{checked_at, last_date, ref_date, days, products:[{ean, name, best, terms:[{key, type, cur, was7, delta, imp, spo}], cats:[…]}], invisible:[{ean,name}]} — read it with sqlite3 ~/.gandalf/metrics.sqlite "SELECT payload FROM report_cache WHERE key='dash-bolrank'". New cache keys must be added to the KEEP_EXACT whitelist in dashboard/refresh.mjs (±line 610), otherwise they never reach the master dashboard.

Thresholds (constants at the top of the pull): top-10 boundary · movement = Δ≥2 (search term) / Δ≥3 (category) · "disappeared" = ≥5 prior impressions. Logs: ~/.gandalf/logs/bol-rank.log.

generated 2026-09-22 06:40 · refresh: node dashboard/bol-rank-pull.mjs