case study · experimentation · liquid · cloudflare-workers
A/B testing without an app: bucketing, stats, and honest data
The job. Merchants ask for split tests constantly — headline copy,
cart messaging, whole PDP layouts — and Shopify has no native theme A/B.
The usual answer is a paid app. This store runs three concurrent
experiments on a framework built from parts already on the table: Liquid,
one shop metafield, localStorage, and the same Worker + KV that run the
promo sync. The panel below is its live dashboard.
Bucketing that survives everything. A pre-paint inline script assigns
each visitor once (a hash of a random visitor id, persisted in
localStorage) and flips a class on <html> before first paint. Both
arms of every copy experiment are server-rendered; CSS shows exactly one.
No flicker, nothing for Dawn’s Section Rendering re-renders to lose, and
visitors without JavaScript simply get the canonical arm. Assignments are
sticky by design: anyone may change the traffic split in the ops
console, and it only affects new visitors — re-hashing on a new split
would silently shuffle existing visitors between arms and poison the test.
The template experiment. The third experiment swaps the entire
product-detail template using Shopify’s own alternate-template mechanism
(product.exp-b.json, reached via ?view=exp-b — pure template JSON,
zero new Liquid). Enforcement is bidirectional: a visitor assigned B who
lands on the default PDP is moved onto the alternate template, and a
visitor assigned A who opens a shared ?view=exp-b link is moved back —
so what renders always equals what was assigned. Internal product links
are rewritten at click time, so normal browsing never pays a redirect.
See both arms yourself. Enforcement cuts both ways — it also stops a
curious reader from ever seeing the arm they weren’t dealt. So the
framework has a preview mode: ?ab_preview=a|b forces the display arm,
skips enforcement, and sends no events at all — preview visits never touch
the data. Store password is on the demo page, then:
-
-
PDP template — default template · product.exp-b
-
Cart phrasing — arm A · arm B (put something in the cart first, below the $80 threshold)
The screenshots above were taken through preview mode on the live store — same visitor, same cart, only the arm forced.
Counting correctly on a database with no atomic increment. Workers KV
can’t do count += 1 safely under concurrency. So no request ever touches
the aggregate: each beacon batch (sent as text/plain via sendBeacon —
CORS-safelisted, no preflight to break) lands in its own unique buffer
key, and a single-flight rollup on the Worker’s 5-minute cron is the only
writer of the results document. Buffer keys younger than 60 seconds are
left for the next cycle, because KV’s list and delete operations are
eventually consistent — a residual double-count window exists in theory
and is stated here rather than hidden. Exposure means seen, not
present in the DOM (the cart drawer’s markup exists on every page — an
IntersectionObserver decides), and a conversion only counts for an
experiment the session was actually exposed to.
Input is a closed vocabulary. The ingest endpoint validates every event against the running config and silently drops the rest; the console lets anonymous visitors change exactly two things — status and split, both preset choices. Variant copy is fixed: letting strangers rewrite the arms mid-test would end the experiment, so it isn’t a setting.
Statistics you can check. Per arm: conversion rate with Wilson 95% intervals; per experiment: a two-proportion z-test, relative lift, a sample-ratio-mismatch check, and a verdict that refuses to speak below 100 exposures per arm.
The honesty mechanism. A portfolio store has no traffic, so the
dashboard’s volume comes from a seeded simulator with known ground
truth — labeled simulated everywhere, tallied separately, and never
merged with real visitor counts in any view. The truths were chosen to
show the full verdict spectrum, because real experimentation mostly isn’t
wins: the headline test carries a genuine lift (it reads significant), the
PDP template change genuinely hurts (a significant loser — the correct
call is roll it back), and the cart phrasing has a true null effect (the
dashboard honestly says “not yet”). The framework’s flagship unit test
runs the simulator through the aggregation and the stats engine and
asserts it recovers the planted lift, catches the planted regression, and
does not false-positive the null — the system verifying its own measuring
stick.
Stated limits. Randomization is per visitor while exposure/conversion dedup is per session — a standard trade-off, named rather than glossed over. Checkout completion isn’t observable from a theme, so the cart experiment’s metric is begin-checkout. Both caveats would be the first thing a client’s data team asks about; better they read it here first.