Experiments (A/B) layer

Read this first

Today we have no way to run an A/B test. Every experiment that has shipped was built by hand, inside one product, with its own split logic and its own analytics. Nothing is reusable and nothing is comparable.

This proposes one generic experiments layer. An operator defines an experiment and its groups in Janus. A deterministic hash of the account id decides which group a user is in. Every analytics event carries the group the user was actually in when the event happened. The readout then slices any existing metric by group.

Phase 1 is the config plane only — DB, API, and Janus. It delivers the ability to create and ramp an experiment and see it stored correctly. Nothing consumes an experiment yet, and no product behavior changes. That keeps the first review small and lets each of the three teams say yes to a self-contained piece.

Phase 2 wires up a consumer — sevro serves a branch, the clients echo the group onto events, and the data pipeline carries the label through to the readout. That work is described here so you can see where Phase 1 leads, but it is not what is being asked for now.

Contents

1. How it works
2. Which services are involved
3. Proposal — Database
4. Proposal — social-v1 API
5. Proposal — Janus
6. Phase 2 — sevro, clients, data
7. Decisions we need ruled on
8. Where the work stands

How it works

1. Assignment is a pure function, never a stored row

A user's group is computed, not looked up. There is no assignment table and no write on the read path. We hash the account id with the experiment's salt to land the user in one of ten thousand buckets:

bucket = fnv32a( decimal_string( fnv32a( salt + account_id ) ) ) % 10000   # → [0, 9999]

This is the GrowthBook v2 FNV32a spec — about ten lines per runtime, and we run no GrowthBook software. The salt never changes after an experiment is created, so a user's bucket never changes for the life of that experiment.

2. Groups own bucket ranges, and ramping moves the ranges

Each group owns an explicit, contiguous slice of the bucket space, stored directly as range_start and range_end. Control is a real group row with its own range, not a leftover gap.

To ramp a group up, an operator re-slices the ranges. Buckets stay fixed and the group that owns a bucket changes.

The fill order is what keeps a ramp safe. Treatment arms anchor to opposite ends of the bucket space and grow inward toward control: the first arm fills up from 0, the second fills down from 9999, and control keeps the middle. So raising an arm's percentage trades buckets with control only — users never switch from one treatment arm to another.

3. Every event carries the group the user was actually in

If the readout asked "what group is this user in now", a ramp would retroactively rewrite history. So the group travels with the event instead. The serving endpoint tells the client which group it served under, and the client echoes that value onto every analytics event it sends.

The result: an engagement carries the group that served the post, not the group as of the tap. Attribution stays correct across any number of ramps.

What this layer is not

Which services are involved

Two facts worth calling out. social-v1 is not on the analytics event path — it stores config and serves the admin API, nothing more. And the ClickHouse column already exists: the analytics pipeline shipped experiment_labels Array(LowCardinality(String)) on impressions, engagements, and sessions. This layer only fills a column that is already there.

Proposals by team

Three new tables in the configuration schema. No changes to any existing table. No data migration. Assignment is never stored, so these tables hold configuration only and stay small — a handful of rows per experiment, plus one rollout row per group per ramp.

Ramping is an UPDATE that closes the previous rows' effective_to plus an INSERT of the new ranges, both in one transaction. The current layout for a group is effective_from <= now() and (effective_to is null or effective_to > now()). Reading the layout as of any past moment uses the same predicate with a different timestamp, which is what makes the audit trail and the Phase 2 re-derive check possible.

Experiments are disabled, never hard-deleted. A disabled experiment means everyone is control.

One new admin controller, Api::V1::Admin::ExperimentsController, structurally mirroring the existing feature-flags admin vertical from MASTO-1578: controller, Panko serializers, routes, apidocs. Same authorization pattern — doorkeeper admin:read / admin:write scopes plus require_admin!.

The one piece with real logic is Configuration::SaveExperimentLayoutService. Saving a layout validates the ranges, closes the previous rollout rows, and inserts the new ones inside a single transaction. Concurrent edits collide on lock_version and return 409 rather than silently merging into a torn layout.

An experiments admin section that mirrors the existing feature-flags vertical exactly. No Janus-local tables and no RabbitMQ — Janus reads and writes experiments only through the social-v1 admin API.

The one thing that is not a straight copy is the ramp editor. Operators think in percentages, not raw bucket indices, so the editor takes percentages and derives the ranges with the anchored fill order described above — first arm up from 0, second arm down from 9999, arms three and up stacked after the first, control holding the middle. Before saving, it shows a mandatory preview naming exactly which bucket spans move between which groups — so nobody ramps a group and discovers afterwards that users moved between arms. A 409 from the API surfaces as a conflict banner.

Phase 1 gives us a config plane with nothing reading it. Phase 2 connects a real consumer. It is sketched here so each team can see what is coming and flag an objection early, but no commitment is being requested now.

The transport is a plain response header so the rule for every client is identical and endpoint-agnostic: echo what the server sent.

→ 200
X-Truth-Exp-Group: foryou_feed_mvp:treatment
   # multiple:  foryou_feed_mvp:treatment,ranking_v3:control
[ {…status…}, {…status…} ]        # body unchanged

Decisions we need ruled on

• 1
  Where range integrity is enforced. Non-overlap and full [0, 9999] coverage are validated in the save service today. DB constraint instead, or as well? Owner: DB.
• 2
  Salt immutability. Enforce with a trigger, or trust the admin API and a read-only field in Janus? Owner: DB.
• 3
  Target branch. The MR targets develop for the stronger admin auth. Confirm. Owner: API.
• 4
  Who may run an experiment. Janus gates on Permission::ENGINEER, copying feature flags. Is that the right bar for creating and ramping? Owner: Janus + product.
• ✓
  Should Janus own the schema and API, since it owns the UI? Raised by Josh Landry. Answer: no — Janus runs on its own database, and sevro must read experiment config on the feed serving path. Putting the schema in Janus would place an admin tool's database on the critical path of every feed request, and give sevro a second database to reach for config it needs on every read. Keeping config in the social-v1 primary means sevro reads a DB it already connects to. Janus stays a pure API client with no local tables, exactly like the feature-flags vertical.
• 5
  Ramp policy. Nothing in the schema stops a range moving in either direction, and disabled means everyone is control. The anchored fill order makes 2-arm ramps safe by construction, but "avoid re-ramping a 3+ group experiment" is currently a convention, not an enforced rule. Should the admin API reject an edit that would move users between treatment arms, or is the Janus save preview warning enough? Owner: product + whoever runs the first test.

One smaller open item for Phase 2: sevro's config cache TTL and the client refresh cadence together size the ramp-transition window, during which events legitimately carry mixed groups. The rollout log gives the exact change boundaries, so a readout can exclude transitions if it wants to.

Where the work stands

What each team can do next

• 1
  DB + API — review MR !3834 and rule on decisions 1, 2, and 3. This unblocks Janus and all of Phase 2.
• 2
  Janus — review MR !896 for shape and answer decision 4.
• 3
  sevro, clients, data — read the Phase 2 section and say now if anything in the shape is wrong. Nothing to build yet.