Paying for an A/B testing tool just to flip the coin? How to analyse those experiments in your warehouse

Why teams use an experimentation tool only to randomise

Plenty of companies pay for a feature flag or A/B testing tool and use it only for the coin flip.

The setup is usually similar. Engineers use the tool's SDK to decide which users see which variant. The tool has a results page, but nobody opens it. The assignments get exported to the data warehouse and the real work starts there. An analyst writes the SQL to join exposures to outcomes. Someone builds a dashboard. The statistics happen in a notebook or a spreadsheet, or in a home-grown package that one person maintains.

When I ask why, the answer is nearly always the same. The data that matters lives in the warehouse, and the tool can't see it.

It's a rational place to end up. It's also expensive in ways that never appear on an invoice.

Why the analysis moved to the warehouse

The analysis moved to the warehouse because the first generation of experimentation tools wasn't built for companies whose important metrics live there.

Some of those tools grew up testing web pages and compute metrics on a copy of your product events. Others are feature flag tools first, with analysis added later. Either way, the metric that matters is often one the tool can't see. Revenue after refunds, retention, support contacts and most of what finance reports only exist in the warehouse. When the tool's numbers stop matching the rest of the business, people stop trusting them. I covered this in more detail in What is warehouse-native experimentation?.

Meanwhile the data team has done the hard work. The warehouse holds modelled, agreed definitions of the metrics the company runs on. Nobody wants to define those again in a vendor's interface, against a second copy of the data that never quite matches.

So teams keep the tool for the part they'd struggle to build themselves, which is safe, fast assignment with targeting and a kill switch, and they set the analysis aside. The data team picks it up by hand.

What the split costs

The real cost of this arrangement is that you've taken on the hardest part of building an in-house experimentation platform, without the budget or the mandate to do it properly.

The analysis is the product. A sample ratio mismatch check. Variance reduction so experiments finish sooner. Sequential testing so people can look at results without inflating false positives. Ratio metrics with the right variance. Outlier handling for revenue. In the teams I've seen, each of these is hand-built in a notebook or, more often, skipped.

Every readout is a project. Someone has to write the query, check the join, run the numbers and write it up. Readouts queue behind other work. Experiments get called on a gut feeling while the proper analysis waits.

Methods drift. Two analysts produce two different answers for the same experiment, because they made different choices about the unit, the window, the outliers or the test. Neither is wrong. Neither is comparable.

Nothing is on record. Results live in slide decks and notebooks. Six months later nobody can say what was tested, what won, or whether the metric definition has changed since. Re-analysing an old experiment means finding the notebook and hoping it still runs.

You end up hiring for it. This is the cost that gets missed, because it sits in headcount rather than tooling. As the program grows, the readout queue grows with it, and the fix is usually another analyst, then another. Those are real salaries spent on work a platform should be doing, like writing the same joins, running the same tests and producing the same readout format. The number of experiments you can run ends up capped by the size of your data team.

The platforms I worked on at Booking.com, Meta and Canva existed so that none of this was manual. A flag tool plus a notebook gives you the assignment half of a platform with the analysis half missing.

The two halves of an experimentation tool

An experimentation tool does two separate jobs, and they don't have to come from the same place.

Assignment is deciding which variant a unit gets. At its simplest it's a hash of a unit ID and an experiment key, bucketed into variants, evaluated in your product in milliseconds. That part is well understood and many tools do it reliably. Doing it well takes more. Holdouts that keep a slice of users out of everything for a quarter. Automated rollouts that take a winner to everyone. Fixing specific IDs to a variant, for QA or for a customer who must never see the test. Keeping assignment consistent across surfaces and sessions. Some tools offer some of these. Few do all of them with the rigour the analysis needs, and when the analysis happens somewhere else, none of it reaches the results.

Analysis is everything that happens after. Joining exposures to outcomes. Choosing the right unit and the right window. Computing metrics the business trusts. Running the statistics. Checking that the experiment can be trusted at all. Turning the result into a decision, and keeping that decision on record.

The platforms I worked on put most of their effort into the second half, and built the first half to serve it.

The rest of this article is about getting the analysis half without touching the first. The two are stronger together, though. When the tool that assigned a unit is the same tool analysing it, holdouts, rollouts and overrides feed straight into the results with nothing to reconcile. That's why I'd move assignment to Cherto eventually. I don't think the analysis should wait for it.

How Cherto analyses experiments from any tool

Cherto was built so that you can connect it to your warehouse and nothing else, and get the analysis half for every experiment you run, wherever the assigning happens.

It works like this:

1. Connect Cherto to your warehouse. That's the integration. No new SDK, and no code to change if your assignments already reach the warehouse.
2. Point it at where your assignments land. That might be the export table your flag tool already writes, an event your own pipeline logs, or the campaign groups your email platform exports. Cherto needs a record of which unit got which variant, and when.
3. Every experiment in that data shows up in Cherto. You don't set experiments up a second time. If your other tool is running it and the assignments are landing in your warehouse, Cherto picks it up, with its variants and its exposures. What the assignment data can't tell it, like the intended split that the sample ratio check tests against, you fill in.
4. Each one gets the full analysis. Metrics defined once in Cherto, on top of the tables and views you already own, and computed in your warehouse. Trust checks on every run, including sample ratio mismatch, crossover and guardrails. The statistics engine, frequentist and Bayesian, with CUPED, sequential testing, the delta method, winsorisation and more.

Raw, unit-level data never leaves your warehouse. Only aggregates reach Cherto, the same as any other Cherto experiment.

Analyses run on a cadence you choose, and in practice that means seeing results within a day at most. The queries run in your warehouse, on your bill, so Cherto's behaviour there matters. The warehouse-native guide covers what to expect on freshness and what a good tool should do in your warehouse.

It isn't limited to experiments starting today. If assignments from last quarter are already in the warehouse, those experiments can be analysed too, against your current metric definitions, including the ones nobody had time to analyse properly the first time.

And because Cherto only needs the assignments to be in the warehouse, it doesn't matter who did the assigning. A product experiment from your flag tool and an email campaign split by your marketing platform get the same metrics and the same statistics, side by side.

Why trust Cherto's numbers over your own SQL

Teams in this situation analyse by hand because they've been burned by vendor numbers they couldn't check. That's a fair objection, and two things answer it.

First, you can check. Every query Cherto runs in your warehouse is available to you, and the metrics are computed from your own tables, so you can rerun the numbers yourself and compare. The warehouse-native guide has more on why that matters.

Second, the statistics are our job. Techniques like CUPED, sequential testing and the delta method carry real trade-offs in how they're implemented, and we spend our time verifying and tuning those choices. Our team has built and run experimentation at Booking.com, Meta, Canva, SEEK and Xero. If you want to go through a specific technique or implementation choice, we're glad to, any time.

One practical way to settle it is to run Cherto alongside what you have now. Point it at the same assignments, compare the output with your own analysis or with another tool, and decide from there.

What your warehouse needs

Cherto needs three things in your warehouse. Teams doing the analysis by hand usually have all of them already.

• An assignment record. Unit ID, experiment key, variant, and the time the unit was first exposed. One row per unit per experiment is ideal. If your source logs every evaluation instead, that's workable, and we'll agree with you how repeats are handled when the experiment is set up.
• A stable unit. Whatever you randomise on, usually a user or account ID, needs to be the same ID your metric data uses, or join to it cleanly.
• The metric data. Orders, revenue, retention, support tickets, whatever you want to measure. If it's in the warehouse and you choose to share it, it can be a metric.

You decide what Cherto can see. Grant it the assignment table and the tables behind your metrics, and nothing else, if that's what suits you.

Most mainstream flag tools can export assignments to the warehouse, or let you log the assignment through your own event pipeline. Most email and marketing platforms can export their campaign groups. Your own code needs to write the exposure somewhere the warehouse can see it.

One habit is worth building whatever tooling you use. Log the moment a unit is actually exposed to a variant, rather than the moment a flag was evaluated somewhere in the background. The closer your record is to real exposure, the more sensitive and trustworthy the analysis. It's harder to get right than it sounds once you have multiple surfaces and flags evaluated far from where the change is seen.

If you go this route with Cherto, you don't work any of it out alone. We step teams through the setup, from finding the assignment data to defining the first metrics.

Do you end up paying for two tools?

Yes, if you keep your flag tool, you pay for both for a while.

Two things soften that. Several vendors sell a flags-only tier, which is worth asking about once their analysis is no longer in use. And it's worth switching off their results page, so there's one set of numbers in the company rather than two.

What you stop paying for is analyst time. The people who were hand-rolling readouts are free for the work only they can do, like designing better experiments, digging into surprising results and building the metrics the business doesn't have yet. Everyone else gets results that are more reliable and more consistent than before, without waiting in a queue.

Later, you can move assignment to Cherto flag by flag, or keep your existing tool where it's deeply embedded. The analysis is the same either way.

The scope changes too. The analysis half of the platform now covers every experiment in the business, including the ones marketing runs, rather than only the ones one tool happens to assign.

Common questions

Can we use our feature flag tool just for flags and analyse experiments in our data warehouse?

Yes. If the tool's assignments land in your warehouse, Cherto can analyse every experiment it runs there, with your warehouse metrics and the full statistics engine. The flag tool keeps doing the assigning.

Do we have to change any code?

Not if your assignments already land in your warehouse. If they don't, you log the exposure where your code acts on the flag, and that is the only change.

How fresh are the results?

Analyses run on a cadence you choose. In practice that means seeing results within a day at most. The exception is launch, where near real-time checks matter most, and the warehouse-native guide covers how that's handled.

Does this add to our warehouse bill?

Yes. The metric queries run in your warehouse, so the compute is yours. How much depends on data volume, the number of metrics and how often analyses run. You can see every query Cherto runs and control when the jobs run.

Can we check Cherto's numbers against our own SQL?

Yes. The metrics are computed from your own tables and the queries are visible to you, so you can reproduce any result in your warehouse. Running Cherto in parallel with your current analysis for a few experiments is a sensible way to build confidence.

Can we analyse experiments we ran before connecting Cherto?

Yes. If the assignment records are still in your warehouse, those experiments can be analysed against your current metrics, including ones that ended months ago.

Which tools does this work with?

Any tool that can get its assignments into your warehouse, and your own code. Where the record comes from doesn't matter to the analysis.

What if our tool doesn't export assignments to the warehouse?

Most do, either directly or through an integration with your event pipeline. If yours doesn't, the fallback is to log the exposure yourself at the point your code acts on the flag. Getting that right matters more than it looks, and it's something we help teams with.

Does Cherto need to see individual users?

No. Metrics are computed in your warehouse and only aggregates reach Cherto. Unit-level data stays where it is.

Can we move assignment to Cherto later?

Yes. Cherto has feature flags, targeting, holdouts and rollouts built in. Teams that want to consolidate can do it gradually, and the analysis doesn't change when they do. What they gain is everything working from one place, with nothing to reconcile between the tool that assigned and the tool that measured.