What is warehouse-native experimentation?

What does warehouse-native mean?

Warehouse-native experimentation is about two things: where your data lives, and where your experiment metrics are computed. In a warehouse-native setup, both happen in your own data warehouse.

To see why that matters, it helps to know how experimentation tools have traditionally worked. You sent the vendor your data, or at least a specific slice of it, usually product analytics events. The vendor stored that data, organised it and structured it in their own databases. You then told the vendor how to turn those events into metrics, they computed them on their side, and you got a statistical analysis back.

A warehouse-native tool flips that around. Instead of asking you to send it your data, it connects to your data warehouse, whether that's BigQuery, Snowflake, Databricks or something similar. The computation happens where your data already is.

The term isn't unique to experimentation. It's a more modern way of describing data tools in general, from product analytics to BI, that work on top of your warehouse rather than on a copy of your data.

It also gets used loosely. Some tools sync data to or from your warehouse but still keep their own copy and compute metrics on their side. A simple test for any tool: if the vendor deleted its copy of your data tomorrow, would your experiment metrics still exist? With a warehouse-native tool they would, because they were computed in your warehouse all along.

How did big tech companies run experiments?

For most of my career I worked at companies that built their own experimentation platforms, including Booking.com, Meta and Canva. Many big tech companies built in-house partly because they started before there were high-quality vendors for product and engineering experimentation.

In those companies, all the data lives in the warehouse. Not just product analytics events, but purchases and revenue, financial data, customer support data, and a huge range of segments and dimensions, all tied to user IDs and other kinds of IDs. When it's time to compute metrics for an experiment, that happens in the warehouse too, connected to everything else the business knows.

Nobody called it warehouse-native. It was just how experimentation worked. What's changed is that you no longer need to build your own platform to work this way.

How does a warehouse-native platform work?

A warehouse-native platform connects to your warehouse and sends queries into it. You set up a mapping between the data you already have and what the platform needs to understand, such as which table records experiment exposures and where your metrics come from. The platform then sends queries into your warehouse to prepare everything the analysis needs. All the computation happens there.

Some data does still leave. At Cherto, for example, we extract aggregates from those metric calculations so we can run our statistics engine and other capabilities over them. But the raw, unit-level data always stays in the customer's warehouse.

One thing worth clearing up, because it gets confused a lot: warehouse-native describes the analysis, not the moment a user is assigned to a variant. Assignment still happens in your product, through an SDK or your own server code, in milliseconds. In a well-designed tool, your feature flags never query the warehouse when a page loads. The exposures they generate land in your warehouse, and that's where the metrics get computed.

Why go warehouse-native?

At Cherto we chose to be warehouse-native from day one. Here's why.

Most companies already have a warehouse

As soon as a company is even a little serious about data, it has a data warehouse. That wasn't true when the first generation of testing tools was built. Today it's the default, so it makes sense to build on it.

You can use all of your data

Traditional tools only ever saw the data you sent them. A warehouse-native tool can use as much of your data as you choose to give it: product events, purchases, revenue, support data and anything else that lives in the warehouse. That dramatically increases what these tools can do, because experiments can be measured against the outcomes the business actually cares about.

You control what the tool can see

Access to all your data doesn't mean handing everything over. You decide exactly what a warehouse-native tool can and can't see. Give it some product analytics tables and a slice of financial data, and nothing else, if that's what suits you.

One source of truth

In the old setup, data ended up replicated across systems. You might load product analytics events into your warehouse and also send them to a third-party vendor. Because of the nature of data, the two copies almost never match exactly. Results differ slightly, it becomes hard to trust either, and you can't reproduce the vendor's numbers yourself.

When metrics are built from exactly what's in your warehouse, you know the tool is reading from a single source of truth. If you want to, you can go into your own warehouse and replicate the results you're seeing.

Better privacy and compliance

Because raw, user-level data stays in your warehouse, a warehouse-native tool never needs to hold it. It's worth checking how a tool logs exposures, though. Some setups can still send exposure events to the vendor rather than to your warehouse.

A customer once asked us whether they could use Cherto to look up which variant a specific unit ID had been assigned to. The answer was no, and that was by design. Unit-level data never leaves their warehouse and never reaches Cherto's infrastructure. But because it's in their warehouse, they can easily query it themselves, and we're happy to help them do that properly.

No more sampling to control cost

With traditional vendors, a lot of the cost comes from how much data you send them, because storing and processing it is a real cost for them to carry. As companies grow, many end up sampling what they send, or sending only part of their data. There can be merit to that, but a warehouse-native approach lets you use everything.

What are the downsides?

Warehouse-native isn't free of trade-offs, and it isn't the right fit for every team.

Warehouse costs sit with you

The flip side of that last point is that storing and processing the data happens in your warehouse, so those costs shift onto you as the customer. Given where the modern data stack is heading, I think that's a reasonable trade, especially as warehouses and platforms keep getting more efficient. But it means the tool's behaviour in your warehouse matters a lot. More on that below.

You need some warehouse maturity

To go warehouse-native, you need a reasonable level of maturity in your own warehouse. Having data flow into BigQuery because Google Analytics set it up for you, and running a few dashboards on it, is different from having data in the right structures, with the right access, for a warehouse-native tool.

Teams that go warehouse-native usually have engineers, data engineers or data scientists who are comfortable working in the warehouse and can set up the integration. Tools that ask you to send them your data still have technical requirements, because you need a reliable way to get the data to them. But less of the burden sits in your warehouse.

Results aren't instant

Warehouse-native results update when the analysis jobs run, not the moment an event happens. At Cherto, experiment results refresh daily by default, and how fresh they can be also depends on your warehouse and how data lands in it. Vendor-hosted tools that ingest events directly can often show results closer to real time.

For most experiments, which run for a week or more, a daily refresh is enough. Launch is the exception, because that's when you want to catch problems fast. Where the setup allows it, we run near real-time launch checks for the first few hours of an experiment, and teams can control when and how often analytics jobs run.

What should you look for in a warehouse-native tool?

The first wave of warehouse-native tools was impressive just for being warehouse-native. The bar is higher now. A warehouse-native tool is running queries in your warehouse, on your bill, so it needs to be a good citizen there. When you're evaluating one, look for:

• Optimised queries. The vendor should invest heavily in making its queries efficient, because every inefficiency shows up on your bill.
• Transparent queries. You should be able to see the SQL the tool runs.
• Tagged, traceable queries. Every query should be labelled so you can trace it back to the platform and audit it.
• Admin and governance controls. Admins should be able to put guardrails in place, like limits on how large queries can get and the ability to stop jobs.

Don't just ask whether a tool is warehouse-native. Ask whether it behaves well once it's in your warehouse. It's something we care a lot about at Cherto, which is why teams can control when and how often our analytics jobs run.

Warehouse-native vs traditional experimentation tools

	Traditional (vendor-hosted)	Warehouse-native
Where raw data lives	Copied to the vendor	Stays in your warehouse
Where metrics are computed	Vendor's infrastructure	Your warehouse
What data experiments can use	Events you send the vendor	Anything in your warehouse you choose to share
Source of truth	A second copy that rarely matches yours exactly	The same data the rest of the business uses
Reproducing results	Hard, because you can't see the vendor's data	Query your own warehouse
Privacy	User-level data held by a third party	User-level data never leaves your warehouse
Result freshness	Often close to real time	Typically daily, depending on your warehouse and setup
Main cost driver	Data volume sent to the vendor, often leading to sampling	Warehouse compute and storage, paid by you
What you need in-house	A reliable way to send events	A reasonably mature warehouse and people who know it

Is warehouse-native the future?

I deeply believe it is, for data platforms and technology in general, not just experimentation. Companies should be able to use all of their data, in a connected way, in the same place every other tool is reading from. That context belongs to them.

The big tech companies figured this out years ago by building their own platforms. Warehouse-native tools make the same way of working available to every team.

Common questions

Does warehouse-native make feature flags slower?

No. Assigning users to variants happens in your product through an SDK or your own server code, in milliseconds. Only the analysis runs in your warehouse.

Do we need a data warehouse?

Yes. Warehouse-native tools build on your warehouse, so you need one, with your experiment exposures and the data behind your metrics in it, and someone who can set up the connection.

Can a warehouse-native tool see individual users?

A well-designed one doesn't need to. Raw, unit-level data stays in your warehouse, and only aggregates reach the platform. If you need to check what a specific user saw, you query your own warehouse.