The Million-Dollar Query

People sometimes ask why we started unLAP.

The short answer is that query optimization should not require weeks of man-hours, millions of dollars, and a team of skilled warehouse performance specialists to do.

The longer answer starts with a BigQuery job that was burning roughly $2,500 a day.

The Million-Dollar Query

Someone in finance once reached out to me about a rogue BigQuery job that was on track to cost the company over a million dollars a year.

I assumed the next question would be how quickly we could get rid of it.

But, to my surprise, the finance guy wasn’t asking us to change the query. What he essentially told us was: “I just want to know where this is coming from. This service matters, it provides real value to the company, and I can justify the cost.”

The Query Was Not the Hard Part

Luckily for him, I already knew exactly what the query was. There was no real visibility into this part of the system, so, six months earlier, I had taken the initiative to build some very ad hoc visibility myself.

That is how I had already found the inefficiency and suggested a change that would have fixed it.

The response back then was basically a shrug.

“It is what it is,” was the attitude from the engineers.

That query had been there for years, and it was still working perfectly. Why would you go through the trouble of even trying to fix it?

It took until six months later, when finance noticed the anomaly through my visibility change - a feature nobody else requested - for the team to finally agree to take action.

Why a Known Fix Still Took Three Months

Technically, it took a few lines of code across a couple orgs to create the necessary optimization artifacts. But even though I knew exactly how to optimize the query, it still took about three months of meetings, follow-ups, technical explanations, and persistent pushing from both me and my manager to get the right people in other teams to make the required changes.

That experience showed me something about the wider OLAP landscape:

• writing efficient OLAP SQL is still treated like a black art
• most teams cannot spot bad SQL patterns in the first place
• teams are rewarded for shipping, not for efficiency
• optimization work never has clear ownership, so it stays everyone’s problem and nobody’s priority
• even small optimizations can require coordination across multiple teams to get done

That is why expensive queries survive so long. Not just because nobody notices them, but also because very few teams have both the expertise and the operational path to fix them quickly. Companies are literally willing to spend millions of dollars per year rather than deal with optimizing them.

The worst part was that I knew of many other queries that also burnt money. But after spending six months gathering evidence and three more months forcing one fix through, I knew what would happen to the rest: each one would have to compete with feature work and cross-team coordination. By the time I had fixed one inefficient query, many more would have been written.

The Day After I Left

Fast forward to the day after I left the company.

I heard a ping on my phone and my notifications lit up. It was the same finance guy, pinging me (who had already left) and another engineer because BigQuery costs had spiked and people were yet again scrambling to figure out what this expensive query was, where it was running, and who it belonged to.

The search was still ongoing days later. It was the same story all over again.

What unLAP Is For

We started unLAP after seeing this failure mode happen over and over again.

Query inefficiencies should not sit in production for years because they were never discovered, the expertise to optimize them is too specialized, or the cost of getting a fix to production is higher than the millions of dollars that proper optimization could save.

That is what unLAP is for: making it actually possible to even identify the problem, produce a fix, verify it, and get it rolled out both quickly and properly.

That is why we started unLAP.