Hive Mind: Our Approach to Self-Service Analytics

How we built a data analytics agent that delivers fast, accurate answers by pairing frontier models with curated context.

At Robinhood, we set out to provide self-serve analytics to every employee. Since launching last spring, 90% of our analytical queries run through agents. Over 400 people who had never written SQL are now pulling their own data.

The hardest part? Teaching the agent which tables to trust and why.

Consider a user asking, “How many Gold subscribers do we have?” Across several runs, the agent might pull from metrics.gold_users, gold.subscription_view, or one of the daily/weekly/monthly rollup views. Each table appears relevant, and each answer looks plausible, but each produces a different subscriber count.

Our first approach was to give agents powered by frontier models direct access to our data lake. A small set of manually run evaluations quickly revealed the problem: the agents produced too many false positives. Fewer than 200 of our 80K tables had adequate documentation, while approximately half of the charts in our BI tool had semantically duplicate names. At a company that moves as quickly as Robinhood, assets get created fast, and institutional knowledge often fills the gaps. Without that knowledge, agents couldn’t reliably determine which assets were correct or why.

This used to be discernible by data experts. But the new consumers of data are not experts. The new users can't tell a confident wrong answer from a right one. With curated context pointing to the canonical table, the agent answered correctly and identically every time. Enter Hive Mind: the context and decision layer for data agents.

To understand how we gave Hive Mind, the name we gave our agent, the context it needed, it helps to start with the key components of Robinhood’s analytics stack: 

• Apache Superset: our BI layer: dashboards, charts, and SQL Lab, the in-browser SQL editor. Where people already go to look at data.

• DataHub: our data catalog: table docs, column metadata, lineage.

• Trino: our query engine over the data lake (the agent gets its own dedicated cluster).

• DEX: our data engineering agent; it writes and modifies the pipelines that produce the tables.

In most cases, we already had a lot of the context our users were asking about, right there on their screens. The issue lies in how to curate all that context. While Robinhood is shifting towards autonomous analytical agents, we still have a treasure trove of curated information stored in Apache Superset. By starting our analytical agent within Superset we gave it the power to bypass the majority of critical decisions when a question appeared to be about the dashboard a user was already on. No more guesswork: the answer is almost always contained within the space the user is already working in.

Hive Mind is what handles everything beyond that dashboard. The flow is simple to state: the agent plans its intent, assembles context from Superset and DataHub, writes SQL, verifies it, and responds with provenance. Every answer shows the exact SQL it ran, turning correctness into a two-second check by a data expert. The loop itself is surprisingly simple, the main piece to build was the agent infrastructure. It gets an isolated runtime with a tailored system prompt explaining clearly and directly the tools on hand, and how/when to use them. We used an off-the-shelf agent harness with a deliberately narrow set of Superset and DataHub tools. The ones that mattered most fell into four categories: retrieving task-specific instructions, searching the catalog and dashboard metadata, reading table documentation, and running or pre-validating SQL. This is the core of self-serve analytics. Everything else is auxiliary functions we added later.

We made safety structural rather than aspirational: the agent runs as the actual user (inheriting all of Superset's access controls), queries are restricted to read-only statements, and everything executes against a dedicated Trino cluster for agents, with its own limits so an enthusiastic agent can never starve production workloads. Separating the infrastructure and drawing real boundaries between agents and the rest of the system is what keeps it safe.

Instead of teaching the agent to explore our data lake, we built the context so it never has to. We set out to tackle the problem at the source by documenting all of the most used tables in the data lake. Today, roughly half of all requests to the agent are answered from those curated tables alone. 

But handwritten docs rot, so we built an agent to automatically update table docs for us. Starting by backfilling >1k tables (bringing us to 90% usage coverage), then wiring up the documentation skill to our data engineering agent “DEX” to keep up with the tedious work. The table documentation agent uses the pipeline source code, DataHub glossary terms, and pre-existing human documents. When the source code alone can't answer something, the agent doesn't guess. It asks the table's owners a handful of targeted questions and waits for a human answer. Humans always review the work of the agents as it is now paired with tasks and tested alongside the changes. 

We measured how much this matters. In a paired experiment, we ran the same question set with table documentation present and withheld. Removing the docs dropped table-choice accuracy from 100% to 51%, stretched median time-to-answer from 49 seconds to roughly 13 minutes, and cut clean completions (answered without a timeout or error) from 96% to 42%. 

Our learning: The table documentation is critical to agent performance. Without it, the agent is guessing. Today, roughly 25k column definitions and 5k transformation expressions sync hourly into DataHub for our agents to leverage. Ask "How many Gold subscribers do we have?" today, and the agent answers from the table that the docs marked as canonical, shows the SQL, and footnotes the other two definitions in case you meant one of those.

The agent's own conversations turned out to be our best map of what's missing as we built out our context layer. At the start, ~13% of user messages contained either frustrated responses or corrections to the agent. We shrank that down to <5% over the next few months. The traces surfaced clear patterns: tables were missing proper documentation or a wrong assumption was made in which metrics to use for given tasks. This gave us power in tackling the questions users actually cared about asking first, and brought accuracy to the covered domains from ~40% to ~95% in our evals. 

Most of that gain came from one fix: tables and metrics that simply couldn't be found before. When metrics for a given domain get documented the agents get the ability to reason over what action to take, rather than attempting to infer it through discovery like sampling tables and searching over our Superset semantic layer for keywords. This directly related to findings we just covered in context, where removing the context for a given domain drastically reduced the accuracy. So we leveraged the feedback to dictate which parts of the context needed to be bootstrapped until our corpus was largely covered.

The corpus will never be covered 100% though. There will always be a long tail of questions that won’t have perfect context to drive optimal decision making for the agents. This led to our next theory, which was to implement a mandatory verification layer for our agents to catch these missing or incorrect table decisions before they surfaced the incorrect decisions to our users.

We tried mandating a critic loop for our agents, as this seemed like a logical step in catching incorrect assumptions. The mandated verification agents ended up being 1.5x slower, consumed ~34% more tokens, and added no statistically significant accuracy gain. Agents that were confidently wrong were still confidently wrong, we only caught ~5% of cases where agents could self correct. The tool couldn’t correct missing context, but we could optimize how the agent leveraged the context it already had.

A simple optionally invoked verify_sql tool ended up being a small cost for significantly more accurate SQL overall. The tool itself layers static parsing, an EXPLAIN validation pass, DataHub table metadata, and a failure-risk classification model. We wanted to see how it could be used to optimize inefficient queries, or debug queries that would fail. We ran 120 hand-picked tasks based on complex user SQLLab sessions and had an LLM-as-judge check correctness. The three-arm comparison made the case clearly:

With the tool, the agent ran EXPLAIN on its actual candidate edits and caught the bad filters before execution. Pooled across both task types, agents with verify_sql failed their first attempt 17% less often, rescuing 15 tasks against 8 hurt. Interestingly, metadata only agents exited early on optimization tasks, leading to more broken queries than having nothing at all. With both the metadata to suggest quick and informed updates, and fast verification on SQL correctness, both tasks hit extremely good results.

At Robinhood, we started with a massive data footprint, 25K jobs feed 80K tables that power 50K charts across 1.5K dashboards. With Hive Mind, agents are now able to run 90% of our analytical queries. Since our launch last spring the data agent embedded in our BI tool “Apache Superset” alone has over 25k messages sent, and interactions from well over a third of all employees. On user surveys users reported 2-5x speedup, and 1/4 of our active users reported that without the tool they couldn’t get the information they needed at all. We have also seen ~45% decline in manual SQL usage, but ~25% increase in total users running SQL. More analytics are being served, and we have maintained high accuracy throughout the increase in AI adoption. 

As of writing, we have more than 50% of dashboards and charts being created/edited by AI every month. This has led to an increased number of data consumers, while the curators of context remained the same data experts. The goal was to accelerate human curation of assets, not create more slop.

The most dangerous failure in data is the confident wrong answer. 

A user can't tell a false positive from the truth, and it travels. Keeping humans in the loop is what got us ahead of it. Humans review the generated docs, curate knowledge for the tables people actually use, and write the evals with experts who know the right answers. If you're building a data agent, measure your false-positive rate before you trust anything else about it. Building a small eval corpus of a few dozen Q/A pairs goes a long way. After that, sample production logs to make sure the agent is making the right choices more often than not.

Capturing feedback and error traces helps tremendously in the bootstrapping of your corpus, but will slowly taper out once you have covered the main missing docs. You shouldn’t play whack-a-mole with adding every decision or failure case to the corpus. The long tail of questions and institutional knowledge will always exist, and there are limitations to what information can be presented to your agents and how useful it will be. Knowing the distinction between different metrics definitions is valuable, but hard coding how to use them leads to confident incorrectness in our experience. 

Close the context gap before anything else

Even with the most powerful models, there still needs to exist context for the agent to leverage. We saw this clearly in our context and verification experiments. This is another reason why putting a data agent inside your existing BI tool is so powerful: humans have already curated a ton of context for it to use. The largest gains come from making sure your key business metrics and table definitions are well defined in your BI/analytics layer, where every agent and every human reads the same answer. That alone gets you most of the value from a data agent. Everything built on top of it is a bonus.

At Robinhood we are looking at more personalization and self-serve of the entire analytics pipeline. Humans are still involved in multiple parts of the chain, but as agents become more powerful the humans can focus on curating and ensuring correctness in their domain for agents rather than serving humans directly. The plan is to give everyone data superpowers wherever they might need it.

Thanks to Adil Faraz and Srikanth Devidi for their leadership and guidance throughout this project, and to the early users across Robinhood whose questions and corrections made the agent better every week.

Brandon Sovran is on the Data Tooling team at Robinhood, where he runs Apache Superset and builds the data agents inside it, making trustworthy self-service analytics available to everyone at the company.

”Hive Mind” and “DEX” are proprietary internal workflow solutions developed by employees of Robinhood. It is not affiliated with or related to any external technology tools. 

Interested in learning more about building at Robinhood? Check out the careers page: https://careers.robinhood.com/