Automating Database Safety: Building a Security-First AI for Storage Operations

Our storage infrastructure is the bedrock of our products, processing millions of financial operations across hundreds of distinct Postgres databases running on Amazon Aurora. The team entrusted with this critical infrastructure, the Storage team, is accountable for its reliability and safety.

As Robinhood scaled, so did the operational load on this team. We hit a tipping point where the manual processes that once ensured safety became bottlenecks. Two challenges emerged:

The Pull Request Queue: The Storage team was responsible for manually reviewing a high volume of schema migration PRs, often exceeding 100 per month. Each review was a meticulous, high-stakes process. An engineer had to cross-reference the proposed change against best-practice runbooks, manually look up table sizes and statistics to gauge the potential impact of a migration, and check for common "gotchas" like acquiring locks on high-traffic tables. This process was time-consuming and prone to human error.

Mean Time to Recovery (MTTR): During a production incident, every second counts. On-call engineers found themselves navigating a fragmented landscape of observability tools, jumping between monitoring dashboards, log aggregators, and bespoke scripts to piece together the state of a database. This "detective work" extended incident response times and created a steep learning curve for engineers, making it difficult to get a holistic view of the system under pressure.

Our existing manual processes could not scale to meet the compounding complexity of our systems. The solution required a new, scalable automation platform.

Our vision was to leverage LLMs to reason over our complex infrastructure data. The MCP, an open standard for exposing tools and data sources to LLMs, was the clear architectural choice.

The most straightforward path seemed to be using an existing open-source Postgres MCP server, which would connect an LLM directly to a read-replica of our production databases. However, after a risk assessment, we concluded that this approach, while simple, presented unacceptable risks for a financial services company.

The Security Risk of Direct Access: Granting an LLM direct query access, even to a read-replica, was a security non-starter. This would expose the model to the production dataset, which contains sensitive customer information. Furthermore, it created a new attack vector. A compromised model or a cleverly crafted prompt could theoretically lead to data exfiltration or other unforeseen security incidents. The principle of least privilege dictated that the LLM should not have database credentials, period.

The Performance Risk of Unpredictable Queries: LLMs are powerful, but they are not database administrators. An LLM with arbitrary query execution rights could generate inefficient SQL, causing performance degradation. A single suboptimal query joining large tables or a "query storm" from the model could overwhelm our production databases, impacting customers.

This led us to our core architectural decision: Instead of connecting our AI to the database, we connected it to our database monitoring tool.

We built a custom MCP server on top of pganalyze, our existing Postgres monitoring solution. pganalyze periodically collects metadata "snapshots" of each database; information about schema, table/index statistics, lock contention, and query performance. Snapshots include normalized query text but no row-level customer data. By building on these snapshots, we created a secure, sandboxed data source for our AI agents, giving them the rich context they needed without exposing customer data or risking production performance.

With our security-first foundation established, we designed a three-layered ecosystem. A key principle was modularity: once a capability is exposed via our pganalyze MCP server, it becomes a composable building block for any number of future Storage AI agents.

The pganalyze MCP Server: This is the heart of the system. We built a custom server that wraps pganalyze’s GraphQL APIs, exposing over 90 discrete "tools" for fetching diagnostics, schema, and performance data. For example, a tool might fetch the top 5 largest tables, check for index bloat, or retrieve recent vacuum activity. All data is sourced from the metadata snapshots, ensuring our AI agents never issue queries against production databases directly. New tools are added declaratively via a JSON config, defining a new pganalyze GraphQL endpoint to expose is a one-line change. The server applies TTL caching at the tool level to keep latency low under high-volume PR analysis. The pganalyze MCP server is also consumable by third-party MCP clients like Cursor and Claude Code, not just our own agents.

Intelligent Agents & The Custom MCP Client: Our agents are the LLM-powered applications that consume the tools from the server. To enable this, we built a custom MCP client. This client is more than a simple API wrapper; it handles tool discovery from the server, serializes requests, manages the context of the conversation, and provides robust error handling. This abstraction allows agent developers to focus on workflow logic without worrying about the low-level mechanics of the MCP protocol. All model calls flow through our internal LLM proxy, which centralizes authentication, audit logging, and per-team usage controls, so the agents themselves never hold model-provider credentials.

Integrated User Interfaces: We surfaced the agents' capabilities in the two places our engineers live every day: GitHub and Slack.

This architecture powers our two flagship applications:

The PR Agent is our automated line of defense against risky schema migrations. When a developer opens a PR containing a .sql or Django migration file, our Buildkite CI pipeline triggers the agent. The agent's workflow is as follows:

• Code Analysis: The agent runs an iterative tool-calling loop, using the LLM to parse the PR's content and diff, identifying the exact nature of the proposed change (e.g., adding an index, altering a column type).

• Contextual Data Fetching: It invokes tools on our MCP server to fetch real-time stats about the affected tables, such as row count, table size in GB, and existing indices.

• Framework-Aware Rule Application: The agent consults framework-specific runbooks. For example, in a Go+sqlc service, adding an optional column is unsafe if any active query uses SELECT * or RETURNING *, since the previously deployed version of the service will fail to scan the result. In a Django service, the equivalent change is generally safe on read paths because the ORM emits explicit column lists, but column drops still require staged removal due to implicit query expansion.

Risk-Labeled Feedback: It posts a detailed, summary back to the GitHub PR as a "check run." This includes the data it fetched and a clear risk label, such as "High Risk: Acquiring an ACCESS EXCLUSIVE lock on `table_name` (1.6 billion rows, 500 GB) will likely cause a production outage."

Example Schema Migration:

Consider the following migration where we add a new index on a large table:

The analysis posted back to the GitHub PR explicitly calls out the size of the table, and the risk of timing out application deployments if this index creation is run as part of it.

Moreover, the PR is explicitly labelled as “High Risk” for clarity.

The above example showcases a schema migration defined as SQL, but we also support a similar workflow for Django (python-based) schema migration definitions.

Separating Judgment from Explanation

The agent never decides whether a schema migration is safe – it only explains the change. That call is made by a deterministic checker that tests each migration against explicit rules and fails closed, so anything it can't prove safe goes to a Storage engineer.

This lets us put AI on a critical path: the model makes engineers faster, while the rules keep migrations safe.

The Storage Assistant democratizes database expertise. It's a conversational bot, built on the same MCP client as the PR Agent, that gives any engineer at Robinhood the ability to query complex database insights using natural language. Engineers can ask questions like:

"What are the top 5 largest tables on the primary_user_db cluster?"

"Were there any long running queries on the crypto_db in the past hour?"

Because engineers reach for Slack in different ways depending on the situation, the assistant supports three modes of interaction:

• Channel mentions: when @-mentioned inside a thread, it returns structured summaries scoped to that thread.

• DMs: in direct messages, it maintains multi-turn investigation context for interactive debugging.

• Empty mentions: when @-mentioned without a prompt, it summarizes the surrounding thread or channel, especially useful during incidents.

From Hours to Minutes: Quantifying the Impact and Looking Ahead

Adopting this AI-powered ecosystem has fundamentally changed how we manage our storage infrastructure.

1. Massively Parallel, Instantaneous Reviews: We eliminated the manual review bottleneck. Automated schema analysis reduced the feedback loop for developers from 10+ minutes to under two minutes.

2. Fleet-Wide Safety and Empowerment: We expanded automated safety checks from a handful of critical services to our entire fleet of hundreds of services. Simultaneously, our Storage Assistant is now used by dozens of application and infrastructure engineers, successfully serving hundreds of complex queries each month, empowering them to self-serve information that was previously siloed.

3. Elevating the Storage Team: The Storage team's operational burden was drastically reduced, freeing them from repetitive toil to focus on higher-leverage work like performance tuning, long-term capacity planning, and building the next generation of our data platform.

One improvement we're excited about is closing the loop. Today the agent is one-way: it posts its assessment and never learns what happened – whether the reviewer agreed, or the migration ran cleanly in production. Persisting those outcomes lets us benchmark against real verdicts, upgrade models with confidence, and eventually feed similar past cases back into the agent's context at review time.

Our journey reaffirmed the power of building on open standards. While we built our own MCP server out of necessity, the ecosystem is now mature enough that teams have multiple options. 

Our key insight was that abstracting our tools behind the MCP protocol was the correct long-term bet. This project's success has reinforced our company-wide strategy of adopting MCP as an integration layer for AI tools, and the pganalyze server is now one of several MCP servers we leverage at Robinhood, connecting our AI agents to a vast array of tools for metrics, logging, and cloud infrastructure management.

• Security-first AI architectures can unlock automation without exposing sensitive production data.

• Metadata snapshots provide rich operational context while preserving customer privacy and database safety.

• MCP-based tooling creates reusable building blocks that can power multiple AI applications across an organization.

• Integrating AI directly into developer workflows accelerates feedback loops and reduces operational toil.

• Open standards and modular abstractions make it easier to scale AI capabilities across infrastructure teams.

Andrew Fan and Ashvin Vinodh are engineers with experience building scalable infrastructure and tooling to improve database reliability, operational efficiency, and platform safety across Robinhood’s storage ecosystem.

Rambabu is a Senior Software Developer on Robinhood's Storage Infrastructure team, focused on database scalability, reliability, and the proxy layer powering Robinhood's Postgres fleet.

Interested in learning more about building at Robinhood? Check out the careers page.