Introducing Genie Code | Databricks Weblog

We’re excited to announce Genie Code, the latest addition to the Databricks Genie household. Up to now six months, agentic coding instruments have essentially modified software program engineering; Genie Code brings that very same transformation to information groups. Genie Code can autonomously perform advanced duties reminiscent of constructing pipelines, debugging failures, transport dashboards, and sustaining manufacturing techniques.

Not like brokers that focus solely on writing code, Genie Code additionally operates as a proactive manufacturing agent. It screens your Lakeflow pipelines and AI fashions within the background, triaging failures, dealing with routine DBR upgrades, and investigating anomalies earlier than your crew even notices.

It does all this by deeply integrating with Unity Catalog in order that it understands your enterprise’s information, semantics, and governance insurance policies. Genie Code considerably outperforms a number one coding agent by greater than 2x on real-world information science duties.

Rise of Agentic Knowledge Work

Agentic coding instruments have reworked software program engineering, transferring builders past autocomplete and towards agent-driven growth. With a single immediate, engineers can now scaffold options, refactor code, and deploy prototypes in seconds. This shift has been pushed by advances in LLMs and by agentic techniques that may interpret the advanced context of recent software program codebases.

Most brokers in the marketplace concentrate on code as the ultimate product. Nevertheless, for information groups, code is merely a car to govern and perceive the underlying information. That is precisely why software-centric brokers usually wrestle with information work. In a knowledge ecosystem, context lives not simply within the script but additionally in utilization patterns, lineage, and enterprise semantics. 

Accessing this context is important as a result of the stakes are excessive. Dashboards drive enterprise choices, pipelines energy manufacturing techniques, and machine studying fashions affect real-world outcomes. For information groups, the pace and leverage supplied by brokers have to be paired with absolute accuracy, reproducibility, and governance.

Genie Code is an AI agent constructed particularly for information. It leverages Unity Catalog to robotically curate essentially the most related information and content material as you’re employed. It creates customized search indexes, customized directions, data shops, and extracts utilization patterns from lineage. Better of all, it will get smarter the extra your crew makes use of it. This deep integration into Unity Catalog is way superior to any system that merely reads the info from the skin.

We have seen the affect of Genie and Genie Code firsthand at Databricks, throughout each technical and non-technical customers. Our gross sales crew makes use of it to get an entire image of each buyer earlier than conferences, summarizing key consumption metrics, assist tickets, and up to date interactions in seconds. Product Managers use Genie Code to construct dashboards from a hand-drawn sketch of charts and graphs. Our finance crew runs budget-versus-actual evaluation and superior ROI modeling. Our management crew solutions information questions in actual time throughout strategic discussions, decreasing follow-up and accelerating advanced choices. Throughout the corporate, these instruments have modified how we work with information.

What Genie Code Does:

• Acts as an skilled machine studying engineer: Genie Code handles full ML workflows end-to-end. It causes by means of advanced issues to plan, write, and deploy fashions, whereas logging experiments to MLflow and fine-tuning serving endpoints for peak efficiency.
• Deep information engineering experience: Whereas a novice engineer may write a script that works on check information, Genie Code designs like a senior architect. It accounts for the variations between staging versus manufacturing environments, builds workflows for change information seize and applies information high quality expectations.
• Proactively maintains and optimizes: Genie Code screens Lakeflow pipelines and AI fashions within the background to triage failures and examine anomalies. It autonomously analyzes agent traces to repair hallucinations and tunes useful resource allocation earlier than a human intervenes.
• Understands enterprise context: Built-in with Unity Catalog, Genie Code enforces present governance insurance policies and entry controls. It understands enterprise semantics and audit necessities and federates enterprise information, together with information from exterior platforms.
• Improves over time: Genie Code grows smarter the extra groups use it. Via persistent reminiscence, it robotically updates inside directions based mostly on previous interactions and coding preferences. On inside information science duties, Genie Code outperforms main coding brokers 77.1% to 32.1% on high quality.

With Genie Code, information groups transfer from prompting a copilot to delegating actual work: constructing pipelines, debugging failures, transport dashboards, and sustaining manufacturing techniques — autonomously, finish to finish.

At SiriusXM, Genie Code helps every thing from authoring notebooks and complicated SQL to reasoning by means of desk relationships and debugging pipelines. It acts as a hands-on growth companion that helps our information groups ship high-quality work in much less time. — Bernie Graham, VP Knowledge Engineering, Sirius XM

Highest High quality Agent for Knowledge and AI Work

Genie Code will not be powered by a single mannequin. It’s an agentic system that routes duties throughout a number of fashions and instruments, robotically selecting the right mannequin for every job, whether or not that could be a frontier LLM, an open supply mannequin, or a customized mannequin hosted on Databricks. This eliminates the necessity for customers to manually change between fashions or guess which one will produce one of the best outcome.

Genie Code can be deeply built-in with Databricks APIs, permitting it to determine the suitable information property, assemble wealthy context, and generate larger high quality queries. Databricks Analysis repeatedly tunes the system, benchmarking the newest fashions from main AI labs alongside customized fashions operating on the platform.

In our current efficiency benchmarking on real-world information science and analytics duties collected from inside customers, Genie Code considerably outperformed a number one coding agent outfitted with the Databricks Mannequin Context Protocol (MCP) servers.

• Genie Code: 77.1% Solved duties
• Main Coding Agent + Databricks MCP: 32.1% Solved duties

Genie Code Helps the Full Lifecycle of Knowledge Work

Prepare and Consider Machine Studying Fashions

Genie Code acts as a devoted ML engineer embedded in your workflow. Ask it to “prepare a forecasting mannequin predicting gross sales in @sales_table” and it’ll purpose by means of the total pipeline: 

• Figuring out and profiling options
• Break up coaching, validation, and check datasets accurately
• Prepare a number of mannequin varieties and evaluate them, operating hyperparameter sweeps to coach the absolute best mannequin.
• Evaluates outcomes throughout metrics like AUC, F1, RMSE, and R²
• Generate plots for characteristic significance, confusion matrices, and ROC curves
• Monitor experiments in MLflow
• Suggest enhancements based mostly on mannequin diagnostics

As soon as deployed on Databricks Mannequin Serving, Genie Code stays within the loop: it could examine endpoint well being, analyze traces, and advocate optimizations. You may learn extra on this within the “From Code to Manufacturing: Observability with Genie Code” part under.

Genie Code modifications how our information groups function. As a substitute of sewing collectively notebooks, pipelines, and fashions manually, we will hand off advanced workflows to an AI companion that understands our information, governance, enterprise context, and inside libraries reminiscent of Repsol Synthetic Intelligence Merchandise. It accelerates every thing from time collection forecasting to manufacturing deployment, with out sacrificing rigor or management. — Emilio Martín Gallardo, Principal Knowledge Scientist, Knowledge Administration & Analytics, Repsol

Create Manufacturing-Prepared Knowledge Pipelines

Genie Code is your skilled information engineer, constructed that will help you design and evolve dependable information pipelines.

• Create pipelines from pure language: Describe what you want and Genie Code generates an entire Spark Declarative Pipeline with ingestion, transformations, and information high quality expectations in-built.
• Prolong present pipelines: Add datasets, modify transformations, write AutoCDC flows, configure Auto Loader, and apply information high quality expectations, all inside the context of your present pipeline.
• Perceive pipeline conduct: Examine outputs, hint information circulation into downstream tables, and floor surprising modifications in row counts or schemas.

Genie Code has moved us past assisted coding into true agentic information engineering. It may well analyze our Lakeflow pipelines, suggest multi-file modifications with diffs, execute runs with safeguards, and iterate by means of failures till points are resolved. It feels much less like autocomplete and extra like a collaborator embedded in our workflow. — Nishit Gajjar, Tech Lead, International Infrastructure Know-how Supplier

Create Dashboards with Reusable Semantic Definitions

Genie Code can generate visualizations, configure filters, and arrange multi-page dashboard layouts, all with reusable semantic definitions. It connects these definitions to filters, calculations, and layouts that scale as dashboards develop, serving to groups transfer sooner whereas sustaining consistency.

With Genie Code, our groups are delivering AI-driven analytics and automatic workflows in weeks, not months. Low-code brokers assist us transfer sooner whereas staying aligned to governance, enabling challenge and engineering groups to get natural-language insights from advanced information with out slowing supply. — Russell Singer, Chief Knowledge Architect, Bechtel Company

Autonomous Multi-Step Planning and Execution 

Present a high-level goal, reminiscent of “Establish flight delay dangers and construct a monitoring dashboard”. Genie Code causes by means of the necessities, formulates a multi-step plan, and executes it throughout all Databricks Notebooks, AI/BI Dashboards, and Lakeflow in a single dialog thread.

What we’re seeing at Danfoss is that Genie Code modifications the roles inside a knowledge crew, supporting our strategic concentrate on digitalization and AI. Knowledge scientists nonetheless present route and overview, however engineers, analysts, and area consultants can now actively work in notebooks with the assistant and contribute to superior analytics workflows. It turns information science into a way more collaborative crew exercise. — Radu Dragusin, Principal Engineer, Knowledge & AI, Danfoss

Exploratory Knowledge Evaluation with Deep Contextual Search

Genie Code makes use of recognition, lineage, code samples, and Unity Catalog metadata to search out essentially the most related datasets for any evaluation. This deep contextual search eliminates the guide effort of looking for information and ensures that your work relies on essentially the most correct and steadily used tables inside your group.

I’m genuinely mesmerized. Genie Code seems like a glimpse into the way forward for how information work will get finished. — Sameer Yasser, Sr. Knowledge Engineer, Sundt Development

Customization and Extensibility

Genie Code is a versatile platform designed to be tailor-made to your crew’s particular requirements and exterior tech stack. There are three major methods to increase its capabilities:

1. Exterior Tooling by way of Mannequin Context Protocol (MCP)
   Genie Code helps Mannequin Context Protocol (MCP), an open customary that enables it to securely work together together with your exterior instruments, APIs, and documentation. This permits autonomous workflows that stretch past the Databricks workspace.

For instance, once you’re assigned a Jira job to coach a brand new ML mannequin, Genie Code can robotically collect context from it, carry out the duty, and replace the ticket with the outcomes.

Join Genie to your inside Confluence, Google Drive, GitHub, or Notion by way of MCP so it could reference your crew’s particular runbooks and information dictionaries when troubleshooting.

2. Agent Abilities: Outline domain-specific capabilities to show Genie Code the right way to carry out advanced duties persistently. Whether or not it’s a selected method your organization handles PII masking or a customized framework for information validation, Abilities make sure that the AI follows your group’s finest practices each time. Abilities comply with the open Agent Abilities format.
3. Reminiscence: Genie Code grows smarter the extra you utilize it. Via persistent reminiscence, the agent robotically updates its inside directions based mostly in your previous interactions. It learns your coding preferences, remembers which datasets you utilize most steadily, and retains context throughout periods.

From Code to Manufacturing: Observability with Genie Code

Writing code is barely step one. Sustaining it’s the actual problem. Genie Code acts as an observability agent to maintain your information and AI workflows wholesome. Whereas 1000’s of shoppers use Databricks to serve refined AI functions, debugging these fashions in manufacturing is commonly essentially the most time-consuming a part of the lifecycle.

Genie Code now integrates immediately with Databricks Mannequin Serving and MLflow 3.0 to automate this course of. As a substitute of manually looking out by means of logs and traces, you need to use Genie for:

• Endpoint well being checks: Get a full standing report throughout compute, request dealing with, and server logs in a single immediate.

• Agent high quality evaluation: Floor refined points like hallucinations, incorrect software calls, and consumer frustration patterns throughout advanced agent traces in actual time.

• Manufacturing troubleshooting: When incidents happen, Genie cross-references server logs and metrics to automate the primary spherical of analysis and cut back time to decision.
• Endpoint optimization: Get suggestions on provisioned concurrency, {hardware} configs, and auto-scaling based mostly on Databricks finest practices.

Background Brokers that Preserve Workloads Wholesome

Genie Code is designed to work within the background in order that your information stays wholesome even after you shut your laptop computer. You may deploy a number of brokers in parallel to deal with the operational work that usually consumes a knowledge engineer’s week. These background brokers transfer past reactive assist towards proactive upkeep by dealing with repetitive duties reminiscent of responding to job failures and managing routine upgrades. When a pipeline breaks, the agent identifies the foundation trigger and suggests a repair solely after validating it in a safe sandbox setting. 

For instance, if a manufacturing pipeline fails as a consequence of a schema mismatch, reminiscent of a column altering from an INT (150) to STRING (“150 USD”), Genie Code will pinpoint the failure and robotically repair the damaged pipeline. 

Background brokers are coming quickly. 

Grounded in Unity Catalog: Built-in Safety and Governance

Genie Code is constructed immediately on Unity Catalog. This integration ensures that the agent follows the identical safety and governance guidelines as the remainder of the Databricks platform.

When Genie Code searches for information, it solely surfaces property the consumer is permitted to entry. When it builds a pipeline, it adheres to present lineage and entry controls.

• Native Revision Historical past: Each edit is tracked by means of the Databricks versioning system. You may roll again modifications throughout notebooks, queries, recordsdata, and Lakeflow pipelines with full confidence.
• Constructed-in Guardrails: Genie Code is designed to proactively ask for affirmation earlier than executing code that may modify underlying tables.
• Entry Management Enforcement: Genie Code by no means exposes information property {that a} consumer will not be permitted to see.
• Complete Audit Logging: Your group maintains full visibility into how Genie Code is getting used by means of present audit infrastructure.

Obtainable in your Workspace at this time

Genie Code is Typically Obtainable in your Databricks workspace proper now. You will discover the Genie Code panel in your notebooks, SQL editor, and Lakeflow Pipelines editor at this time—no advanced configuration required.

Study Extra

If you need to be taught extra about Genie Code:

• Go to our internet web page to grasp key Genie Code options and use instances and be taught the way it works throughout the Databricks platform
• Watch the demo to see Genie Code plan and execute actual information workflows finish to finish
• Learn the documentation to begin utilizing Genie Code in your personal workspace at this time

We’re excited to see what you construct with Genie Code and the way autonomous brokers will reshape the best way your information groups work in Databricks.