Agentic data science with the Data Agent Kit
Google’s Data Agent Kit integrates with IDEs to automate end-to-end data workflows, from ingesting raw logs to deploying fraud-detection pipelines in Airflow.
The Data Agent Kit, available as a plugin for coding agents or an extension for Visual Studio Code-based IDEs, enables data scientists to build automated workflows directly within their development environment. It provides tools to connect agents to Google Cloud data services and offers structured instructions for common tasks like querying databases or deploying pipelines. The extension streamlines interactions with Google’s Agentic Data Cloud, reducing the need to switch between tools during complex data projects.
To address a compliance team’s fraud investigation workflow, the kit automates the process of loading raw transaction logs from Google Cloud Storage into BigQuery using a generated Jupyter notebook. The notebook runs on a preconfigured Spark Serverless runtime, and the Data Agent Kit extension allows users to inspect the resulting table and its metadata directly in the IDE, streamlining data preparation steps.
After loading the data, the agent generates a dbt project with SQL models to clean and enrich the transaction records, including deduplication and joins with sender and receiver information. Running dbt build creates staging and enriched tables in BigQuery, along with a table for records missing transaction IDs, while also executing a uniqueness test to ensure data integrity before model training.
The final step automates the fraud-detection pipeline using a Random Forest classifier trained on labeled data. The agent creates a notebook to score new transactions, flagging those with a fraud probability of 50% or higher for review in Spanner. The workflow is then scheduled to run daily via Managed Service for Apache Airflow, packaging notebooks and dbt projects into an Airflow DAG deployed directly from the IDE.