How to Use Python UDF Logging for Pipeline Observability

BigQuery now supports Python’s standard logging library inside User-Defined Functions, allowing developers to emit structured logs directly into Cloud Logging without additional plugins or connectors. This feature enables row-level diagnostics during query execution, helping teams isolate problematic records, trigger alerts, and triage issues without pausing workloads.

In an e-commerce event ingestion pipeline, the Python UDF parses JSON payloads, routes valid records to the destination table, and logs errors for corrupted or non-compliant rows using logging.error(). Cloud Monitoring then triggers alerts when error rates exceed thresholds, while engineers use BigQuery’s _AllLogs view to diagnose and recover quarantined data via SQL.

The demonstration shows how to configure log-based metrics in Cloud Monitoring and set notification channels such as Slack, PagerDuty, or email to alert on-call teams when malformed payloads surpass acceptable limits. Users must enable BigQuery Log Analytics on their Cloud Logging bucket and create a linked dataset to query logs via SQL.

Additional use cases include shadow execution for safe migrations, PII redaction auditing, ML inference drift detection, and selective debug output via session properties. Cost management is advised for high-throughput pipelines by logging selectively or sampling to control ingestion volume and avoid exposing sensitive data in logs.