Custom Incrementalization for Dynamic Tables

Thrive Learning, a UK-based enterprise learning platform, migrated its analytics to Snowflake’s Dynamic Tables to maintain continuous data freshness for customer-facing products like Analyse and Analyse with AI. The team initially used a simple incremental Dynamic Table for its RecordStore pipeline, but complexity from unreliable upstream IDs and multi-source joins forced full refreshes, increasing costs to hundreds of credits daily. To mitigate, they split the pipeline into fast and slow lanes, trading transformation costs for auto-clustering expenses, yet the underlying issue persisted. The root cause—unreliable Content IDs—remained unresolved, leaving the team seeking a scalable solution.

The solution emerged with Snowflake’s custom incrementalization feature, REFRESH_MODE = CUSTOM_INCREMENTAL, which allowed Thrive to define explicit refresh logic using MERGE INTO SELF over CHANGES(). This approach processed only delta changes, reducing refresh latency from hours to seconds while maintaining Snowflake’s managed scheduling and transactional guarantees. The pipeline’s cost structure shifted from scaling with total data size to scaling with the rate of change, eliminating the need to trade transformation costs against clustering expenses. Thrive’s senior data engineer, Rob Howe, deployed the solution in production within weeks, demonstrating rapid adoption of the new capability.

Governance and lineage were critical factors in Thrive’s decision to adopt custom incrementalization. The team manages transformations in dbt, and moving logic to a hand-built streams-and-tasks pipeline would have disrupted lineage tracking and required separate maintenance. Custom incremental Dynamic Tables remained integrated within the dbt DAG through a custom materialization, preserving lineage, tests, and documentation without additional orchestration layers. This alignment with existing workflows ensured consistency across Thrive’s analytics estate.

Thrive is now applying the custom incrementalization pattern to other pipelines facing similar join and deduplication challenges, anticipating comparable reductions in cost and improvements in data freshness. The benefits extend beyond operational efficiency; Thrive’s agentic AI products, which rely on this data, can now operate more cost-effectively and with near-live data, positioning the platform for further AI-driven expansion. The team’s experience underscores the practical impact of custom incrementalization in addressing complex transformation scenarios while maintaining scalability and governance.