Migrating ETL Workflows to Azure Databricks
In this post, I’ll share my experience leading the migration of ETL workflows from legacy systems to Azure Databricks at Zürich Insurance. This project presented unique challenges and opportunities for modernizing our data infrastructure. Project Overview The goal was to migrate existing ETL workflows from legacy systems to Azure Databricks, improving scalability, maintainability, and performance. The migration involved multiple data sources and complex transformations. Key Challenges Legacy System Complexity Complex SQL-based transformations Multiple data source integrations Custom scheduling mechanisms Data Quality Assurance ...