Cloud Computing

Project Overview Developed a robust, event-driven ETL pipeline that monitors filesystem events and automatically processes and uploads data to Azure Data Lake Storage Gen2. The system used YAML configuration files for pipeline definition, making it highly configurable and maintainable. Business Context The business needed a flexible solution to continuously monitor specific directories for new data files, process them according to predefined rules, and reliably upload the results to cloud storage. This enabled near real-time data processing without the complexity of a full streaming solution. ...

In this post, I’ll share my experience leading the migration of ETL workflows from legacy systems to Azure Databricks at Zürich Insurance. This project presented unique challenges and opportunities for modernizing our data infrastructure. Project Overview The goal was to migrate existing ETL workflows from legacy systems to Azure Databricks, improving scalability, maintainability, and performance. The migration involved multiple data sources and complex transformations. Key Challenges Legacy System Complexity Complex SQL-based transformations Multiple data source integrations Custom scheduling mechanisms Data Quality Assurance ...