Azure

Python Watchdog YAML-Based ETL Pipeline for Azure Data Lake Project Overview Developed a robust, event-driven ETL pipeline that monitors filesystem events and automatically processes and uploads data to Azure Data Lake Storage Gen2. The system used YAML configuration files for pipeline definition, making it highly configurable and maintainable. Business Context The business needed a flexible solution to continuously monitor specific directories for new data files, process them according to predefined rules, and reliably upload the results to cloud storage. This enabled near real-time data processing without the complexity of a full streaming solution. ...

Led the development of a cloud-based analytics platform at A1 Telekom Austria, enhancing data processing and decision-making capabilities.

Migrating ETL Workflows to Azure Databricks: A Case Study In this post, I’ll share my experience leading the migration of ETL workflows from legacy systems to Azure Databricks at Zürich Insurance. This project presented unique challenges and opportunities for modernizing our data infrastructure. Project Overview The goal was to migrate existing ETL workflows from legacy systems to Azure Databricks, improving scalability, maintainability, and performance. The migration involved multiple data sources and complex transformations. ...