Job Overview
We are seeking a skilled Data Engineer with strong expertise in the Azure cloud ecosystem to design, build, and maintain scalable data pipelines and cloud-based data solutions. The ideal candidate will work closely with data analysts, data scientists, and business stakeholders to develop efficient data architectures and ensure reliable data delivery across the organization.
Key Responsibilities
- Design, develop, and maintain cloud-based data pipelines and ETL/ELT workflows.
- Build and optimize data architectures for processing both structured and unstructured data.
- Develop and manage Azure Data Factory pipelines, datasets, and data flows.
- Collaborate with data analysts, data scientists, and business teams to understand data requirements and translate them into scalable solutions.
- Implement and enforce data quality, governance, and security best practices.
- Monitor, troubleshoot, and optimize data workflows to ensure high availability and performance.
- Improve database and storage performance while maintaining cost efficiency.
- Support cloud adoption, migration, and modernization initiatives within the organization.
Required Skills
- Strong hands-on experience with the Microsoft Azure cloud platform.
- Expertise in Databricks and Databricks SQL.
- Strong experience building datasets, data flows, and pipelines in Azure Data Factory (ADF).
- Hands-on experience with ETL/ELT tools and data integration frameworks.
- Proficiency in SQL, Python, and data modeling.
- Experience with CI/CD pipelines and Infrastructure-as-Code (IaC) tools.
- Knowledge of data governance, security, and compliance practices.
Preferred Skills
- Experience with API integrations and microservices architecture.
- Exposure to Azure Kubernetes Service (AKS) and ETL processes for applications deployed in containerized environments (AKS/EKS).
- Familiarity with Azure Synapse Analytics and SQL Server SSAS.
- Experience working in Agile environments.
- Strong analytical and problem-solving abilities.
- Azure certifications or prior cloud migration experience are a plus.
Key Technologies
- Azure Data Factory
- Azure Synapse Analytics
- Databricks Platform / Databricks SQL
- SQL Server SSAS
- Azure Kubernetes Service (AKS)
- Python
- Windows Azure Cloud
- Agile Methodology