- Design, build, and optimize scalable data pipelines using Azure Data Factory and other Azure services to ensure seamless data flow and transformation.
- Develop and manage data storage solutions with Azure Data Lake Storage and Blob Storage, ensuring high availability and data security.
- Collaborate with cross-functional teams to define data architecture and integrate data from diverse sources, including RESTful APIs and third-party platforms.
- Optimize ETL/ELT processes to support data analysis and reporting needs, ensuring high performance and cost-efficiency.
- Implement and maintain data quality processes, including monitoring via Azure Monitor to identify and address pipeline errors or bottlenecks.
- Design and develop Azure solutions for advanced data warehousing and reporting.
- Mentor junior data engineers in Azure best practices, coding standards, and process optimization to foster team growth and productivity.
- Stay updated on emerging trends in Azure technologies and incorporate them into the company's data engineering practices.
Qualifications
- Experience: 5-7 years in data engineering roles, with at least 3 years of hands-on experience in the Azure ecosystem.
- Skills:
- Expertise in Azure Data Factory, Azure Data Lake Storage, and Blob Storage.
- Proficiency in SQL and programming languages like Python, PySpark, or Scala.
- Strong understanding of data integration techniques, including working with REST APIs.
- Knowledge of monitoring and optimization tools like Azure Monitor and Azure Logic Apps.
- Education: Bachelor's degree in computer science or a related field. Master's degree preferred.
- Certifications: Azure-specific certifications like Microsoft Certified: Azure Data Engineer Associate or Azure Solutions Architect Expert are highly valued.
- Soft Skills: Excellent problem-solving abilities, strong communication skills, and a collaborative mindset.