Job Summary: The Data Engineer will be responsible for designing, developing, and maintaining data pipelines and ETL processes, ensuring the smooth integration of data from various sources into Snowflake. This role will focus on using Matillion for data transformation, working with Snowflake for cloud data warehousing, and leveraging Microsoft Azure tools for data orchestration and automation.
Key Responsibilities:
ETL Development & Data Pipeline Management:
Design, develop, and optimize ETL processes using Matillion to integrate, transform, and load data into Snowflake data warehouse.
Collaborate with data architects and other teams to gather requirements and build robust, scalable data pipelines.
Troubleshoot and optimize existing ETL workflows to enhance performance and minimize bottlenecks.
Cloud Data Engineering:
Work with Snowflake to design, implement, and manage data models, schemas, and tables.
Ensure data quality, integrity, and consistency across various platforms.
Manage the lifecycle of data stored in Snowflake and ensure proper governance, access control, and security protocols are followed.
Integration with Azure:
Leverage Azure Data Factory and other Azure services (e.g., Azure Databricks, Azure Data Lake) for orchestration and automation of data workflows.
Manage and monitor the data pipeline performance, ensuring timely data movement across cloud environments.
Utilize Azure Blob Storage or Azure Data Lake Storage for data storage and processing needs.
Collaboration & Documentation:
Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data needs.
Document processes, architectures, and pipelines to ensure clear knowledge sharing within the team.
Data Optimization & Performance Tuning:
Optimize Snowflake queries and data models to ensure high-performance analytics.
Implement best practices for query performance, data compression, and cost optimization in Snowflake.
Required Skills and Qualifications:
Experience:
Proven experience as a Data Engineer, Data Analyst, or similar role, with hands-on experience working with Matillion, Snowflake, and Azure technologies.
Strong experience in designing and managing ETL pipelines using Matillion.
Solid experience working with Snowflake data warehousing solutions (data modeling, SQL, query optimization).
Expertise in Azure cloud services (Azure Data Factory, Azure Blob Storage, Azure Databricks, etc.).
Experience in cloud-based data integration and transformation workflows.
Technical Skills:
Strong proficiency in SQL for data querying, optimization, and debugging.
Familiarity with scripting languages such as Python or Bash for automation tasks.
Knowledge of Data Warehousing, Data Lakes, and Data Modeling concepts.
Understanding of cloud architecture and design patterns.
Soft Skills:
Strong analytical and problem-solving skills.
Excellent communication skills, with the ability to articulate technical concepts to non-technical stakeholders.
Ability to work independently as well as part of a collaborative team.
Desired Qualifications:
Experience in other Azure services such as Azure Synapse Analytics.
Knowledge of data governance frameworks and security best practices.
Familiarity with version control systems like Git.
Certification in Matillion, Snowflake, or Azure is a plus.