We are seeking an experienced AWS Glue Specialist to join our team and lead the migration of ETL processes from IBM DataStage to AWS Glue. This role requires hands-on expertise in designing, developing, and optimizing ETL pipelines, with a focus on migrating data from various sources, both on-premises and cloud-based. The ideal candidate will have a proven ability to work independently and collaborate with multi-domain teams to ensure successful data integration and transformation.
Key Responsibilities
Lead the migration of ETL processes from IBM DataStage to AWS Glue, ensuring efficiency and reliability.
Design, develop, and optimize ETL pipelines using AWS Glue, incorporating best practices for performance, scalability, and cost-efficiency.
Analyze existing DataStage ETL workflows to identify and implement optimized solutions in the AWS environment.
Extract data from diverse sources (on-premises and cloud) and transform it into a cloud-compatible format using AWS Glue.
Collaborate with cross-functional teams to understand data requirements and ensure seamless data integration across multiple domains.
Independently troubleshoot and resolve technical challenges, ensuring data integrity and compliance with project timelines.
Document technical specifications, data flows, and transformation logic for ETL processes.
Monitor and maintain ETL pipelines to ensure continuous operation, making enhancements as needed.
Support the team with best practices in cloud data management and AWS Glue usage.
Qualifications
5+ years of experience in ETL development, with a focus on data migration.
3+ years of hands-on experience with AWS Glue, including job creation, workflow orchestration, and optimization.
Proven expertise in IBM DataStage and experience in migrating ETL jobs to cloud platforms, particularly AWS.
Strong knowledge of AWS services (S3, Lambda, CloudWatch, Redshift, RDS, etc.) and how they integrate with AWS Glue.
Experience with Python or PySpark for ETL scripting and transformations.
Familiarity with SQL and relational databases, as well as NoSQL databases.
Strong problem-solving skills with the ability to work independently and collaborate across multiple teams.
Excellent communication skills for interacting with stakeholders and documenting processes.
Preferred Skills
Experience with AWS Data Migration Services (DMS) or other data migration tools.
Knowledge of data warehousing concepts and best practices.
AWS certifications (e.g., AWS Certified Data Analytics, AWS Certified Solutions Architect) are a plus.
Familiarity with Agile methodologies and experience working in a fast-paced environment.