Location: Hybrid (2 times a quarter in Reston, VA)
Purpose:
The Senior Data Engineer is responsible for orchestrating, deploying, maintaining, and scaling cloud or on-premises infrastructure targeting big data and platform data management. This role emphasizes reliability, automation, and performance, focusing on developing solutions that transform the company's platforms to deliver data-driven insights and value.
Essential Functions:
Data Requirement Gathering: Collaborate with Business Analysts and the Product team to understand and gather data requirements.
Data Pipeline Development: Design and build Ab Initio data graphs and data pipelines to extract data from various databases, flat files, and message queues.
Data Transformation: Transform the extracted data to create a consumable data layer for various applications.
Support and Maintenance: Provide ongoing support for data pipelines, including bug fixes and enhancements.
Documentation: Document technical designs, operational runbooks, and other relevant materials.
Qualifications:
Education:
Bachelors Degree in Computer Science, Information Technology, Engineering, or a related field.
Experience:
10+ years of IT experience, predominantly in Data Integration and Data Warehouse.
Minimum of 5 years of ETL design and development experience using Ab Initio.
1-2 years of data integration project experience on the Hadoop platform, preferably Cloudera.
Familiarity with Ab Initio CDC (Change Data Capture) in an ETL project setting is a plus.
Working knowledge of HDFS, Hive, Impala, and related Hadoop technologies.
Experience with AWS services is beneficial, particularly with Glue as the company transitions to the cloud.
Strong SQL skills with the ability to write well-performing SQL queries.
Good understanding of OLTP and OLAP data models and data warehouse fundamentals.
Rigor in maintaining high code quality, automated testing, and adhering to engineering best practices.
Ability to unit test thoroughly and troubleshoot issues in production environments.
Experience with Unix/Linux shell scripting is required.
Ability to work independently while supporting junior developers as needed.
Some Java development experience is a plus.
Knowledge of Agile development practices is required.
Key Attributes:
Proven leadership experience in managing projects at this level.
Hands-on experience with Ab Initio, particularly in developing continuous flows for real-time processing.
Experience replicating data from mainframe systems, processing it through Ab Initio continuous flows, and publishing to MongoDB.
Strong SQL proficiency and ability to guide team members effectively