Kafka, Spark, Scala, EMR and EKS are used to build the data pipelines for the data lake in the AWS environment.
Experience in Python scripting is highly preferred.
Experience in IAM, SAML, MFA are nice to haves. Additionally, experience in Fintech is nice to have but not required.
Must Have Qualifications: Experienced Lead with 10+ years of experience, having knowledge of data engineering and pipelines in AWS environment, knowledge of Kafka, EMR, EKS, Scala, Spark.
Qualifications:
Bachelor's degree in information technology, engineering, computer science, related field, or equivalent experience. Advanced studies/degree is preferred.
10+ years of overall information technology experience with an emphasis on integration and delivery of virtual/cloud platforms to enterprise applications, recent public cloud experience preferred.
At least three years proven experience with AWS cloud formation, Terraform, or similar tools.
5+ years of related industry experience in an enterprise environment.
3+ years of experience with engineering and supporting containerization technology (Kubernetes, AWS(ECS/EKS) etc.) at scale.
Experience with Python, Ansible and shell scripting to automate routine operation tasks.
Experience in identity and access data correlation, normalization and building of cohesive identity and access data models for large enterprises.
Single Sign On (SSO) integration and session management for multiple web and cloud applications.
Identity Federation (SAML) configuration and integration across multiple trusted third parties, applications, and systems.
Multi Factor Authentication (MFA) such as Duo or Microsoft Authenticator security integration into the authentication, authorization, and single sing on process for applications and systems.
API security and API integration with IAM systems for sharing identity contexts.
Experience in Jenkins, Git and Artifactory.
Formal design patterns and industry best-practices.
Scripting languages (e.g., Groovy / Perl / Python / Ruby).
2+ years of experience across a variety of technologies such databases, directory services, application servers, network infrastructures, Linux operating systems, and an understanding of fundamental security and data flows within these components.
Scaling CI/CD systems and microservices.
Code coverage analysis / static analysis tools.
Agile programming processes and methodologies such as Scrum.
Overseeing / managing production application deployment.
Excellent verbal and written communication skills.
Self-motivated, driven, and creative individual.
Ability to provide 24x7 operational support on periodic basis and triage complex issues to restore availability.
Documentation of Engineering Solutions and Designs.