Looking for senior level Data Engineers with a proven track record of creating highquality software products in AML Quantexa workstream You will be managing transforming cleansing and Resolving Entities of high volume data. You'll be working closely with client Technical Leads and Principal Data Engineer with everyone following the same goal of meeting our project expectations and delivering a firstclass service. You'll be using open source bigdata technology and tools such as Spark Hadoop Scala Data Fusion Entity Resolution with our platform being hosted on Red Hat machines and Cloudera Distribution Projects primary language is written in Scala and Spark so you should be comfortable and confident with using our platform Being Agile is an integral part to the success we have and having regular team sprints and Scrum meetings with the Project team is essential
Responsibilities
Demonstrating deep technical knowledge and expertise in software development including programming languages frameworks and best practices Providing guidance and mentorship to junior team members
Actively contributes to the implementation of critical features and complex technical solutions Write clean efficient and maintainable code that meets the highest standards of quality
Write defensive fault tolerant and efficient code for data processing
Provide guidance on scalable robust and efficient solutions that align with business requirements and industry best practices
Use emerging and opensource technologies such as Spark Hadoop and Scala
Collaborate on scalability issues involving access to massive amounts of data and information
Take on adhoc tasks as required
Required Qualifications
8 years experience of implementing data intensive solutions using agile methodologies
Proficient in programming languages commonly used in data engineering such as Scala Spark Python and Java
Experience of building data processing pipelines for use in production hands off batch systems including either or preferably both traditional ETL pipelines andor analytics pipelines
Strong coding experience in the likes of Scala Java or Python
Enthusiasm to learn and develop emerging technologies and techniques
Preferred Qualifications
Quantexa Certified Data Engineer
Familiarity with opensource data engineering tools and frameworks eg Spark Kafka Trino is a valuable asset
Exposure to a range of table and file formats including Hive Parquet and JSON
Experience in cloud native technologies and patterns AWS Google Cloud
Experience of driving andor influencing the data strategy of your team or organization