Why you should be excited: Working on the engineering team at EcoMap is a blend of product development to serve our end users, challenging data modeling and orchestration, and complex data analysis and warehousing problems. There are projects centered around modernizing old pieces of infrastructure as well as green field projects where you can stretch your legs building the infrastructure which will be central to our platform moving forward.
As a senior Data Engineer, your expertise will be pivotal in developing and optimizing scalable engineering solutions for EcoMap's core products. You will work closely with a team of cross-platform engineers, boasting collective expertise in software engineering, machine learning, Conversational AI, and more, to define and shape the future of the EcoMap data infrastructure and pipelines. You will also partner with product managers, data scientists, and operations teams to leverage engineering innovations to simplify business requirements into scalable solutions.
Key Responsibilities
Create and maintain data pipeline architecture,
Assemble large, complex data sets that meet functional / non-functional business requirements.
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
Ensure secure handling and storage of data assets in line with industry standards
Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
Requirements
Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
Experience building and optimizing 'big data' data pipelines, architectures and data sets.
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
Strong analytic skills related to working with unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
A successful history of manipulating, processing and extracting value from large disconnected datasets.
Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.
Strong project management and organizational skills.
Experience supporting and working with cross-functional teams in a dynamic environment.
Skills and Qualifications:
We are looking for a candidate with 3+ years of experience in a Data Engineer role, who has attained a degree in Computer Science, Statistics, Informatics, Information Systems or equivalent work experience. They should also have experience using the following software/tools:
Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
Experience leveraging cloud services to manage infrastructure and workloads (AWS, Google Cloud, Azure, etc.)
Experience with big data tools: Hadoop, Spark, Kafka, etc.
Experience with stream-processing systems: Storm, Spark-Streaming, etc.
Bonus Skills:
Familiarity with advanced distributed web scraping techniques
Experience leading data science/analytics projects
Experience with or interest in machine learning
Background in coaching and mentorship
Benefits
Competitive compensation.
Unlimited PTO with flexible work & travel policies.