As a Data Engineer , you will play a critical role in designing, building, and maintaining the infrastructure required for optimal data extraction, transformation, and loading (ETL) from a variety of data sources. You will work closely with data analysts, data scientists, and other stakeholders to ensure the smooth and efficient flow of data across our systems, enabling the organization to make data-driven decisions.
Key Responsibilities:
Design, develop, and maintain ETL pipelines: Build robust and scalable data pipelines to collect, transform, and store data from various sources.
Data modeling: Create and optimize data models to support the needs of the business, ensuring data integrity and consistency.
Data storage management: Manage and optimize databases and data warehouses, ensuring data is stored efficiently and securely.
Data integration: Collaborate with cross-functional teams to integrate data from multiple sources, including APIs, databases, and third-party applications.
Performance tuning: Monitor and improve the performance of data systems, including optimizing queries and storage strategies.
Automation: Develop automated processes for data ingestion, transformation, and validation to improve efficiency and reduce manual intervention.
Data quality: Implement and maintain data quality checks and validation rules to ensure data accuracy and reliability.
Documentation: Document data processes, pipelines, and data models to ensure clear understanding and maintainability.
Required Qualifications:
Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).
2-5 years of experience in data engineering, data architecture, or related roles.
Proficiency in programming languages such as Python, Java, or Scala.
Experience with SQL and database management systems (e.g., MySQL, PostgreSQL, Oracle).
Knowledge of ETL tools (e.g., Apache NiFi, Talend, Airflow).
Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka).
Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and related data services.
Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.
Excellent communication skills to work effectively with technical and non-technical stakeholders.
Preferred Qualifications:
Experience with NoSQL databases (e.g., MongoDB, Cassandra).
Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
Knowledge of data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).
Experience with data visualization tools (e.g., Tableau, Power BI) is a plus.
Benefits:
Competitive salary and performance-based bonuses.
Comprehensive health, dental, and vision insurance.
401(k) with company match.
Paid time off and holidays.
Continuous learning and development opportunities.