We are seeking a skilled and experienced Data Engineer with strong Python programming capabilities to join our growing data team. The successful candidate will be responsible for building, maintaining, and optimizing data pipelines, ensuring the availability, reliability, and scalability of our data infrastructure. The ideal candidate will have a solid understanding of database systems, data integration techniques, and modern data processing tools, along with a focus on automation and efficiency.
Key Responsibilities:
Design, build, and maintain scalable and efficient data pipelines using Python and other relevant technologies.
Develop ETL processes to collect, clean, and transform data from various sources (e.g., APIs, databases, flat files) into usable formats.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement appropriate solutions.
Monitor and optimize data pipelines to ensure performance, reliability, and scalability.
Integrate data from various structured and unstructured data sources into a centralized data warehouse or data lake.
Maintain accurate documentation of data engineering processes, data flows, and architecture.
Implement and enforce best practices for data governance, security, and privacy.
Collaborate with DevOps and other engineering teams to ensure smooth data pipeline operations in production environments.
Troubleshoot data-related issues and provide ongoing support to business users and teams.
Key Qualifications:
Bachelor's degree in Computer Science, Information Technology, or related field. Master's degree is a plus.
Proven experience (3+ years) as a Data Engineer or in a similar role.
Strong proficiency in Python, with experience building and optimizing data pipelines.
Experience with ETL tools and frameworks such as Airflow, Luigi, or Apache Nifi.
Hands-on experience with SQL databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
Familiarity with cloud-based data platforms (e.g., AWS, Azure, GCP) and cloud services like AWS Lambda, Azure Functions, or GCP Cloud Functions.
Experience working with big data technologies such as Apache Spark, Hadoop, or Kafka.
Strong understanding of data modeling, schema design, and data warehousing concepts.
Experience with containerization (e.g., Docker) and orchestration tools (e.g., Kubernetes) is a plus.
Strong problem-solving skills and attention to detail.
Excellent communication and teamwork skills.
Preferred Skills:
Experience with machine learning and data science workflows is a plus.
Familiarity with data visualization tools (e.g., Power BI, Tableau).
Knowledge of data governance, security, and privacy regulations (e.g., GDPR, HIPAA).
Experience with version control systems (e.g., Git).