This role requires a blend of technical expertise in data engineering, full-stack development skills, and a strong understanding of data architectures. The ideal candidate will be able to bridge the gap between traditional SQL environments and modern big data platforms, while also contributing to the full stack of data-driven applications.
Data Engineer Technical Skills Required:
Expert proficiency in Python, C#, and T-SQL
Strong experience with Databricks and Apache Spark (especially Spark SQL)
Proficient in SQL Server and ASP.NET Core
Experience with Apache Kafka
Skilled in query optimization and SQL performance tuning
Familiarity with data warehousing concepts and dimensional modeling
Knowledge of containerization technologies (e.g., Docker, Kubernetes)
Experience with cloud platforms (preferably Azure)
Proficiency in Git and CI/CD pipelines
Familiarity with data governance and security best practices
Understanding of RESTful API design principles
Experience with data visualization tools (e.g., Power BI, Tableau)
Data Engineer Responsibilities:
Design, build, and maintain scalable data jobs and workflows in Databricks using Spark SQL and Python
Migrate and optimize SQL stored procedures from T-SQL to Databricks
Develop and maintain RESTful backend APIs using C# and ASP.NET Core
Implement advanced data transformations, aggregations, and analytical workloads using Spark SQL
Collaborate with data scientists and analysts to deliver optimized data solutions
Implement and maintain data governance practices, ensuring data quality, integrity, and security
Optimize Databricks jobs and SQL queries for performance and cost-effectiveness
Automate data workflows using Python scripting and orchestration tools
Design and implement data pipelines for real-time and batch processing
Contribute to the design and maintenance of data warehouses and data lakes
Participate in code reviews and mentor junior team members
Stay current with emerging data engineering technologies and best practices
Troubleshoot and resolve complex data-related issues
Develop documentation for data processes, APIs, and architectures
Collaborate with cross-functional teams to integrate data solutions into broader applications and services