The Data Engineer / Azure Specialist plays a critical role in our IT team, focusing on designing, implementing, and optimizing ETL pipelines to manage and transform raw hospital insurance data into database tables leveraged by our in-house applications. This position requires a combination of technical expertise in data engineering, Azure Data Factory, and a background in healthcare or insurance. The ideal candidate will demonstrate high intensity, attention to detail, and urgency in their work.
Key Responsibilities
ETL Pipeline Development: Design, implement, and optimize ETL pipelines, often using Azure Data Factory, to process large volumes of hospital insurance data.
Data Integration: Integrate raw data files containing millions of rows into our database environment, ensuring data consistency and integrity.
Data Transformation: Develop and maintain processes to clean, transform, and load data for analysis and reporting.
Performance Optimization: Regularly monitor and optimize ETL pipeline performance for efficiency and reliability.
Collaboration: Work closely with the Database Admin and development teams to ensure smooth data operations and integration.
Documentation: Maintain and create detailed documentation of ETL processes, data flows, and system architecture.
Data Security: Implement and uphold data security measures to protect sensitive healthcare data.
Urgency and Attention to Detail: Approach tasks with a high sense of urgency and meticulous attention to detail, ensuring the accuracy and reliability of data processing.
Essential Skills and Qualifications
Excellent collaboration skills.
Experience with Azure Data Factory and Azure SQL Database.
Strong background in data engineering and database management.
Python scripting abilities for data manipulation and automation.
Experience in handling large datasets, especially in healthcare or insurance.
Solid understanding of SQL and database optimization techniques.
Strong focus on data security and compliance (HIPAA).
Proven ability to work under pressure with high attention to detail.
Experience in ETL pipeline development and performance tuning.
Knowledge of data backup and recovery procedures.
Relevant Microsoft Certifications (e.g., DP-300: Administering Relational Databases on Microsoft Azure) are a huge plus - this will automatically advance your application through the first stage of our consideration process.