The Senior Data Engineer will work with our data management team. This role focuses on leveraging advanced technologies such as Databricks, Delta Lake, FiveTran, and Google Cloud Platform (GCP) to enhance our data infrastructure and drive innovative data solutions. Utilizing experience with Databricks administration, Unity Catalog, and JAR file management to optimize our data processes and support advanced analytics initiatives.
Essential Functions and Duties
Design and implement robust data pipelines using Databricks and integrate these with Delta Lake for efficient data storage and management.
Manage end-to-end data workflows from ingestion to insights, utilizing FiveTran for seamless data integration and GCP for scalable cloud solutions.
Administer Databricks environments, ensuring optimal configuration, security, and performance across all data operations.
Utilize Unity Catalog to manage data governance across all Databricks workspaces, ensuring compliance with data privacy and security policies.
Develop and maintain scalable and efficient data models and architecture, supporting the strategic goals of the organization.
Collaborate with data scientists and analysts to deploy machine learning models and complex analytical projects.
Monitor, troubleshoot, and optimize data systems, proposing and implementing improvements to enhance performance.
Document all data procedures and systems, ensuring clear and accurate records are maintained for compliance and operational efficiency.
Stay updated with the latest innovations in data engineering and introduce new tools and techniques to keep the data infrastructure at the forefront
Always represent the Company in a professional manner and appearance.
Understand and internalize the Company's purpose,
Display loyalty to the Company and its organizational values.
Display enthusiasm and dedication to learning how to be more effective on the job and share knowledge with others.
Work effectively with co-workers, internal and external customers and others by sharing ideas in a constructive and positive manner; listen to and objectively consider ideas and suggestions from others; keep commitments; keep others informed of work progress, timetables, and issues; address problems and issues constructively to find mutually acceptable and practical business solutions; address others by name, title, or other respectful identifier, and respect the diversity of our work force in actions, words, and deeds.
Comply with the policies and procedures stated in the Injury and Illness Prevention Program by always working in a safe manner and immediately reporting any injury, safety hazard, or program violation.
Ensure conduct is consistent with all Compliance Program Policies and procedures when engaging in any activity on behalf of the company. Immediately report any concerns or violations.
Other duties as assigned.
Qualifications
Education, Knowledge, Skills, and Experience
Required Education:
Bachelor's degree in computer science, engineering, information technology or related field or four (4) years of relevant experience in lieu of degree.
Master's degree in related field a plus.
Required Knowledge:
General knowledge of advanced concepts, practices and procedures related to design and implementation of robust data pipelines
Proven expertise in Databricks administration and managing large-scale data environments.
Strong experience with SQL, Python, and other scripting languages commonly used in data engineering.
Familiarity with Fivetran or similar data integration tools and understanding of ETL processes
Previous experience managing end-to-end data workflows
Required Experience:
Minimum of five (5) years of experience [nine (9) years for non-degreed candidates] in a data engineering role with significant exposure to Databricks, Delta Lake, and cloud platforms like GCP.
Preferred Experience:
Experience with Unity Catalog and managing data access and security within a Databricks environment.
Knowledge of Java Archive (JAR) files and their applications in data projects.
Required Skills:
Must have strong organizational skills
Must have a detail orientation and the proven ability to prioritize work
Must have effective verbal and written communication skills
Must have the ability to work with limited supervision and as part of a team
Sound decision-making abilities
Excellent problem-solving skills and the ability to work in a dynamic, fast-paced environment.
Strong communication and collaboration skills, capable of working effectively across multiple teams.
Preferred Skills:
Experience with machine learning and AI workflows.
Familiarity with Apache workflows, such as Apache Airflow or Apache NiFi, showcasing the ability to design, implement, and manage complex data pipelines and workflows.
Knowledge of data visualization and reporting tools (e.g., Power BI, Tableau).
Professional Certification:
Certifications in data engineering or a related field a plus
Physical requirements
Vision, hearing, speech, movements requiring the use of wrists, hands and/or fingers. Must have the ability to view a computer screen for prolonged periods and the ability to sit for extended periods. Must have the ability to work the hours and days required to complete the essential functions of the position, as scheduled. The employee occasionally lifts to 20 lbs. and occasionally kneels and bends. Must have the ability to travel occasionally. Working condition include normal office setting.
Mental Demands
Learning, thinking, concentration and the ability to work under pressure, particularly during busy times. Must be able to pay close attention to detail and be able to work as a member of a team to ensure excellent customer service. Must have the ability to interact effectively with co-workers and customers, and exercise self-control and diplomacy in customer and employee relations' situations. Must have the ability to exercise discretion as well as appropriate judgments when necessary. Must be proactive in finding solutions.
Direct Reports
No
EEO/AAP Statement
FFF Enterprises/Nufactor is an equal opportunity employer to all and prohibits discrimination and harassment based on the following characteristics: race, color, caste, religion, religious creed (including religious dress and grooming practices), national origin, ancestry, citizenship, physical or mental disability, medical condition (including cancer and genetic conditions), genetic information, marital status, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), gender, gender identity, gender expression, age (40 years and over), sexual orientation, veteran or military status, medical leave or other types of protected leave (requesting or approved for leave under the Family and Medical Leave Act or any state protected leaves), domestic violence victim status, political affiliation, reproductive health decision-making, and any other characteristic protected by state or federal anti-discrimination law covering employment. These categories are defined according to Government Code section 12920. The Company prohibits unlawful discrimination based on the perception that anyone has any of those characteristics or is associated with a person who has or is perceived as having any of those characteristics.