Note : Also candidate should be ready for F2F in Portland OR for final round if asked.
Skills to target: Hands-on
AWS, Airflow, Databricks
Pyspark
SQL
Key Responsibilities:
Collect, organize, and analyse large sets of data from various sources.
Develop and implement data analyses, data collection systems, and other strategies that optimize statistical efficiency and quality.
Interpret data, analyse results using statistical techniques, and provide ongoing reports.
Identify, analyse, and interpret trends or patterns in complex data sets.
Work with management to prioritize business and information needs.
Locate and define new process improvement opportunities.
Work closely with the ETL / DE team to ensure data integrity, quality and security.
Perform data mining and analyse data to identify opportunities for improvement.
Build End to End ETL pipelines on Databricks
Create Automated Workflows, Manage clusters, experience with autoloader
Data modelling experience with Snowflake and Star Schema
Prior ETL orchestration experience in Airflow
Data ingestion techniques using kafka and flume
Worked with processing data from various file types like csv, json, parquet
Worked on End-to-end pipeline construction using AWS services like EMR, EC2, Cloud Watch, Lambda, Redshift
Data Processing techniques in Snowflake
Strong hold on Pyspark Coding as well as Spark optimizationa
Worked on Big Data Services like Apache Hive, Spark, Hadoop, Sqoop
Strong SQL Skills with good experience in SQL tuning
Qualifications:
Proven working experience as a Data/Big Data Engineer Analyst.
Technical expertise regarding data models, database design development, data flow, data mining, and segmentation techniques.
Strong knowledge of and experience with reporting packages (Business Objects etc.), databases (SQL etc.), programming (ETL frameworks or Python).
Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
Adept at queries, report writing, and presenting findings.
Experience with data visualization tools such as Tableau, COGNOS
Excellent communication and presentation skills.
Prior experience of working with Finance data is good to have.