Senior Data Engineer

Experience: 4–6 years  

Company: Invent Health  

Role Description  

The Data Engineer will be responsible for designing and developing data engineering solutions, creating and maintaining data models, building Extract Transform Load (ETL) pipelines, implementing data warehousing, and performing data analytics on large datasets.  

Qualifications  

• Data Engineering and Data Modeling skills  
• Experience with Extract Transform Load (ETL) pipelines  
• Expertise in implementing data warehousing solutions  
• Analytical skills for performing data analytics on large datasets  
• Strong programming skills in languages like Python, Java, or Scala  
• Experience with Big Data and cloud-based technologies like Hadoop, Spark, and AWS  
• Understanding of data governance and security principles  
• Bachelor's degree in Computer Science, Information Systems, or related field  
• Strong communication skills and ability to work independently and collaboratively  

Experience Required: 4-6 years in Data Engineering or related field  

Primary Capabilities:  

SQL Expertise:  

• Proficiency in writing complex SQL queries.  
• Experience with database design and optimization.  
• Ability to perform data manipulation and analysis.  

Databricks Experience:  

• Hands-on experience with Databricks platform.  
• Familiarity with Databricks' integration with various data sources and sinks.  
• Experience in building and optimizing big data pipelines using Databricks.  

PySpark Proficiency:  

• Strong skills in PySpark for large-scale data processing.  
• Understanding of PySpark's architecture and its application in data processing and transformation tasks.  
• Ability to write efficient, scalable, and reusable code in PySpark.  

Python Programming:  

• Strong programming skills in Python.  
• Experience with Python libraries relevant to data engineering (e.g., Pandas, NumPy).  
• Ability to write clean, maintainable, and efficient code in Python.  

AWS Experience:  

• Knowledge of AWS services relevant to data engineering (e.g., S3, Lambda, CodeCommit, etc.,).  
• Knowledge in building and maintaining data infrastructures on AWS.  
• Understanding of cloud computing concepts and how they apply to data engineering tasks.  

Analytics Skills:  

• Ability to derive insights from large datasets.  
• Knowledge of data visualization tools and techniques.  
• Proficiency in statistical analysis and using data to inform decisions.  

Logical Thinking and Problem-Solving:  

• Strong logical reasoning and analytical skills.  
• Proven ability to troubleshoot and solve complex problems.  
• Capacity to break down and simplify complex problems, developing practical, efficient solutions.  

Other Desirable Qualifications:  

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.  
• Good communication skills and the ability to work collaboratively in a team.  
• Experience with version control tools like Git, CodeCommit.  
• Familiarity with Agile methodologies and DevOps practices.  
• Certification in relevant technologies (e.g., Databricks, AWS Certified Data Analytics) is a plus.