Data Engineer

Apps Associates

View Salaries, Reviews, and more

Job Description

Skills

JOB DESCRIPTION

Strong background in data engineering or cloud data platform development with proven expertise in AWS Glue and Snowflake
Strong proficiency in Python and PySpark, with hands-on experience building, tuning, and orchestrating large-scale ETL pipelines
Solid grasp of data modeling, data quality, and governance principles supporting finance or payment transaction data
Demonstrated analytical/problem solving skills
Experience working for a global tea
Ability to work collaboratively in a team environment
Ability to work effectively with people at all levels in an organization

Responsibilities

Design, develop, and maintain robust and scalable data pipelines using AWS Glue for ETL/ELT processes, data transformation, and data integration from various sources (e.g., S3, relational databases, APIs)
Develop and manage AWS Glue jobs (Python/PySpark) for data ingestion, transformation, and loading into Snowflake.
Utilize AWS Glue Data Catalog for metadata management and integration with other AWS services
Implement and optimize data warehousing solutions in Snowflake, including schema design, table creation, data loading, performance tuning, and query optimization
Ensure data quality, consistency, and accuracy throughout the data lifecycle, implementing data validation and cleansing routines
Collaborate with product managers and engineering teams to understand data requirements and deliver solutions that meet business needs
Monitor and troubleshoot data pipelines and Snowflake performance, identifying and resolving issues proactively

Implement best practices for data governance, security, and compliance within AWS and Snowflake environments