Job Description

We're looking for a highly skilled and experienced Senior Data Engineer to join our team. You'll be instrumental in designing, building, and maintaining our next-generation Data Ingestion platform and its infrastructure. This role requires a deep understanding of distributed systems and hands-on expertise with cutting-edge big data technologies in a cloud-native environment.

You will be responsible for creating robust, scalable, and fault-tolerant data pipelines that process massive datasets for both real-time and batch-based analytics. Your work will directly enable our data scientists, analysts, and business teams to derive critical insights and make data-driven decisions.

Perform general application development activities, including unit testing, code deployment to development environment and technical documentation. Works on one or more projects, making contributions to unfamiliar code written by team members.

What You’ll Do

Apply the knowledge of data characteristics and data supply pattern, develop rules and tracking process to support data quality model.
Prepare data for analytical use by building data pipelines to gather data from multiple sources and systems.
Integrate, consolidate, cleanse and structure data for use by our clients in our solutions.
Perform design, creation, and interpretation of large and highly complex datasets.
Stay up-to-date with the latest trends and advancements in GCP and related technologies, actively proposing and evaluating new solutions.
Understand best practices for data management, maintenance, reporting and security and use that knowledge to implement improvements in our solutions. Implement security best practices in pipelines and infrastructure.
Develop and implement data quality checks and troubleshoot data anomalies.
Provide guidance and mentorship to junior data engineers.
Review dataset implementations performed by junior data engineers.
Perform general application development activities, including unit testing, code deployment to development environment and technical documentation.
Work on one or more projects, making contributions to unfamiliar code written by team members.

What Experience You Need

BS degree in a STEM major or equivalent discipline; Master’s Degree strongly preferred
3+ years of experience as a data engineer or related role
3+ years of software engineering experience
Advanced skills using programming languages such as Python or Java
SQL and intermediate level experience with scripting languages
3+ years experience with Cloud technology: GCP, AWS, or Azure
3+ years experience designing and developing cloud-native solutions
Experience building and maintaining moderately-complex data pipelines, troubleshooting issues, transforming and entering data into a data pipeline in order for the content to be digested and usable for future projects
Demonstrates advanced Git usage and CI/CD integration skills

What could set you apart

3+ years of hands-on experience in data engineering or a related field.
Expert-level proficiency in SQL, Python or relevant programming languages
Demonstrated experience designing and implementing distributed data processing solutions at scale.
In-depth, hands-on experience with BigData tools such as Apache Spark or Apache Beam.
Proven experience with Google Cloud Platform (GCP) services for data engineering, including Dataflow, Dataproc, BigQuery, and Pub/Sub .
Experience with workflow orchestration tools like Apache Airflow (or GCP's Cloud Composer).
Solid understanding of data modeling, data warehousing, and ETL/ELT best practices.
Background of Large Data Analysis and building AI ML Models would be a huge plus.
Excellent problem-solving skills and the ability to troubleshoot complex distributed systems.
Effective communication and collaboration skills, with the ability to articulate technical concepts to both technical and non-technical stakeholders