Backend Engineer (Spark) - Data Infra team

About The Team

Shopee’s Data Infrastructure team builds the company’s stable, efficient, secure, and easy-to-use big data infrastructure and platform for the company. Committed to providing the company's various business teams and data teams, data analysts, machine learning teams, BI teams, etc. with its data-efficient and stable data storage, calculation, query, and analysis of the big data basic system; development and production platform, and data analysis platform. Including data collection, storage, offline calculation of massive data, real-time stream computing, online analysis and processing, instant return query, and other aspects of data infrastructure support, as well as big data development, production scheduling, data quality monitoring, data maps, and other platform services. Provide the upper-level business team and data team with a basic platform for various technical directions such as computing scheduling, batch computing, real-time computing, structured data query, big data analysis, and KV query. Help business teams build data reports, data monitoring dashboards, real-time business data processing, data mining, and analysis, etc.

Job Description

• Design and develop backend services for large-scale data platforms, including data development, scheduling, data assets, and analytics services.
• Participate in Spark kernel-related development to address actual production business needs, ensuring the efficient and stable operation of massive-scale computations.
• Participate in the development of tools and technical solutions within the Spark ecosystem to support business requirements for rapid development, issue identification, and performance tuning of big data tasks.
• Be responsible for maintaining Spark clusters and participate in the planning, management, optimization, and risk mitigation of large-scale clusters.
• Design and promote best practices for Spark usage.
  
  

Requirements

• Bachelor’s Degree in Computer Science, Information Technology, Information Security or related field.
• Minimum 1 year of full time backend experience using Java/Scala.
• Proficient in high-concurrency backend development using Java/Scala.
• Proven experience in big data development using Spark for distributed data processing is preferred.
• Excellent communication and collaboration skills, as well as the ability to drive projects forward.