Our client is seeking an experienced Data Engineer with 8-10 years of expertise to design and build scalable data pipelines using PySpark, Snowflake, and AWS. You will modernize legacy ETL workloads into cloud-native solutions while optimizing performance and ensuring data quality across the platform.
Responsibilities & Qualifications
- Design and build scalable, production-grade data pipelines using PySpark for batch and streaming processing, including data transformations, deduplication, and performance optimization (partitioning, caching, shuffle optimization)
- Develop cloud-native data solutions on AWS, leveraging S3, Glue, EMR, Lambda, Step Functions, and Redshift
- Engineer comprehensive Snowflake data platforms including warehouses, schemas, data models, Iceberg Tables, Dynamic Tables, and Materialized Views
- Modernize legacy ETL workloads to cloud-based ELT/ETL patterns and implement orchestration using Airflow or AWS Step Functions
- Optimize Snowflake and PySpark workloads through tuning strategies and performance monitoring to ensure efficiency at scale
- Ensure data quality, governance, and documentation across all data pipelines, including technical designs and runbooks
- Collaborate with data architects, analysts, and stakeholders to align solutions with business requirements
Requirements
- 8-10 years of experience as a Data Engineer with proven expertise in PySpark, Snowflake, and AWS cloud services
- Advanced proficiency in Python and SQL with the ability to write optimized, production-grade code
- Strong experience with AWS services including S3, Glue, EMR, Lambda, Step Functions, and Redshift
- Demonstrated expertise in Snowflake including Iceberg Tables, Dynamic Tables, and Materialized Views
- Solid understanding of data pipeline design, ETL/ELT patterns, data modeling, and data warehousing best practices
- Experience with orchestration tools such as Airflow or AWS Step Functions
- Strong communication skills and ability to work collaboratively with cross-functional technical and business teams
