Technical Lead/Senior Data Engineer
J&M Group
Date: 22 hours ago
City: Mississauga, ON
Contract type: Contractor
Responsibilities
Design, develop, and maintain scalable data pipelines using Python, PySpark, and Airflow
Architect and manage data infrastructure on AWS (S3, EC2, EMR, Redshift, Glue, Lambda)
Use CDK to implement infrastructure as code
Optimize database schemas and queries using SQL
Ensure data quality through comprehensive unit testing strategies
Monitor, troubleshoot, and enhance pipeline performance
Collaborate with data scientists, analysts, and engineering teams to meet data requirements
Maintain documentation for pipelines, infrastructure, and workflows
Implement monitoring and alerting systems
Contribute to DevOps practices, including CI/CD pipelines, automated deployments, and containerization (Docker)
Qualifications
610 years of experience in data engineering or related fields
Strong proficiency in Python and SQL
Expertise with PySpark and distributed data processing
Solid experience with Airflow for pipeline orchestration
In-depth knowledge of AWS services for data engineering
Experience with CDK (Cloud Development Kit) for infrastructure provisioning
Familiarity with relational and NoSQL databases
Strong problem-solving and communication skills
Exposure to CI/CD, DevOps, and containerization (e.g., Docker)
Design, develop, and maintain scalable data pipelines using Python, PySpark, and Airflow
Architect and manage data infrastructure on AWS (S3, EC2, EMR, Redshift, Glue, Lambda)
Use CDK to implement infrastructure as code
Optimize database schemas and queries using SQL
Ensure data quality through comprehensive unit testing strategies
Monitor, troubleshoot, and enhance pipeline performance
Collaborate with data scientists, analysts, and engineering teams to meet data requirements
Maintain documentation for pipelines, infrastructure, and workflows
Implement monitoring and alerting systems
Contribute to DevOps practices, including CI/CD pipelines, automated deployments, and containerization (Docker)
Qualifications
610 years of experience in data engineering or related fields
Strong proficiency in Python and SQL
Expertise with PySpark and distributed data processing
Solid experience with Airflow for pipeline orchestration
In-depth knowledge of AWS services for data engineering
Experience with CDK (Cloud Development Kit) for infrastructure provisioning
Familiarity with relational and NoSQL databases
Strong problem-solving and communication skills
Exposure to CI/CD, DevOps, and containerization (e.g., Docker)
How to apply
To apply for this job you need to authorize on our website. If you don't have an account yet, please register.
Post a resume