Data Engineer

On behalf of our public sector client, Affinity is looking for a Data Engineer to be part of the Contact Centre Renewal Project. The Data Engineer / Tester will develop and test data pipelines according to the standards and practices in Information Management at the organization.

Responsibilities:

• Analyze the source data, prepare data model & mapping, and develop Scala/Spark programs and related components in the work areas of Information Management ("IM") relating to claims, insurance and finance data.

• Use data pipelines to extract data from data sources which are in various formats (viz - flat files, XML, relational tables, Oracle logs), and Use tools (such as Stream Sets, Scala and Spark programs) to transform and store the data in big data platform (Data Lake) after data validation.

• Ingest data from cloud to Big Data.

Deliverables:

• Develop Spark and Scala pipelines for data ingestion and Data transformation

as per the mapping document from Cloud to Big Data

• Data Analysis as per requirements and develop data models/mapping

• Data validations and automated data quality pipelines

• Relevant documentation for sustainment

Qualifications:

• Applied knowledge in Big Data platforms, ideally with exposure to Hadoop ecosystem (HDFS, Pig, Hive, SPARK, Big SQL, NoSQL, YARN); Solid experience in Scala

• Experience in designing efficient and robust data pipelines using Spark.

• Excellent skill in SQL development.

• Knowledge of data modeling and understanding of different data structures and their benefits and limitations under particular use cases.

• Hands-on experience with structured and unstructured data like JSON.

• Experience with enterprise systems such as Guidewire Claim Center, Guidewire Policy Center would be an asset.

• Hands-on experience Cloud, data ingestion using REST API.

• Knowledge on Contact Centre business.

Affinity Earn:

Know someone who’s great for this, or any of our open roles? Earn up to $4,000/year for each successful referral through Affinity Earn. You can also earn up to $50,000 for helping us find new clients. Learn about our referral program at https://affinity-group.ca/earn/ or browse our jobs & follow us at https://www.linkedin.com/company/affinity-staffing/jobs/

About Affinity:

Affinity Group is a technology and business consulting and services company. We believe in creating long term relationships between clients and consultants that foster a mutually beneficial partnership. Affinity is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided on the basis of qualifications, merit and business need.

For more information on Affinity, please visit www.affinity-group.ca

Job Number: 12332