CTC007780 - Senior Data Engineer Team Lead

Secteur industriel: Information Technology

Type d'emploi: Contract

Durée: Nine months

Mode de travail: Remote

Description

Remote work. Activities may be performed, remotely (within Canada), however on-site attendance may be required as requested by the Manager, Data Engineering, A/Director, Data Governance and Engineering or the Executive Director, Digital Health.

Main tasks for the Data Engineer - Team Lead are outlined as follows:

• Provides strategic oversight of data engineering activities, shaping the overall platform strategy and data architecture.

• Acts as a technical leader for the team setting best practices for other developers to follow

• Collaborates with senior leadership and other external stakeholders by acting as the senior technical resource for the group

• Monitor and optimize ARC platform data pipelines to ensure efficient data flow and system performance, applying best practices in data engineering for ongoing maintenance.

• Collaborate with stakeholders and data SMEs to identify and implement enhancements to ARC platform capabilities, ensuring it meets evolving needs.

• Conduct advanced data profiling and quality assessments to monitor the health of the data within the platform, using tools and techniques for ongoing quality assurance.

• Implement and manage data access policies in coordination with the Data Governance and Engineering team, ensuring secure and compliant access to data for new features and capabilities.

• Troubleshoot and resolve data-related issues in collaboration with operational support teams.

• Work with source systems to identify data for transport as required.

1. Profile source datasets.

2. Retrieve data using different methodologies such as SQL, NoSQL, and RESTful.

3. Prepare documentation containing high-level descriptions of data.

4. Optimize transfer by defining and using change data capture techniques.

• Apply and/or update classifications and levels of sensitivity to all personal health information (PHI) data elements in data sets landing on the platform, as assigned.

• Provide custom data quality rules and associated data quality scans on the data estate.

• Facilitate access to all draft materials on request and ensure that adequate feedback is obtained from appropriate stakeholders prior to delivery of any final documents.

Main deliverables/work products for this resource, are focused in the following areas:

• Comprehensive documentation of data pipeline optimizations, enhancements, and operational support activities, serving as a reference for the Data Governance and Engineering team and stakeholders.

• Operational support documentation and issue resolution reports, detailing troubleshooting processes, solutions, and preventive measures for future incidents.

• Documentation of enhancements or updates to data profiles, extraction processes, and transport designs to improve ARC platform capabilities.

• Data extract and transport testing/quality assurance documentation for any platform capability changes or updates.

• Training and support materials for ARC platform users and support staff, as required.

• Proactive knowledge transfer to ensure seamless collaboration with the Data Governance and Engineering team, operational stakeholders, and technical teams on evolving ARC platform requirements and capabilities.

• Written status reports which outline activities completed to date. Reports to include any outstanding issues/risks that may impact the achievement of objectives at the current time or in the immediate future.

• Other deliverables as required.