Description
- Datastage experience required
- Required : Hadoop, Spark, Kafka experience for existing ETL pipeline support
- Work schedule : 3 days in Office (Monday, Tuesday, Wednesday) + 2 days remote
Specific projects:
Maintain our existing data structures batch + streaming in Hadoop (Cloudera) while we transition to GCP Cloud. Manage service desk inquiries by debugging, suggesting, implementing changes and/or improvements following business guidelines and procedures.
The Big Data Engineer will be a key member of the Field Services BI team. You will work closely with Hadoop Administrators, Data Scientists, and business stakeholders.
Job Duties/Accountabilities:
Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources
Works to identify gaps and improving the platform’s quality, reliability, robustness, maintainability, and speed
Contributes design, code, configurations, and documentation for components that manage data ingestion, real time streaming, batch processing, data extraction, transformation, and loading across multiple systems
Manage service desk inquiries by debugging, suggesting, implementing changes and/or improvements following business guidelines and procedures
Critical Qualifications/Competencies:
Training/Education
Bachelor’s degree in Engineering, Computer Science, Operations Research, Science, Mathematics or equivalent experience and / or education
Skills
Autonomous, resourceful, strong tolerance to ambiguity
Strong analytical, communication and time management skills
Strong troubleshooting skills
Excellent ability to articulate and document technical concepts
Technical Knowledge
Experience working with distributed data technologies (e.g. Hadoop, MapReduce, Spark, Kafka)
Experience building streaming and batch jobs in Spark
Experience with Enterprise class ETL tools such as IBM DataStage
Working Conditions:
Ability to work efficiently in virtual teams with geographically dispersed team members
Fast paced, continuous learning environment with new technologies
Good work / life balance opportunities
Bilingualism is an asset (English and French); adequate knowledge of French is required for positions in Quebec.
