Senior Assistant Manager, AIO Innovation Office (1 year contract)

Reporting to the Technical Lead, you will support the design, development, and operations of enterprise data platforms and data engineering solutions. The role involves building scalable batch and streaming data pipelines, developing ETL/ELT processes, integrating healthcare and enterprise data, optimising data platform performance, preparing technical documentation, and supporting automated deployment and production operations.

You will be responsible for the following:

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, PostgreSQL, and AWS services.
• Build and optimise ETL/ELT workflows for batch and real-time streaming data processing.
• Process and transform structured and semi-structured data from HL7, XML, JSON, CSV, Parquet, and ORC formats.
• Optimise Databricks workloads, Spark jobs, PostgreSQL scripts, stored procedures, views, and SQL queries for performance and cost efficiency.
• Implement secure data processing, including compression, encryption, secure transfer, data quality, governance, and metadata management controls.
• Develop and maintain CI/CD pipelines for automated deployment and release management.
• Prepare technical documentation, solution designs, data mapping documents, and operational runbooks.
• Monitor production data pipelines, troubleshoot issues, provide Level 2 support for Microsoft SQL Server, and drive continuous improvements.
  
  

Requirements

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Minimum 5 years of experience in Data Engineering, ETL Development, or Data Platform Engineering.
• Strong hands-on experience with Databricks, PySpark, Spark SQL, SQL, PostgreSQL, and AWS data services.
• Experience developing scalable ETL/ELT pipelines, data integration frameworks, and batch or streaming data processing solutions.
• Experience processing HL7, XML, JSON, CSV, Parquet, and ORC data formats; exposure to HL7/FHIR healthcare standards is preferred.
• Good understanding of data modelling, data governance, metadata management, data quality controls, secure data processing, compression, and encryption.
• Experience with CI/CD pipelines, deployment automation, production support, and Microsoft SQL Server administration or performance tuning.
• AWS Certified Data Engineer – Associate, Databricks Certified Data Engineer Associate, cloud lakehouse architecture, Infrastructure as Code, DevOps, governance, security, and cloud cost optimisation experience will be advantageous.
• Good analytical, troubleshooting, communication, documentation, collaboration, and stakeholder management skills.