Investigo have been retained to search and recruit for a Lead Data Engineer on a permanent basis with an exciting customer in the UK.
Our client is implementing an all new end to end cloud data platform and adopting a Dev/DataOps working culture. They need to constantly align our data platform strategy to our business strategy, ensuring our work is driven by business use cases and generated value.
The adoption of Product Management will drive this mind set. You will report to the Product Manager, Data Platform. Working closely together, the Lead Data Engineer will apply their data management and infrastructure knowledge to interpret the backlog of epics and user stories into technical developments, features and tasks for the engineering team that will achieve the defined business outcomes.
You will lead and manage a data engineering team who will use knowledge of cloud based data architecture, automated ELT and testing tools, manual coding and SPC to produce a robust, automated and efficient data platform. Key customers will include data science teams and the Data Visualisation team. Running as a Dev/DataOps unit, service ownership over support and maintenance will be prioritised against new developments to suit business outcomes.
Your Key Responsibilities:
Build and maintain the Enterprise Data Platform
* Build and maintain a robust and efficient data pipeline to bring various raw data sources (internal and external) into our cloud based storage and processing platform
* Develop and run appropriate routines, including automation and SPC, to deliver a robust and scalable service with demonstrable data quality
* Work with the Data Architect and adopt best practice methodologies to maintain a data lake archive and appropriate transformed layers to support business needs
* Support the Product Manager, Data Platform to translate and assign the backlog into features and developments to achieve the prioritized business outcomes
* Stay connected to the data management industry and constantly challenge the platform design for improvements to service, efficiency and business outcomes
Leading Data Engineering
* Coach, mentor and develop a data engineering team to deliver the backlog
* Set the appropriate technical direction for the data engineering team to support the Product Manager, Data Platform and the data strategy
* Operate a Dev/DataOps culture and working practice within the team
* Build quality through a CI/CD and release management capability
* Drive and deliver the data engineering service ownership SLAs
* Support data governance and ensure data structures are aligned to policies and captured in the data catalog
Enabling data science
* Partner closely with data science and analytics teams across the enterprise, understanding and developing solutions for their needs
* Ensure the data platform provides the functionality for a suitable Lab environment, developing over time as enterprise capabilities evolve
* Build a "Lab to Factory" process that takes data science team models into production, then constantly evaluate and retrain these models
* Prepare, maintain and make available datasets for prescriptive and predictive modelling
Experience Needed:
* Extensive experience building cloud based data engineering/data science solutions
* Relevant experience leading and developing Dev/DataOps data teams
* Extensive experience of coding and tool based solutions to know when either will offer benefits
* Extensive technical experience of building various methods of data processing, data models using a variety of database and infrastructure types
* Experience building data structures to support first and third party analysis under GDPR guidelines
* Experience migrating legacy physical environments to cloud based platforms
Skills needed:
* Capable of working with a data architect to design and build complex data environments
* Very strong SQL skills and at extensive experience in least one other coding language (Python/R/Java)
* Expert knowledge of data modelling and different data structures and their benefits and limitations under particular use cases
* Embraces diversity and proactively manages data issues
* Ability to influence peers, team, and business stakeholders
Please apply if you're interested
