Big Data Engineer
- Full-time
Company Description
Our client collaborates with high profile clients from multibillion-dollar insurance and financial industry worldwide. Combining innovative ideas and high-level engineering expertise with latest technologies they deliver tomorrow’s enterprise software, AI/Data and R&D solutions today.
Job Description
- Work on a Big Data project, building a data lake for a London, UK based client, one of Europe’s largest asset managers and a major global investor.
- Create and maintain optimal data pipeline architecture required for extraction, transformation, and loading of data from a wide variety of data sources using SQL and Hadoop ‘big data’ technologies
- Assemble large, complex data sets that meet functional/non-functional business requirements.
- Identify, design, and implement internal process improvements including automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Explore ways that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, data quality and reliability.
- Work with stakeholders including the Executive, Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs.
Qualifications
- Advanced working SQL knowledge and experience working with relational databases.
- Understanding in ETL, Data Engineering and Business Intelligence best practices.
- Experience in building and optimizing ‘big data’ data pipelines, architectures and data sets.
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Experience using at least one of the following:
- Big data tools: Hadoop, Spark, Kafka, etc.
- Relational SQL and NoSQL databases and tools: HDFS, Hive, Impala, HBase, PostgreSQL, MongoDB, etc.
- Design and implementation of BI solutions (DWH, ETL, monitoring, etc.).
- Data pipeline, orchestration, validation and workflow management tools: Oozie, Airflow, NiFi, HUE, etc.
- Stream-processing systems: Storm, Flume, Spark-Streaming, etc.
- Object-oriented/functional/scripting languages: Python, Java, Scala, Bash, etc.
- Experience with Fintech or Insurtech is a plus
Additional Information
This is an exciting opportunity to work with a highly innovative and creative team, in a great working environment using the latest technologies, methodologies and frameworks. A competitive salary and benefits package are on offer to the right candidate.