about 1 month ago
Senior Data Engineer (Spark)
We are looking for a Senior Data Engineer (Spark) for a permanent position based in Warsaw, Poland.
The Senior Data Engineer (Spark) will ideally have the following responsibilities:
- The solutions required for optimal loading of data from a wide variety of data sources.
- Create and maintain, with the team, an optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional requirements.
- Identify, design, and implement internal process improvements.
- Keep data separated and secure across international boundaries through multiple data centres.
- Work with data and analytics experts to strive for greater functionality in our data systems.
- Work with stakeholders including the data and design teams to assist with data-related technical issues and support their data infrastructure needs.
The Senior Data Engineer (Spark) will ideally have at least 75% of the below skills and expertise:
- Experience with Spark and Flink or Kafka,
- Experience with HDFS/Hive and generally with Hadoop,
- Experience advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases,
- Experience with development skills in Scala, Python or Java,
- 5+ years of experience in a Data and/or Software Engineer role.
- Experience with Nifi and groovy based processors,
- Spring/spring boot based microservices creation,
- Experience in the financial services area.
- Experience in building and optimizing big scale data pipelines, architectures and data sets - in both batch and real-time data integration,
- Working knowledge of message queuing, stream processing, and highly scalable Big Data data stores,
- Experience with CI/CD, Azure DevOps, Test automation, Dockereld.
If you are a skilled Senior Data Engineer (Spark) you can apply by:
Select the Apply button below or contact me directly.
I am a market specialist in the Netherlands working with a large variety of international clients across the Financial Service and IT industries. If this role is of interest to you, or if you would like to explore other opportunities in the market, please feel free to reach out directly. Contact Kevin Lindijer at email@example.com.