Marcin Kuthan

Big data engineer leading Allegro engineering team that develops highly scalable and fault-tolerant data ingestion and analytics platform.

Apache Spark

GCP Dataproc and Apache Spark tuning

March 24, 2022 8 minute read

Dataproc is a fully managed and highly scalable Google Cloud Platform service for running Apache Spark. However, “managed” does not relieve you from the prop...

Long-running Spark Streaming jobs on YARN cluster

September 30, 2016 16 minute read

A long-running Spark Streaming job, once submitted to the YARN cluster should run forever until it is intentionally stopped. Any interruption introduces subs...

Spark application assembly for cluster deployments

March 11, 2016 12 minute read

When I tried to deploy my first Spark application on a YARN cluster, I realized that there was no clear and concise instruction how to prepare the applicati...

Spark and Kafka integration patterns - part 2

January 29, 2016 21 minute read

In the world beyond batch, streaming data processing is a future of dig data. Despite of the streaming framework using for data processing, tight integrati...

Spark and Kafka integration patterns - part 1

August 6, 2015 less than 1 minute read

I published post on the allegro.tech blog, how to integrate Spark Streaming and Kafka. In the blog post you will find how to avoid java.io.NotSerializableExc...

Spark and Spark Streaming unit testing

March 1, 2015 11 minute read

When you develop a distributed system, it is crucial to make it easy to test. Execute tests in a controlled environment, ideally from your IDE. Long develop-...