Marcin Kuthan

Big data engineer leading @allegro real-time, highly scalable and fault-tolerant data ingestion platform. It processes a few billion events every day with e2e latency below one minute.

I published post on the allegro.tech blog, how to integrate Spark Streaming and Kafka. In the blog post you will find how to avoid java.io.NotSerializableException exception when Kafka producer is used for publishing results of the Spark Streaming processing.

http://allegro.tech/spark-kafka-integration.html

You could be also interested in the following part of this blog post where I presented complete library for sending Spark Streaming processing results to Kafka.

Happy reading :-)

Updated:

Comments

You May Also Enjoy

Flink Forward 2024

9 minute read

This week, I attended Flink Forward in Berlin, Germany. The event celebrated the 10th anniversary of Apache Flink. Below, you can find my overall impressions of the conference and notes from several interesting sessions. If an aspect was particularly appealing, I included a reference to supplementary materials.

Foundations of scalable systems – book review

1 minute read

Foundations of scalable systems written by Ian Gorton, the book with my highest rate of 5 ⭐️ I highly recommend that every software engineer grasps the distributed system principles outlined in this book.

Building evolutionary architectures – book review

1 minute read

September this year is dedicated to reviewing software architecture books. This time, I’ve read Building Evolutionary Architectures written by Neal Ford. I appreciate the effort put into compiling this book. However, I found it to be more of a collection of existing books, articles and talks rather than offering new...

Fundamentals of software architecture – book review

13 minute read

Recently I’ve read Fundamentals of Software Architecture written by Mark Richards and Neal Ford. I found this book valuable, even though my company doesn’t have a formal architect role. At Allegro, the most experienced senior software engineers take on the responsibilities of a software architect in addition to thei...