What’s the best way to partition time-series data in BigQuery? By time-unit column or by ingestion time? Daily or hourly? It depends, keep reading to learn trade-offs, pitfalls, and other traps.
I started my professional software engineer career around the year 2000. For the next 14 years I was in an individual contributor role. When I joined Allegro in 2014 I had a choice: be a senior software engineer or become a team leader. I wanted to try something new and I started working as a team leader. After 9 ye...
This evening I decided to learn something about type classes. In the past I would’ve picked a book with good reviews, or read a set of articles written by functional programming masters. This time, I wanted to check if could talk with machines 😜
Today I would like to show you how to manage technical debt by updating project dependencies continuously. Surprisingly, with this technique you will get much more goodies than just updated dependencies:
I’ve managed this blog since 2011. I’m also a primary contributor to the documentation managed by my team. For example: on-boarding guide for new colleagues or a tourist, technical guides, definition of done, architecture decision records and more. As you perhaps already know, software engineers don’t like writing t...
Go is a modern programming language designed at Google to improve developer productivity. But how to improve productivity? By limiting engineers’ power and flexibility 😂
Recently I’ve read “Software Engineering at Google” curated by Titus Winters, Tom Manshreck and Hyrum Wright.
If you are a BigData engineer who develops batch data pipelines, you might often hear that stream processing is the future. It unlocks the full potential of data that’s often unbounded in nature. You don’t need batch pipelines anymore, implement everything in a streaming fashion. There are plenty of modern and ea...
Last week I virtually attended Apache Beam Summit 2022 held in Austin, Texas. The event concentrates around Apache Beam and the runners like Dataflow, Flink, Spark or Samza. Below you can find my overall impression of the conference and notes from several interesting sessions. If some aspect was particularly appeal...
When you deploy any non-trivial data pipelines using complex infrastructure you should expect some troubles sooner or later.