Awesome notes

When I found an interesting article or resource, if something opened my eyes I documented it here.

• Cost optimization of Dataflow pipelines - beyond optimization, a collection of references to valuable materials

• How we built a new powerful JSON data type for ClickHouse - variant and dynamic types, dense storage, arbitrary nesting.

• Consistency Models - consistency models for concurrent systems.

• Big Data is dead - by Jordan Tigani, ex googler, DuckDB advocate.

• Apache Druid tuning guide - how to achieve sub-second responses.

• Apache Beam Design Documents - These documents, managed by Googlers, provide an explanation of the Beam programming model and more.

• Dataflow Streaming Execution Model - Processing within exactly-once guarantees and with latencies consistently in the low single-digit seconds.

• How NAT traversal works - How to traverse firewalls, SNAT, CGNAT in mesh networks.

• Tracking health over debt - How to evaluate projects quality using lenses for: development, operations and architecture.

• How to write efficient Flink SQL - What is a difference between: dual-stream, lookup, interval, temporal and window joins.

• Part 1, part 2 and part 3 - Fleet-wide refactoring at Spotify, how to get 70% adoption of shared library in 7 days?

• Lessons from Building Static Analysis Tools at Google - Finding bugs is easy, but how make them actionable at scale?

• BigQuery metadata - When Metadata is Big Data.

• Prompt engineering - How to talk to GenAI (in polish)?

• Datadog incident - Multi-region and multi-cloud architecture didn’t necessarily help to avoid the outage if you forget about legacy security in-place update channel.

• The Engineer/Manager Pendulum - You don’t have to choose to be one or the other, but that you must choose one at a time.

• JavaScript event loop and Node.js event loop - Runtime models for JavaScript in the browser and server side.

• Learn You a Haskell for Great Good! and Learning Scalaz - Side by side comparision.

• Valhalla - Mechanical sympathy for Java.

• test && commit || revert - How to reduce size of the change to increase velocity? Revert the change always if tests fail!

• Learning different technologies based in the Scala Programming Language - Coding exercises for Standard library, FP in Scala, Cats and other libraries.

• Abstract Type Members versus Generic Type Parameters in Scala - trait Collection[T] {} vs. trait Collection { type T }.

• The Principles of the Flix Programming Language - Design principles of Flix programming language and Flix compiler. Awesome checklist.

• Local-first applications with automerge - Conflict-Free Replicated Data Type (CRDT), binary protocol, lossless snapshots, JavaScript and Rust (WebAssembly) implementations.

• Re-architecting Apache Kafka for the cloud - Tiered storage, Kraft cluster management (instead of ZK), multi-tenancy (consumer/producer quota, threads, connections, replication quota).

• Understanding the Python GIL - Global interpreter lock, the reason why Python doesn’t scale on multi-core architectures.

• Roblox Return to Service 10/28-10/31 2021 - Postmortem from non-trivial 73h outage due to Consul failure, why it took so long? Consul monitoring didn’t work without Consul 😜

• Exception Handling Considered Harmful - If you have always perceived that exception handling doesn’t really work.

• The Scheduler Saga - Implement Goroutines from scratch: global run-queue, distributed run-queue, work stealing, handoff, cooperative preemption, FIFO vs LIFO.

• A Guide to the Go Garbage Collector - Interesting how it differs from JVM, one or two knobs for configuration.

• Minimal version selection (MVS) - Dependency management algorithm to produce high fidelity builds.

• Documenting architecture decisions - One or two pagers with the motivations behind important architecture decisions, without ADRs this kind of knowledge evaporates.

• Code review automated comments - Framework used by many tools for commenting code reviews (documentation linters, static code analysis, test coverage).

• Write a documentation like a pro - Apply Microsoft or Google style guide for your writing.

• Test sizes - Google way for tests classification: small, medium, large (in addition to test scopes: unit, integration, e2e).

• Pre-commit: Don’t Git hooked! - Don’t use Git pre-commit hook for checking code quality, valuable checks take too much time to run.

• Don’t deploy applications with Terraform - Terraform is for static resources management, not for application deployment.