New Website & Scala Days 2025 Announcement

Tickets & Details↑

➡️ Details here: scaladays.org/editions/2025/talks/using-scala-in-a-go-first

➡️ Tickets: scaladays.org/tickets

Context↑

We run a small, but very impactful Scala codebase @ work. It’s a set of self-hosted Flink jobs (using ngrok + Kubernetes), written in Scala 3 with the wonderful flink-scala-api.

They are responsible for ingesting a lot of important, real-time networking and metadata we use for support, analytics, GTM and other stuff. We average ~20k events/s, made of up various, rather complex data streams. I wrote a bit more about it here not too long ago.

Introducing a new language (and one that isn’t Python) into our engineering stack was certainly a big decision. The Data Engineering world is tightly integrated in our regular product engineering stack and not a separate silo. I advocate heavily for this and hightly suggest ever Data Engineer to do the same. I’m talking about using the same repo(s), CI, tooling, developer CLI, formatting etc. as the main codebase, as well as using the same processes (on-call, deployments) and auxillary tooling (observability etc).

And Scala offer a lot of neat features that make this work, especially around writing generic, safe code. Writing truly generic code - using more than just basic invariant generics and simple interaces, like go - that has actual compiler support to help us enforce it (unlike Python) is one of the few ways we keep a large Data Platform maintainable by a team of one (and we do use Python, too, just not for real-time stuff).

Speaking of: I’m the only full-time DE, and I don’t even spend my entire time on the Data Platform, so making things generic and re-usable is one of the top priorities to keep this maintainable.

Almost all code is expressed as Thing[A : MaybeATypeClassOrTwo] extends MaybeATrait (no effect system), which makes extending it reasonably easy - it’s impossible for me to maintain hand-grown, artisinal schemas like shops with large DE teams might. It is possible to maintain a largely generic, self-hosted ecosystem with good observability and tooling.

Since the rest of the code base is almost all go, this could have gone horribly wrong. But there’s a lot of lessons learned over the past year and a half or so that I found worth sharing. And in this talk, I’ll try to do just that.

Hope to see you there!

What↑

I’ve been dabbling in frontend/TypeScript stuff - you’re looking at it - and essentially heavily customized the default Astro theme with almost all features of ink-free, namely:

• Table of contents
  • + Floating TOC
• Tag support
• Tag search
• Word count + reading stats [1]
• Several internal components, such as the preview boxes, dynamic icons with gradients and such
• Some custom mdx components, such as images with proper attribution, admonitions etc
• Correct Open Graph support, I think?
• Reasonable mobile support

If you find something is broken, please reach out.

[1]: Those aren’t 100% correct, since I use pandoc to do it and I think I need a Lua script to filter code blocks…

Markdown translation↑

It’s still using markdown for the blog articles, but I was forced to turn them into mdx to support all the existing features.

I figured writing a full article about this wouldn’t be too interesting, since the most interesting thing (other than me figuring our “modern” web dev) was translating all 35 articles (150k+ words) into mdx with the new syntax. This was done with throwaway Python.

While most things are the same, things like admonitions, need a different syntax:

735,736c742,744
< > [!TIP]
< > Code diff [here](https://github.com/chollinger93/bridgefour/pull/6/files).
---
> <Aside variant="tip">
> Code diff [here](https://github.com/chollinger93/bridgefour/pull/6/files).
> </Aside>
1250c1258
< {{< figure src="assets/raft_state_machine.png" caption="State transitions for leader election" attr="Howard, Heidi, Malte Schwarzkopf, Anil Madhavapeddy, and Jon Crowcroft. 'Raft Refloated: Do We Have Consensus?.' *ACM SIGOPS Operating Systems Review* 49, no. 1 (2015): 12-21." attrlink="https://api.repository.cam.ac.uk/server/api/core/bitstreams/c9bcee5b-a1cb-4147-9281-1a05632f5aa3/content">}}
---
> <ImgAttr src={img_raft_state_machine} caption="State transitions for leader election" attr="Howard, Heidi, Malte Schwarzkopf, Anil Madhavapeddy, and Jon Crowcroft. 'Raft Refloated: Do We Have Consensus?.' *ACM SIGOPS Operating Systems Review* 49, no. 1 (2015): 12-21." attrlink="https://api.repository.cam.ac.uk/server/api/core/bitstreams/c9bcee5b-a1cb-4147-9281-1a05632f5aa3/content" />

I also used it to automatically parse all images - ![img](path) syntax or Hugo’s {{ < figure}} - and turned the first match of either into a OpenGraph title image (which the previous blog usually didn’t have).

In the same pass, I also made sure all slugs matched (by querying the old blog while it was live for each to ensure we have no dead links):

< slug: "a-distributed-system-from-scratch-with-scala-3-part-3-job-submission-worker-scaling-and-leader-election-consensus-with-raft"
---
> slug: "2025/05/a-distributed-system-from-scratch-with-scala-3-part-3-job-submission-worker-scaling-and-leader-election-consensus-with-raft"
> img: /assets/og/2025-05_bridgefour-part3.png
8a10,12
> import img_raft_state_machine from 'assets/raft_state_machine.png';
> import img_dependency_2x from 'assets/dependency_2x.png';

Similar things happened for GitHub gist support and others.

Why↑

Largely because I wanted more features, flexibility, and a consolidated website that is independent of LinkedIn as a “professional” presence.

I also noticed my web dev knowledge was growing more and more outdated - my most up-to-date JavaScript book is from 2011 and I own a largely unread CSS book from 2012 - which caused me some pain when working with Slidev, for instance. I have no interest in doing FE or full stack full time (I like my backend work, thank you very much), but I think it’s a useful thing to know.

It’s still static site generation at the end of the day, so I didn’t quite jump into the React (and others) rabbit hole.

Minimalism↑

It’s also a bit of a departure from my previous minimalism. My old website was a plain html file with some inline-CSS that could be maintained via ssh + vim with a dial up connection from the middle of the woods. The point here was to avoid the “modern web” bloat of shifting content that trick you into clicking something, scrolling popups, auto-playing videos, endless trackers and CDNs (…).

However - one can do this with modern tools. I have some work to do to make this site more accessible to slower connection speeds (it’s pretty managable via simulated 3G, but not great), but I don’t think using tools made after the year 1991 is a bad thing - provided I’m not the one actively supporting dark patterns.

That said, in the interest of transparency: I am using a self-hosted umami instance for web analyitics. It does respect your “Do not track” header, but I figured I’d let you know. Using goaccess is great and all, but this is better - it’s nice to know what people like to read, I guess. This seemed like an implementation that’s compatible with my privacy standards.

ink-free & Hugo↑

Hugo is great and I plan to support the previous blog’s template project (ink-free - contributions welcome), but working on a private theme allows me to make changes that might not be suitable for a re-usable theme, which makes this a lot easier to maintain for myself, since I can e.g. hard-code text or URLs without having to think about how I can make this configurable for others. While writing this, I adjusted the <blockquote> margins without having to create a PR, test it, cut a release etc.

I’m not necessarily happy about that (I love Open Source), but I unforunately have to spend my time on side projects somewhat wisely.