02 Nov 2026
The end of part 2 alluded to a problem with the API handling multiple events. By design, Minitrace.add_field and Minitrace.add_fields make it so we don’t have to pass a Minitrace::Event object around all over the code, but that bakes in the assumption that we only work on one event at a time (from the nearest enclosing Minitrace.with_event).
And this is largely true! We typically expect control flow to move in a predictable fashion, only exiting the .with_event block when all the work is done, so the event can/should be fired at the end of the block as well. But that picture changes when we add concurrency to the mix. Now instead of the block having to finish first, it could actually be performing its work at some arbitrary other time. What would need to change about how we handle the corresponding event?
Read more...
28 Oct 2026
Note: This post has been in my drafts folder for the past 5+ years. All the concepts should have survived any intervening bitrot, so I’m dusting it off and trying to bring the series roughly up to parity with the source code I wrote so long ago, since I think the material is still interesting.
In part 1 we got basic structured events working: create an event object, add fields to it, and fire it to the backend when it’s done. But doing this by hand is at best cumbersome, if not error-prone. A good tracing framework would have an ergonomic way of managing the events under construction. In this post, we’ll develop the fundamental API that will serve us as we build out additional tracing functionality.
Read more...
03 May 2021
A while back, I wrote a talk about the path from logs to traces explaining how flat, human-readable log files can be transformed into rich, machine-parseable data structures. It’s an important conceptual overview, but leveraging traces requires the proper tooling. So I want to go one step further by building a tracing library from the ground up.
Specifically, we’ll look at the development of minitrace, a minimalist tracing framework written as a Ruby gem. The aim isn’t to be minimal in functionality, but in implementation. We’ll get to the same features found in “big” libraries like the Honeycomb Beeline or OpenTelemetry, but we’ll do it with less code and make it easier to understand. Don’t worry too much about knowing Ruby; the lessons are broadly applicable to most major object-oriented programming languages.
Read more...
13 Mar 2021
I’m starting a new job at Honeycomb soon. (No, not because of the slides.) I’ve been working as a backend engineer for the better part of a decade, but this role is different: I’m going to be working as a support lead.
I can already feel the sting of that word, “support”. It’s like engineers are allergic to it. And I don’t think I’m just projecting. When I ran the idea by one programmer friend, his initial shock betrayed his disdain. Like support is beneath us programmers. We spent some time unpacking it, stepping through what support would mean at a company like Honeycomb, what it’d mean for me, and so on. But I think it’s telling that the gut reaction was there in the first place.
What is it that makes support a second-class citizen? If I’m honest, I can’t say for sure that the disdain isn’t warranted. Maybe all that “unpacking” was really just an exercise in rationalization. I don’t know; I’ve never been in a support role before.
…Or have I? I think Chris Shively, my former coworker, might argue otherwise.
Read more...
25 Feb 2021
A blog post linking to the slides for a talk that (a) no one ever asked for and (b) I have no plans on giving. 🙃
But I can explain!
Read more...
The technical writings of