The Datable team just got back from Monitorama. Below are a few of my key takeaways from the conference, and what they mean for the broader observability landscape.
My impression of the talks and attendees
The first thing that jumped out at me was how many attendees were from vendors, which felt like about half the conference. Given this was predominantly an open source conference, it makes me think these vendors are feeling the pressure from the open source world! Huzzah!
The second half of the audience were largely practitioners. Of that cohort, half were from either enterprise companies like Workday, Bloomberg, IBM, and Priceline, or from tech pioneers like Netflix, Fastly, Cloudflare, Netlify, etc. In short, places that take reliability seriously and have budgets for observability teams.
The quality of the talks was high - of course not all talks floated my boat, some coming across too abstract or self-congratulatory for me - but most were packed with anecdotes and learnings I can bring back to Datable. All in all, it was a fantastic conference, and I look forward to going next year. I would recommend you go, too!
Observability pipelines are going mainstream
At Datable.io, we’re building our own observability pipeline product to give developers and telemetry data consumers more control over what data they collect, what it looks like, and where it goes.
But we aren’t the only ones! I was surprised by the number of vendors emerging in the streaming observability pipeline space at Monitorama. Just from sponsors, you have Cribl, who are known for defining the market, but you also have ObservIQ, Chronosphere, Axiom, and Mezmo. These vendors are split between control planes for telemetry collectors (ObservIQ, Chronosphere) and centralised pipelines like us for telemetry processing (Mezmo, Axiom). Let me tell you - after the fourth sponsored talk about observability pipelines, it was beginning to feel like a crowded market!
Given the surge of these tools, it seems like the vendors are reacting to some market pressure. Perhaps the success of Cribl, or perhaps because cost and control are becoming more critical in buying decisions. Regardless, it was unclear to me if the practitioners at the conference really resonated with the pitch. Almost all large companies have their own equivalent solution in place (running through kafka), but when I asked them about it, they didn’t think of them as Observability Pipelines. Time will tell if this gets widely adopted into the monitoring stack (Here’s hoping 🤞!)
Clickhouse was the darling of the show
Clickhouse was a major sponsor of the event, but they pretty much received unfettered praise from all the speakers. There was even talk of “legacy observability vendors building their own databases” and “the next generation built on Clickhouse”. It’s a pretty extraordinary shift!
You don’t need to listen to your vendor
One major theme that emerged was that the traditional guidance of storing every raw event is usually only beneficial for the vendor. Several talks focused on the surprising cost of storing all events and it’s impact on query performance. A big message that stuck with me was it’s OK to aggregate! Aggregations and sampling at the right place can be essential components to reducing costs and keeping query performance high. You can aggregate at the collector layer, the pipeline layer, or at rest. The further away from the data source the more context you have, but the greater the cost of that aggregation.
OpenTelemetry is hard - traces are harder
One recurring theme I heard a lot was that OTel is difficult to implement at an organizational level. Several speakers mentioned the difficulty of adopting distributed traces in particular. In most cases, adoption came down to strong stakeholder support, teams building shared internal tooling, and a big need (reducing MTTR, for example). Still, a persistent downside of distributed traces is that a single team can do them wrong and break the entire trace for everyone else.
Which leads me to my final bonus takeaway -
The future of observability is standardization
While this wasn’t explicitly stated, I walked out with a strong feeling that the future of observability is a standardization on storing data in an object store (which we kinda already have with Parquet and Arrow formats), a standard on how to index the data, and layers of aggregation sitting over the top to serve use cases. Then you can drop a vendor over your data to serve your use cases, and also get the best of low cost storage, control of your data, and cost efficient queries.
We heard a lot of this in the larger companies talking about their observability strategy - and it seems the only significant piece missing is the indexing and aggregation strategy.
Wrapping up
It’s an interesting time to be in the observability space! As OpenTelemetry matures, the old guard of observability vendors are slowly turning the ship to embrace it and an entire ecosystem of new tooling is sprouting up around it. That’s certainly the case at Datable.io, where we support OTel collectors and the OpenTelemetry protocol as first-class data sources.
Overall, we had a great time at Monitorama. I would highly recommend it to any dev who’s curious about how we monitor the world!
If you’re interested in learning more about Datable.io, we are currently accepting applications for our private beta. Sign up for a demo here!