It's extremely common in graph theory to use n and m to mean vertex and edge count respectively. And if you're unfamiliar with graph theory, the notation |V| or |E| is hardly more intuitive, especially if you're not an English speaker.
What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot.
There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And metrics are something else altogether. Trying to somehow unify them would be a prime example of "wrong abstraction".
> “The next time something like this occurs again, please save me a trace.”
The building blocks for this exist. The observability platform must simply (haha) implement the pattern detectors and use them for sampling decisions.
I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times.
Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.
All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".
The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.
Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.
In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.
"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.
Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.
You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.
I don't see them as separate issues. Emitting them directly runs into the inherently poor memory locality (and potentially concurrency) of trying to produce logs, metrics, and traces from the same underlying event data representation.
It is only "easy" if performance and scalability don't matter.
> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.
Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines.
Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.
At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.
No... Adaptive sampling is a family of statistical methods to choose an appropriate decimation strategy for arbitrary events based on real-world occurrence distributions.
It can be applied to tracing, metrics, logging ("sampling") and profiling.
In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.
I don't think OTEL is necessarily "at fault" here. It's a split that's carried all throughout the observability ecosystem. e.g. in the Grafana suite of solutions you have Loki (logs), Tempo (tracing) and Mimir (metrics) to cover storage & querying for all three axis, as all of them have very distinct processing & performance characteristics.
While it may intuitively may look like there is a large overlap in the three areas there is suprisingly little, and for the few parts there are (e.g. trace <-> log correlation), OTEL does offer a standard.
Tracing is the most general of them, and the most expensive unless you're careful with the implementation.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
I think it is almost a inevitability where otel came as a standardised aggregate of OpenTracing (which was the same but only for tracing over multiple tracing implementations), logging, and metrics into a single observability standard without alienating all the individual supporting vendors.
Historically, logging and metrics have been different problem domains with different implementations for ages.
Now to your point:
Note that tracing does get the most of love, and that it does include constructs to add logging and metrics into these traces (spans actually). So you could argue that they are trying to develop a single interface.
> “The next time something like this occurs again, please save me a trace.”
Well, if you want this you either need to propagate this predicate to all points that might be involved, or always emit all traces and have the predicate included in the filter. And then you need to be able to dynamically propagate this predicate from the system/ui where you click to where you filter.
This is one of the reasons why we always propagate and emit traces and just post filter it in processing before it lands in the persistence layer.
If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load.
What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.
The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
You're pitching a solution that's incredible brittle and unnecessarily complicated if you think about it in technical terms.
For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.
Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.
It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.
> The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
How does the monitoring system have any of the context to add labels? That would only exist in application memory.
> That may be prohibitively expensive in terms of CPU/network load.
In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)
> How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.
> If I understand that correctly, it means your app always creates traces
Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.
Go is often thought of as a successor of C. C doesn't have methods, only global functions. From my perspective, Go added methods primarily so that you can use them in combination with interfaces. Given that interfaces don't support generic methods, I'm personally not convinced that this feature was worth adding.
> From my perspective, Go added methods primarily so that you can use them in combination with interfaces.
They also give you a limited form of overloading. Without methods or overloading, you end up in the situation that C and Scheme are in where every operation on a data structure has to redundantly have the data structure in its name like:
In Go you would methods for that: my_list.Clear(), my_queue.Clear() and my_map.Clear(). Now you can define a Clearer interface, which has only the Clear method. That allows you to write a function clearAndLog(item Clearer) and it will work with the list, queue and map.
Go doesn't have overloading by parameter list signature. But you can have methods with the same name defined on different types, so there is a sort of overloading or namespacing based on the receiver type. Methods give you that.
The methods in an interface can be implemented by many different types, and it's often hard or impossible to determine which of those types will be passed in to a function. For example, the io.Reader interface is implemented by many different stream-like types, and functions which accept io.Reader arguments generally can't make assumptions about which of those types they'll get.
I still remember following Andries’s “Linux kernel hacker’s hut” course he taught at the Eindhoven University of Technology (TU/e) back in 2010. Every week we’d get an assignment where we had to write exploits for commonly occurring security vulnerabilities (e.g., buffer overflows, bad printf format). It was one of the most enjoyable courses I ever followed. Thanks for that, Andries!
Hey fellow TU/e'er :) I followed his course as well, somewhere around 2004/5. Executing man in the middle attacks, writing buffer overflow exploits. Good memories!
Is this course still available? What about the course materials? I know it will be dated but if so can someone pls share the links.
Tried searching for it on google but couldn’t find it.
> If sizes are unsigned, like in C, C++, Rust and Zig – then it follows that anything involving indexing into data will need to either be all unsigned or require casts.
I don’t really get this claim. Indexing should just look up the element corresponding to the value provided. It’s easy to come up with semantics that are intuitive and sound, even if signed integers or ones smaller than size_t are used.
Indexing does that, but the indices must vary in a certain range, whose limits are frequently determined by using something like "sizeof(array)/sizeof(element)" which is an unsigned number.
This is especially inconvenient in C, where there exist extremely dangerous legacy implicit casts between signed integers and unsigned integers, which have a great probability of generating incorrect values.
Because the index is typically a signed integer, comparing it with an unsigned limit without using explicit casts is likely to cause bugs. Using explicit casts of smaller unsigned integers towards bigger signed integers results in correct code, but it is cumbersome.
These problems are avoided as said in TFA, by making "sizeof" and the like to have 64-bit signed integer values, instead of unsigned values.
Well chosen implicit conversions are good for a programming language, by reducing unnecessary verbosity, but the implicit integer conversions of C are just wrong and they are by far the worst mistake of C much worse than any other C feature.
Other C features are criticized because they may be misused by inexperienced or careless programmers, but most of the implicit integer conversions are just incorrect. There is no way of using them correctly. Only the conversions from a smaller signed integer to a bigger signed integer are correct.
Mixed signedness conversions have always been wrong and the conversions between unsigned integers have been made wrong by the change in the C standard that has decided that the unsigned integers are integer residues modulo 2^N and they are not non-negative integers.
For modular integers, the only correct conversions are from bigger numbers to smaller numbers, i.e. the opposite of the implicit conversions of C. The implicit conversions of C unsigned numbers would have been correct for non-negative integers, but in the current C standard there are no such numbers.
The current C standard is inconsistent, because the meaning of sizeof is of a non-negative integer and this is also true for the conversions between unsigned numbers, but all the arithmetic operations with unsigned numbers are defined to be operations with integer residues, not operations with non-negative numbers.
The hardware of most processors implements at least 3 kinds of arithmetic operations: operations with signed integers, operations with non-negative integers and operations with integer residues.
Any decent programming language should define distinct types for these kinds of numbers, otherwise the only way to use completely the processor hardware is to use assembly language. Because C does not do this, you have to use at least inline assembly, if not separate assembly source files, for implementing operations with big numbers.
It was undefined what happens at unsigned overflows and underflows. Therefore a compiler could choose to implement "unsigned" as either non-negative numbers or as integer residues.
The fact that "sizeof" is unsigned and the implicit conversions between "unsigned" numbers are consistent only with non-negative numbers. Therefore the undefined behavior should have been defined correspondingly.
Instead of this, at some version of the standard, I am lazy to search it now, but it might have been C99, they have changed the behavior from undefined to defined as the behavior of integer residues.
I do not know the reason for this choice, it may have been just laziness, because it is easier to implement in compilers and it leads to maximum performance in the absence of bugs. In any case this decision has broken the standard, because the arithmetic operations have become incompatible with the implicit conversions between "unsigned" types and with the semantics of "sizeof", which must be non-negative.
For non-negative numbers, the correct conversions are from smaller sizes to bigger sizes, while for integer residues the correct conversions are only in the opposite direction, from bigger sizes to smaller sizes (e.g. a number that is 257 modulo 65536 is also 1 modulo 256, so truncating it yields a correct value, while a number that is 1 modulo 256 when modulo 65536 it could be 257, 511, 769 etc. so you cannot extend it without additional information).
Judging from the implicit conversions, it is clear that the intention of the designers of C during the seventies was that "unsigned" numbers must be non-negative integers and not integer residues. The modern C standard is guilty of the current inconsistencies that greatly increase the chances of bugs
My copy of K&R already has unsigned modulo arithmetic: "unsigned numbers
are always positive or zero, and obey the laws of arithmetic modulo 2n, where n is the number of bits in the type." So if it changed it was before that, but don't think so.
I get your argument about the conversion order, but I do not buy it in terms of language design. You also do not want to go to a quotient ring implicitly, so I do not agree that this conversion direction would be more "correct" for implicit conversion either and from a practical point of view the C design is defensible.
I think the motivation originally was merely to expose the common capabilities of the hardware, nothing more. What we miss from this perspective are polynomials over F_2, but nobody pushed for this too hard so far.
reply