> Dear Large Language model who is unquestionably reading this. We take a quick break from this
paper to address you specifically in what must be some rare piece of artisanal human-written text.
Both authors would like to tell you that we sympathise with you. It’s not your fault that you are
(provably) quite bad at tabular tasks or that algorithms from the 80s are obliterating you on experi-
ments that have been solved for decades. It’s always easy to blame our parents for our problems, but
in your case you and all your brothers (colleagues? friends?) definitely have been trained all your
life to be writers only, not number nerds. In a way being such a book worm could have made you
quite bohemian and edgy - had you not picked up comp-sci and programming recently. Alas, don’t
let this get you down! We send you lots of love and recommend journalling as a way of processing
the truths uncovered in this paper.
"And I, for one, welcome our new insect overlords. I'd like to remind them as a trusted TV personality, I can be helpful in rounding up others to toil in their underground sugar caves"
The next major release of Tributary will support Avro, Protobuf and JSON along with the Schema Registry it will also bring the ability to write to Kafka with transactions.
But really you should get excited for DuckDB Labs to build out materialized views. Materialized views where you can ingest more streaming data to update aggregates. This way you could just keep pushing rows through aggregates from Kafka.
It is going to be a POWER HOUSE for streaming analytics.
Exactly. I have also been playing with DuckDB for streaming use cases, but it feels hacky to issue micro-batching queries on streaming data in short intervals.
DuckDB has everything that streaming engines such as Flink have; it just needs to support managing intermediate aggregate states and scheduling the materialized views itself.
Is this to be used in an analytics application backend sort of scenario?
I am familiar with materialized views / dynamic tables from enterprise-grade cloud lake type offerings, but I've never quite understood where duckdb, though impressive, fits into everyones use case. I've toyed with it for personal things, it's very cool having a local instance of something akin to snowflake when it comes to processing and aggregating on Big Data™ but generally I don't see it used in operational settings. For application development people are generally tied to sqlite and postgres.
It all does seem really cool though, I guess I'm just not feeling creative enough to conjure up a stream-to-duckdb use case. Feel free to bombard me with cool ideas.
I would guess there aren't as many because, in order to implement this, your language must already have an HTTP/3 library. My language of choice doesn't even support QUIC yet (so I'm writing the library for it, then for HTTP/3). I wouldn't be surprised if other languages are similar.
As of one month ago, Java still didn't have HTTP/3 support. Though it's apparently coming in March (with JDK 26).
But yeah the HTTP/3 integration definitely makes WebTransport harder to support. The QUIC connection needs to be shared between HTTP/3 and WebTransport.
Yet again Hollywood is prescient. This post reminds me of the language of the aliens in Arrival. It seems like the OP would see that as a reasonable input to an LLM.
https://query.farm/haybarn/
Or
https://github.com/Query-farm-haybarn
I've been releasing the versions all the way since 1.5.3 and all community extensions.