The Pydantic team published an engineering writeup on how Logfire, their observability platform, sped up some queries by a factor of 1000 or more. The technique is called dynamic shredding, and the headline number deserves a closer look, both because it is real and because the interesting part is not the trick itself but the automation around it.
The setup
First, a correction to what you might assume. Logfire does not store telemetry in Postgres with JSONB columns. Spans arrive as Parquet files in object storage, queried through Apache DataFusion. Alongside fixed-schema columns like timestamps, span_name, trace_id, and span_id, each file carries a attributes column holding arbitrary user data as a single JSON value.
That layout is convenient on write and miserable on read. When a query filters on an attribute buried in that JSON, the engine has to download the whole column from object storage, decompress it, and parse every value to dig out one key. The team found early on that the bottleneck was not CPU time spent parsing. It was the IO of dragging large JSON blobs out of object storage when the query only needed one field.
From static to dynamic shredding
The first fix was static: a hardcoded set of common attributes like http.response.status_code extracted into typed columns at write time, with queries transparently rewritten from attributes->’key’ syntax to the physical columns. That helps only the attributes somebody thought to predict.
Dynamic shredding generalizes it. During ingestion and compaction, Logfire automatically extracts up to the top 128 largest attribute keys, ranked by total size across all rows in each Parquet file, into typed columns. No configuration, no schema declaration. Rare and pathological keys stay behind in a JSON remainder column, where they still pay the parsing cost, but the hot 95 percent of your queries hit real columns.
The 128-key cap is not arbitrary. Production analysis showed 99 percent of Parquet files have fewer than 128 unique top-level attribute keys, with a median around 32. Parquet and DataFusion handle roughly a thousand total columns comfortably; past that, metadata parsing overhead starts to show, so the cap keeps the worst case bounded.
Type handling is done with a widening rule: Int -> Float -> JSON, Bool -> JSON, String -> JSON. If an attribute holds mixed types across rows, the column falls back to a parseable JSON string rather than failing or corrupting values.
The real work was in the engine
Extracting columns is half the story. The other half is making the query engine actually use them, which meant pushing expressions and predicates on JSON accessors down into the Parquet reader. That work landed in DataFusion 52.1.0 after roughly six months of refactoring core parts of the execution engine. Combined with zone maps, min-max statistics that let the reader skip files, row groups, and pages, and column pruning that avoids downloading irrelevant columns entirely, queries stop paying for data they never touch.
This is where the 1000x number comes from. It is a best case against a pathological baseline: a query on an unshredded attribute across hundreds of gigabytes that used to time out at 30 seconds now finishes in under a second. Most queries will see far less. Some will see nothing, because their keys were already shredded. That is still the right trade for an observability product, where one abandoned query timing out is worse than ten ordinary queries staying the same.
The HN reaction, and why it is half right
The Hacker News thread pushed back hard: ClickHouse shipped the same idea before the word shredding existed, its JSON type and the Parquet Variant type both do this, and one commenter called the whole thing DB 101, which is fair. Normalizing hot fields out of blobs is textbook.
But the textbook version assumes you know the schema. In multi-tenant observability, the schema is arbitrary user data that nobody declared. The genuinely hard engineering is the discovery heuristic: ranking candidate keys by size and frequency per file, capping them, widening types on conflict, and keeping a remainder column so nothing breaks. ClickHouse’s creator showed up in the thread to note his database did it first, which is true, and also not the point. Logfire’s stated long-term plan is to adopt the standardized Parquet Variant type once that matures, which is the right destination for a technique this fundamental.
If you run Postgres
The same idea maps directly, just with different machinery. The Logfire query rewrite from attributes->’key’ to a physical column is the same move as creating a Postgres generated column or a btree expression index on attributes->>’user_id’. If you profile your JSON columns and find a handful of keys dominating your queries, extract those. GIN indexes on the whole document are the “parse everything every time” trap in Postgres clothing.
And the rollout pattern is worth stealing regardless of platform: Logfire is rewriting the query path first so it reads both layouts, keeping the old write path, and only then flipping the switch for all customers in February 2026. Physical layout migrations go wrong in ways that are hard to undo. Being able to read both formats while you verify is the difference between a migration and an incident.
- Profile your JSON blob columns for key frequency and size, then move the top recurring keys into typed columns or expression indexes.
- If you use DataFusion or Parquet for semi-structured data, 52.1.0 and later carry the pushdown work.
- Cap automated extraction, keep a remainder column, and widen types instead of failing on heterogeneous data.
- Migrate layouts read-both-first, write-new-second, flip-last.