Time-Series Databases in IoT: Why Your Storage Choice Really Matters
Here's how it usually goes wrong.
You build an IoT prototype. Fifty sensors, one reading a minute. You throw the data into PostgreSQL because that's what you know — and it works beautifully. Queries are instant. Everyone's happy.
Then you go to production. Five thousand devices. One reading per second. Suddenly you're writing 5,000 rows every second — 432 million rows a day — and things start breaking in ways that feel personal. Dashboard queries that took 50ms now take 40 seconds. Your storage bill triples in a month. Adding an index makes writes worse. Removing it makes reads worse.
Nothing is technically broken. You just picked the wrong tool, and IoT scale found the flaw. 📊
The Short Version
IoT data has four properties that break traditional databases:
- Writes are relentless and append-only — high-frequency, continuous, never-ending; traditional B-tree indexes choke at sustained write rates
- Cardinality explodes at device scale — 10,000 devices × 5 sensors × 20 firmware versions = millions of unique series; this is the number one thing that kills TSDB performance (more on this below)
- Retention windows are long but value decays fast — raw second-resolution data is invaluable for last Tuesday's incident and nearly worthless for last year's report; you need automated lifecycle management, not a big disk
- Queries are almost always time-ranged — "last hour," "average per device per day for the past quarter"; no random-row lookups, ever
Traditional databases miss all four. TSDBs are built for exactly them.
How TSDBs Actually Work
Three engineering bets explain most of the performance gap:
LSM-trees instead of B-trees — new data appends sequentially to memory, then flushes to disk in sorted batches; no in-place index updates per insert; write amplification collapses under sustained IoT ingestion
Columnar storage + compression — all temperature readings sit together, all timestamps sit together; delta encoding reduces consecutive timestamps to almost nothing; delta-of-delta and run-length encoding handles slow-changing sensor values; real deployments report 97% compression ratios vs raw storage 🗜️
Time partitioning — data automatically chunked by time interval; "last hour" touches one chunk instead of scanning billions of rows; old chunks compress, tier to cheaper storage, or drop — as a single cheap operation
The Cardinality Trap (Read This Carefully)
This is the failure mode that catches experienced teams.
Cardinality = the number of unique time series in your database = device count × sensors per device × every tag dimension multiplied together.
10,000 devices × 5 sensors = 50,000 series. Fine. Add a firmware version tag with 20 values: 1 million series. Add a session ID as a tag — a genuinely common mistake — and you've created unbounded cardinality that grows forever.
When cardinality spikes, performance doesn't dip — it can collapse entirely. Index bloat undermines the ingestion efficiency the architecture was designed to deliver.
The rule: never tag with unbounded values. Session IDs, request IDs, timestamps-as-tags — all forbidden. Keep high-churn identifiers in fields, not tags. ⚠️
The 2026 Landscape
- TimescaleDB — PostgreSQL extension; its superpower is JOIN: sensor data and relational metadata (device records, maintenance schedules, asset metadata) in one query, one database, one SQL dialect. Best when context joining is a core requirement — which it usually is in serious IoT deployments
- InfluxDB 3 — purpose-built, rebuilt on the FDAP stack for millions of points/second; unlimited cardinality; mature IoT ecosystem with Telegraf and line protocol designed for constrained devices
- QuestDB — fastest out-of-the-box ingestion in TSBS benchmarks; extends SQL with real-time temporal joins; best for lowest-latency and most demanding high-throughput workloads
- TDengine — built explicitly for industrial IoT; clustering in the open-source core; strong for large industrial fleets and edge deployments
- ClickHouse — not a TSDB but a columnar analytics engine; best when batch ingestion and analytical scale matter more than per-point latency
(All vendor benchmarks come from the vendors. Test against your own workload. 🧂)
💡 Final Thought
Storage feels like the boring part of IoT architecture. But every dashboard, every alert, every predictive model, every digital twin reads from that layer. When it's wrong, everything above it feels broken — and the fix means migrating billions of rows while production keeps writing.
The failure mode is delayed. Traditional databases work fine right up until they suddenly don't, usually at the worst possible scale.
Pick for the fleet you'll have in three years, not the prototype you have today.
→ Full breakdown: why B-trees break, the full data lifecycle strategy, the cardinality trap in depth, the complete 2026 database comparison, and the builder's decision guide: Read the deep dive
Follow for more IoT architecture and data engineering deep dives — part of my ongoing 101-story series. 🔬
Comments
Post a Comment