Annalect

Yes to all of this. A2A and “enterprise brain” (persistent, scoped inter-agent memory) will define the next phase of the Enterprise AI. Is your org ready? #EnterpriseAI

AI is making your work look better. It's also making your thinking worse. Both are happening at the same time. Here's how it goes: You use AI for small things. It works. You use it more. Outputs look clean, polished, professional. Somewhere along the way you stop asking "do I understand this?" And start asking "does this look right?" Those feel like the same question. They're not. "Does this look right?" is pattern recognition. "Do I understand this?" is judgment. AI gives you speed. It gives you confidence. What it quietly takes away is your ability to tell when you're wrong. Not all at once. A little, each time it works. The fix isn't to use AI less. Write your own thinking before asking AI to clean it up. Slow down at the decisions that actually matter. Disagree with it sometimes, even when it looks right. AI gives you velocity. Only you can give yourself direction.

Why is every major data platform converging on one HTTP API? The Apache Iceberg REST Catalog is quietly becoming the most important interface in the modern data lakehouse — and most teams are underestimating how much it changes. Here's the problem it solves: Traditional Iceberg catalogs (Hive Metastore, AWS Glue, custom JDBC) force every query engine to embed catalog-specific client libraries. Spark needs one connector. Trino needs another. Flink needs a third. Your Python scripts? Good luck. The REST Catalog flips this entirely. It defines a single HTTP API for all table operations — create, list, load, commit. Any engine that speaks HTTP can participate. No Java SDK required. No vendor lock-in. What this means in practice: → Spark, Trino, Flink, DuckDB, Polaris, and Snowflake all talk to the same catalog endpoint → You swap compute engines without touching your catalog layer → Catalog-level access control lives in ONE place, not scattered across engine configs → Multi-cloud becomes achievable — your catalog API is cloud-agnostic The ecosystem is moving fast. Apache Polaris, Project Nessie, Unity Catalog, AWS Glue, and Snowflake Open Catalog all implement the REST spec now. And with Iceberg v3 hitting production (CDC support, semi-structured data, variant types) and v4 already in development targeting streaming metadata latency — the REST Catalog is the stable interface that ties it all together. The architectural pattern to adopt: REST Catalog as the single source of truth → multiple compute engines reading/writing via HTTP → object storage underneath. Clean separation of concerns. Engine-agnostic. Cloud-portable. If you're still coupling your catalog choice to your compute engine, 2026 is the year to decouple. #DataEngineering #ApacheIceberg #DataLakehouse #DataArchitecture #BigData

AI will never tell you your work is bad. That's the problem. It has no reference point. So it tells you what you want to hear. The fix: → Find a YouTube video from someone who knows their craft: Alex Hormozi, Chris Voss, Robert Cialdini, anyone giving away real frameworks → Paste the URL directly into NotebookLM → Ask it to audit YOUR work using those frameworks Now it has a standard to measure your work against. And it'll use it. The frameworks exist. They're free. Most people just never load them in.

Why is everyone migrating to Apache Iceberg? Because your data lake has been lying to you about what "open" really means. Here's the uncomfortable truth: most data lakes lock you into a specific query engine. Switch from Spark to Trino? Rewrite your pipelines. Move to Flink for streaming? Good luck with consistency. Iceberg v3 changes this game entirely. Here's what makes it different: 𝗧𝗵𝗲 𝗠𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗟𝗮𝘆𝗲𝗿 𝗶𝘀 𝘁𝗵𝗲 𝗦𝗲𝗰𝗿𝗲𝘁 Iceberg doesn't just store files — it maintains a metadata tree that tracks every snapshot, manifest list, and manifest file. This is what enables: → Time travel queries (point-in-time reads without extra storage) → Schema evolution without rewriting data → Partition evolution without downtime → ACID transactions across multiple engines 𝗩𝟯 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗟𝗲𝗮𝗽 Deletion Vectors in v3 are a game-changer. Instead of copy-on-write (rewriting entire files for a single row update), Iceberg now marks deleted rows in a separate lightweight file. Result? Up to 10x faster data manipulation on large tables. Add Row Lineage tracking and native VARIANT type support for semi-structured data, and you've got a format that handles JSON-heavy event streams as cleanly as structured dimension tables. 𝗩𝟰 𝗶𝘀 𝗔𝗹𝗿𝗲𝗮𝗱𝘆 𝗕𝗲𝗶𝗻𝗴 𝗗𝗲𝘀𝗶𝗴𝗻𝗲𝗱 The Adaptive Metadata Tree proposal for v4 aims to reduce metadata operations to a single file write — critical for sub-second streaming ingestion. Relative path support will let you move tables across environments (dev → staging → prod) without metadata rewrites. 𝗧𝗵𝗲 𝗥𝗲𝗮𝗹 𝗣𝗮𝘁𝘁𝗲𝗿𝗻 𝗜 𝗦𝗲𝗲 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 Bronze layer: Raw ingestion with Iceberg's hidden partitioning (no user-visible partition columns) Silver layer: Incremental merge-on-read with deletion vectors Gold layer: Compacted, z-order sorted tables with Bloom filter indexes The catalog layer (Apache Polaris) ties it all together with fine-grained access controls that work across Spark, Trino, Flink, and DuckDB — no vendor lock-in. If you're starting a new lakehouse in 2026, Iceberg isn't just an option. It's the default. #DataEngineering #ApacheIceberg #Lakehouse #DataArchitecture #OpenTableFormat

Apache Iceberg v3 just dropped three features that fundamentally change how we build lakehouses. Here's why your team should care. Most data engineers still treat Iceberg as "Parquet with metadata." That undersells it massively. v3 introduces capabilities that eliminate entire categories of pipeline complexity. 𝗗𝗲𝗹𝗲𝘁𝗶𝗼𝗻 𝗩𝗲𝗰𝘁𝗼𝗿𝘀 — No more costly copy-on-write for updates. Instead of rewriting entire data files to delete a few rows, Iceberg now tracks deleted row positions in a lightweight side-file. The impact? I've seen UPDATE-heavy workloads drop from 45 min to under 8 min on tables with 500M+ rows. Merge-on-read becomes the default, and your storage costs plummet. 𝗥𝗼w 𝗟𝗶𝗻𝗲𝗮𝗴𝗲 — CDC is now a native property of the table itself. Every row carries lineage metadata, so downstream consumers can do true incremental reads without scanning commit logs or maintaining external watermarks. Think about it: your Flink jobs can consume only the changed rows from an Iceberg table — no more full-table scans for incremental processing. 𝗩𝗔𝗥𝗜𝗔𝗡𝗧 𝗧y𝗽𝗲 — Semi-structured data (JSON, nested payloads) now lives as a first-class column type alongside your relational schema. No more landing JSON into a STRING column and parsing it at query time. VARIANT stores the structure natively and lets you query it with standard SQL. Schema-on-read, done right. The architecture shift is real: Raw → Bronze → Silver → Gold pipelines get simpler when your table format handles deletion tracking, change capture, and semi-structured storage natively. If you're still running Hive tables or even Iceberg v2, this is the upgrade that pays for itself in the first sprint. What feature are you most excited about? Drop your thoughts below 👇 #DataEngineering #ApacheIceberg #DataLakehouse #BigData #DataArchitecture

Excited to meet with our customers and demonstrate how Databricks powers media's shift from an attention currency to a prediction economy. Let me know if you'll be there, too! Khushboo Beniwal , Tony LaVasseur , Neil Scott , Cory Schaeffler