Posts

All the articles I've posted.

• 14 MIN READ•May 24, 2026

  Choosing Vector Stores for Retrieval Workloads

  pgvector, Milvus, Weaviate, and LanceDB each make different tradeoffs on index type, hybrid search, scale, and operational complexity. Learn which fits your retrieval workload.

  Vector Store Comparison Retrieval WorkloadsPgvector HnswMilvus Hybrid Search
• 24 MIN READ•May 23, 2026

  Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail

  Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn when to transition to Dremio MPP.

  DuckDBApache ArrowDataFusion
• 20 MIN READ•May 23, 2026

  An In-Depth Overview of the Apache Iceberg 1.11.0 Release

  Apache Iceberg 1.11.0 delivers manifest list encryption, the new pluggable File Format API, credential lifecycle refreshes, and Spark/Flink improvements.

  Apache IcebergData LakehouseOpen Table Format
• 22 MIN READ•May 22, 2026

  Open Table Format Benchmarks: Why They Require Critical Evaluation

  An in-depth analysis of open table format benchmarks comparing Apache Iceberg, Delta Lake, and Apache Hudi, detailing the pitfalls of standard benchmarks and how to choose a format.

  open table formatsapache icebergdelta lake
• 27 MIN READ•May 22, 2026

  Apache Iceberg SCD Type 2 and CDC Patterns: Building Historical Lakehouse Tables

  A deep dive into implementing Slowly Changing Dimension Type 2 (SCD Type 2) patterns and Change Data Capture (CDC) pipelines on Apache Iceberg, using PySpark and Dremio.

  apache icebergcdcscd type 2
• 25 MIN READ•May 22, 2026

  Setting Up an AWS-Native Open Lakehouse: Querying Apache Iceberg with AWS Athena and AWS Glue Catalog

  A comprehensive guide to building an open, high-performance lakehouse on AWS using Apache Iceberg, AWS Glue Catalog, Amazon S3, and S3 Tables, with query acceleration via the Dremio engine.

  Apache IcebergAWS AthenaAWS Glue Catalog
• 24 MIN READ•May 22, 2026

  Apache Iceberg Catalogs Explained: REST, Glue, Hive Metastore, Polaris, Nessie, and Snowflake

  A deep dive into Apache Iceberg catalog architecture, comparing REST catalogs, AWS Glue, Project Nessie, Polaris, and Snowflake. Learn catalog role, credential vending, and cross-engine configurations.

  apache icebergcatalogsNessie
• 24 MIN READ•May 22, 2026

  Maintaining Apache Iceberg Tables: Compaction, Snapshot Expiration, and Orphan File Cleanup

  An in-depth guide to orchestrating maintenance operations on Apache Iceberg tables, covering bin-packing, sort-based, Z-Order compaction, snapshot expiration, and orphan file removal, with query acceleration details for the Dremio engine.

  Apache IcebergCompactionData Engineering
• 24 MIN READ•May 22, 2026

  Apache Iceberg with Spark: Create, MERGE, Upsert, and Evolve Tables End to End

  A comprehensive developer guide to configuring Apache Spark with Apache Iceberg, executing transactional writes, and managing schema evolution.

  apache sparkapache icebergdata engineering
• 21 MIN READ•May 22, 2026

  Building a Multicloud Agentic Lakehouse Reference Architecture

  A reference architecture for building an open, multicloud Data Lakehouse optimized for AI Agents using Apache Polaris, Apache Iceberg, and Dremio.

  data lakehouseagentic lakehouseapache polaris
• 21 MIN READ•May 22, 2026

  Common Misconceptions About Data Lakehouse and Apache Iceberg

  Addressing common search queries and reader confusion about Data Lakehouse architectures, Apache Iceberg catalogs, partitions, and lock-in.

  data lakehouseapache icebergdata engineering
• 7 MIN READ•Apr 29, 2026

  Migrating to Apache Iceberg: Strategies for Every Source System

  Migrate to Iceberg from Hive, data warehouses, or raw files using in-place migration, full rewrite, or the zero-downtime view swap pattern.

  migrating to Apache IcebergHive to Iceberg migrationIceberg migration strategy