Running Open Data Formats in Production

Open data formats like Apache Iceberg are one of the most exciting shifts in data engineering right now. For a long time, storing analytical data meant tying it closely to a single engine. Moving toward shared, open formats lets different tools work on the same data, makes collaboration across ecosystems much easier, and gives teams real long-term flexibility.

At the same time, adopting an open format changes how you run your data stack. Integrated databases and data warehouses bundle storage, compute, and metadata together. That tight coupling limits your options, which is why teams look for open alternatives. But it also lets a single system take care of a lot of routine maintenance for you.

When you unbundle those pieces around an open table format, the trade-offs look a bit different than with portable processing frameworks like Apache Beam, where a single runner still manages each job once it starts. With open tables, your data lives as files in cloud storage tracked by a catalog, without one central database server sitting in front of every request. That openness is what lets so many engines build on the same data, and it also means it helps to have a clear plan for running things day to day.

Areas Worth Planning for Early

Getting two tools to read the same test table takes an afternoon, which makes it easy to assume everything else will take care of itself. In production, a few operational details deserve attention up front.

Feature parity across tools

Open specifications evolve quickly, and different query engines adopt new capabilities on their own timelines. One engine might already support newer features like row-level updates and deletes, while another tool in your stack starts with basic reads and adds full write support or newer update modes later. Because of that, a multi-tool setup is usually shaped by the subset of features all your tools handle well.

Shared tables also mean teams need to talk to each other more when troubleshooting. Let’s say one team’s pipeline writes small updates every minute to keep data fresh, and another team’s hourly report slows down as a result. Neither team broke the specification, but their read and write patterns still affect each other.

Ongoing file and metadata upkeep

In an integrated database, you rarely think about how files are laid out on storage because the engine merges small files and cleans up deleted rows behind the scenes.

With open table formats, frequent updates and deletes create new files and table versions over time. The formats include solid mechanisms to merge small files and expire old versions. Both open-source projects and managed services are automating more of this every year, but someone still has to set it up, pay for it, and keep an eye on it:

• Running cleanup too aggressively uses extra compute and can compete with active writes, while waiting too long slows down reads.
• Teams writing high-frequency updates are not always the same teams paying for background maintenance, so costs can surprise a central platform team if nobody tracks them.
• Just like backups or indexing, cleanup jobs need a clear owner and alerts for when they fall behind.

Consistent permissions and data locality

In a single database, access rules live in one place. You grant access to a table or a set of columns, and that engine enforces it.

When data sits in shared cloud storage and multiple tools query it, access control takes more care. Giving every tool broad access to the storage folders bypasses table-level rules, so teams need a consistent catalog and permission setup across SQL engines, Python scripts, and notebooks.

Physical location matters too. Storing data in an open format does not automatically remove network latency or transfer costs across regions and clouds. Reading across them works well with smart caching or thoughtful data placement, but direct reads over long distances can get slow and costly.

Batch roots and interactive workloads

Many open data formats started in the batch processing world, where pipelines ran hourly or overnight. If a scheduled job took twenty minutes instead of fifteen, nobody minded.

Today, teams want to use open tables for much more than batch pipelines, including interactive dashboards and ad-hoc exploration where people expect answers in seconds. That shift raises the bar for table health. Too many small files or bloated metadata quickly show up as slow queries unless your stack keeps up with file layout, cleanup, and caching.

Setting Your Team Up for Success

The tooling has come a long way over the last couple of years, and all of this is much easier to handle than it used to be. A few practical habits help teams get the upside without taking on unnecessary complexity:

1. Grow into complexity step by step Start from the problem you actually need to solve today.¹ You do not have to run five engines on day one to benefit from open formats:

   • Starting with one primary engine that reads and writes open tables already keeps your options open, with little operational overhead.
   • When other teams need access, letting multiple tools read while keeping writes and maintenance in that one engine avoids most concurrency headaches.
   • Save concurrent writes from multiple engines for cases that genuinely need it, since that setup takes the most work to run well.

2. Test a normal week, not just an afternoon demo Writing a hundred rows in one tool and reading them in another is a fun first milestone, but it only tests the happy path. Before moving critical workloads, simulate a realistic week:

   • Run continuous writes and read queries together for several days.
   • Exercise column changes, updates, and deletes under normal load.
   • Let file-merging and cleanup jobs run alongside live traffic, and check what they cost.
   • Interrupt a write job halfway through to see how it recovers.

3. Make maintenance ownership explicit Whether your team runs open-source maintenance tools directly or relies on a managed service that handles optimization in the background, make sure someone clearly owns table health from day one. It also helps when teams generating heavy update traffic can see the maintenance cost their pipelines create. Sharing that visibility early keeps product and platform teams on the same page.

4. Keep portability real over time Flexibility is one of the biggest reasons to invest in open formats, and like disaster recovery, it is worth testing every now and then. Over time, pipelines can quietly start depending on tool-specific assumptions or extensions. A simple periodic check where a second open-source tool reads (and, in a test environment, writes) a few sample tables is an easy way to make sure your data stays as portable in practice as it is on paper.

It is an exciting time to work in data infrastructure. Open formats are giving teams far better ways to share data and adopt new tools without painful migrations, and pairing that openness with good operational habits makes the experience much smoother.

I hope you find these observations useful. If your team has been adopting open data formats, I would love to hear what you have learned along the way. Feel free to reach out.

Disclaimer: Opinions are my own and not the views of my employer. This post is based on my own observations, and I used AI tools to help write it.