Tune Performance for Iceberg Topics

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

This guide covers strategies for optimizing the performance of Iceberg topics in Redpanda, including improving downstream query performance, tuning the Iceberg translation pipeline, and monitoring translation throughput.

After reading this page, you will be able to:

  • Apply compaction, partitioning, and Bloom filters to improve query performance

  • Choose appropriate flush threshold and lag target values for your workload

  • Identify translation performance signals using Iceberg metrics

Prerequisites

You must be familiar with how Iceberg topics work in Redpanda. See About Iceberg Topics.

Optimize query performance

Query engines read Parquet files from object storage to process Iceberg table data. Compaction, partitioning, column statistics, Bloom filters, and schema design affect how efficiently those reads perform.

Compact Iceberg tables

Over time, Iceberg translation can produce many small Parquet files, especially with low-throughput topics or short lag targets. Compaction merges small files into larger ones, reducing the number of metadata operations query engines must perform and improving read performance.

  • Automatic compaction: Some catalog and data platform services, such as AWS Glue and Databricks, automatically compact Iceberg tables.

  • Manual or scheduled compaction: Tools like Apache Spark can run compaction jobs on a schedule. This is useful if your catalog or platform does not compact automatically.

If you observe degraded read performance or a high number of small files, investigate whether your catalog or platform supports automatic compaction or schedule periodic compaction jobs.

Use custom partitioning

To improve query performance, consider implementing custom partitioning for the Iceberg topic. Use the redpanda.iceberg.partition.spec topic property to define the partitioning scheme:

# Create new topic with five topic partitions, replication factor 3, and custom table partitioning for Iceberg
rpk topic create <new-topic-name> -p5 -r3 -c redpanda.iceberg.mode=value_schema_id_prefix -c "redpanda.iceberg.partition.spec=(<partition-key1>, <partition-key2>, ...)"

Valid <partition-key> values include a source column name or a transformation of a column. The columns referenced can be Redpanda-defined (such as redpanda.timestamp) or user-defined based on a schema that you register for the topic. The Iceberg table stores records that share different partition key values in separate files based on this specification.

For example:

  • To partition the table by a single key, such as a column col1, use: redpanda.iceberg.partition.spec=(col1).

  • To partition by multiple columns, use a comma-separated list: redpanda.iceberg.partition.spec=(col1, col2).

  • To partition by the year of a timestamp column ts1, and a string column col1, use: redpanda.iceberg.partition.spec=(year(ts1), col1).

To learn more about how partitioning schemes can affect query performance, and for details on the partitioning specification such as allowed transforms, see the Apache Iceberg documentation.

  • Partition by columns that you frequently use in queries. Columns with relatively few unique values (low cardinality) are good candidates for partitioning.

  • If you must partition based on columns with high cardinality, for example timestamps, use Iceberg’s available transforms such as extracting the year, month, or day to avoid creating too many partitions. Too many partitions can be detrimental to performance because more files need to be scanned and managed.

Skip data files with column statistics

Redpanda 26.2.3 and later writes column statistics for each data file to the Iceberg table metadata. Query engines use these statistics to skip data files that cannot match a query filter. For example, a query with WHERE redpanda.offset > 1000 can skip each file whose largest offset is 1000 or less.

Redpanda writes the following statistics for each column in each data file:

  • Lower and upper bounds

  • Null value count

  • Value count

  • Column size

You do not have to configure column statistics. Redpanda writes them for all columns. Redpanda does not read the Iceberg write.metadata.metrics.* table properties, so you cannot change which statistics it writes.

Column statistics have the following limits:

  • Redpanda always truncates each bound to 16 bytes, the same length as the Iceberg default. If many values in a string column share the same first 16 bytes, the bounds cannot separate those values, and query engines skip fewer files.

  • Bounds rarely help for columns with random values, such as UUIDs. Each file contains values from almost the full range, so the bounds of most files overlap with the filter. For exact-match lookups on these columns, use Bloom filters.

  • Data files written by Redpanda versions earlier than 26.2.3 have no column statistics. An upgrade does not add statistics to existing files. Query engines cannot use column statistics to skip these files.

    To add statistics, rewrite the files with your query engine, for example with a compaction job. The engine that rewrites the files decides which statistics the new files contain, based on the write.metadata.metrics.* table properties. By default, it collects statistics for only the first 100 columns. See the Iceberg table configuration. For Databricks, see Improve query performance.

Use Bloom filters for point lookups

A Bloom filter lets a query engine skip row groups that cannot contain a value. The engine then reads less data. Use Bloom filters for exact-match lookups on columns with many distinct values, such as a transaction ID or UUID. For example, a query with WHERE transaction_id = 'a1b2c3' can skip each row group that does not contain that ID.

Column statistics rarely help for columns with random values, such as IDs, because each file’s bounds span almost every value. Bloom filters do not help range queries. They also add little for columns with few distinct values.

A Bloom filter helps only if your query engine reads it. Before you turn on Bloom filters, check that your query engine uses Parquet Bloom filters. Each filter makes the data files larger.

By default, Redpanda writes no Bloom filters. Redpanda 26.2.3 and later can write Parquet Bloom filters for the columns that you select. To select the columns, set Iceberg table properties on the table in your catalog. You do not set a Redpanda property. Redpanda builds one filter for each selected column in each Parquet row group. Redpanda reads the table properties each time it writes a data file, so a change applies to new data files only.

The following table lists the table properties that control Bloom filters. Replace <col> with the path of the column. Start the path at a top-level field, and use dots to separate nested field names, for example user.address.zip. To select a Redpanda-defined column, use its full name, for example redpanda.key.

Table property Default Description

write.parquet.bloom-filter-enabled.column.<col>

Not set

Set to true to write a Bloom filter for the column. Redpanda ignores letter case.

write.parquet.bloom-filter-ndv.column.<col>

10000

The number of distinct values (NDV) that you expect in the column in one row group. Redpanda sizes the filter for this number. Valid values are 1 to 10,000,000. If you set a larger value, Redpanda uses 10,000,000. If you set 0 or a value that is not a number, Redpanda uses the default. In each case, Redpanda logs a warning.

Redpanda ignores this property if write.parquet.bloom-filter-enabled.column.<col> for the same column is not true.

You can set table properties with any tool that writes Iceberg table metadata, such as Apache Spark or the REST catalog API. For example, to turn on a Bloom filter for the transaction_id column with Apache Spark:

ALTER TABLE `<catalog-name>`.`<namespace>`.`<table-name>` SET TBLPROPERTIES (
  'write.parquet.bloom-filter-enabled.column.transaction_id' = 'true',
  'write.parquet.bloom-filter-ndv.column.transaction_id' = '100000'
);

Replace the following placeholders:

  • <catalog-name>: The name of the catalog in your Spark configuration.

  • <namespace>: The namespace of the table. This is the value of the iceberg_default_catalog_namespace cluster property, which defaults to redpanda.

  • <table-name>: The name of the table. By default, this is the topic name.

Keep the backticks around each part of the name. Without them, Spark rejects names that contain a hyphen, such as my-topic. For a nested namespace, put backticks around each level separately.

Redpanda does not write a Bloom filter for a column in the following cases:

  • The bloom-filter-enabled value is not true or false. Redpanda logs a warning.

  • The path does not match a column in the table schema. Redpanda logs a warning.

  • The column is a boolean column. Redpanda does not log a warning.

If a field name contains a dot, a Bloom filter path can match more than one column. For example, user.id can match a top-level field named user.id and the id field in a user struct. Redpanda writes a Bloom filter for each matching column and logs a warning. Avoid schemas that contain both.

Iceberg catalogs do not record which data files contain Bloom filters. To check a data file, inspect it with a Parquet tool, such as parquet-cli. For example, the following command checks the transaction_id column in each row group of a data file:

parquet bloom-filter --column transaction_id --values a1b2c3 <data-file>

Replace <data-file> with the path to a Parquet data file in the table. If a row group has no Bloom filter for the column, the output says that the column has no Bloom filter.

The Bloom filter properties are standard Iceberg table properties. Query engines that support Parquet Bloom filters read these properties too. When such an engine rewrites data files, for example in a compaction job, it writes new Bloom filters. The new filters can differ from the filters that Redpanda writes. For example, Iceberg’s write.parquet.bloom-filter-max-bytes property limits each filter that the engine writes to 1 MiB by default. See the write.parquet.bloom-filter-* properties in the Iceberg table configuration.

Set the NDV close to the number of distinct values in one row group. Redpanda ends a row group each time it flushes translated data to disk. Most row groups are therefore much smaller than the 128 MiB maximum. By default, Redpanda finishes a data file after about 32 MiB. See Tune translation performance.

  • If the actual number of distinct values is more than about 1.5 times the NDV, the filter becomes too full to be useful. Redpanda then drops the filter for that column in that row group. The data file stays valid, but query engines cannot skip the row group.

  • A larger NDV makes a larger filter. Each filter uses about 10 bits for each expected distinct value. At the maximum NDV, one filter uses about 12 MiB.

Avoid high column count

A high column count or schema field count results in more overhead when translating topics to the Iceberg table format. Small message sizes can also increase CPU utilization. To minimize the performance impact on your cluster, keep to a low column count and large message size for Iceberg topics.

Tune translation performance

Translation is the process in which Redpanda converts topic data into Parquet files for the Iceberg table. Each round of translation processes one topic partition at a time.

Under typical conditions, Iceberg translation has the following performance characteristics:

  • Throughput: Approximately 5 MiB/s per core.

  • Flush threshold: Controlled by datalake_translator_flush_bytes (default: 32 MiB). Each translation process uploads its on-disk data when accumulated data reaches this threshold. This is the primary control for Parquet file size.

  • Lag target: Controlled by iceberg_target_lag_ms (default: 1 minute). Redpanda tries to commit all data produced to an Iceberg-enabled topic within this window.

The flush threshold and lag target together determine the size of the Parquet files written to object storage. Larger Parquet files generally improve downstream query performance by reducing the number of metadata operations query engines must perform.

Tune flush threshold and lag target

Increase both the flush threshold and the lag target together to produce larger Parquet files with more records per file. This is the primary way to tune Iceberg translation performance.

  1. Increase datalake_translator_flush_bytes to control the size of Parquet files. A good starting value depends on your workload:

    rpk cluster config set datalake_translator_flush_bytes <bytes>

    For example, to set a 64 MiB flush threshold:

    rpk cluster config set datalake_translator_flush_bytes 67108864
  2. Increase iceberg_target_lag_ms to give translators more time to accumulate data before committing:

    rpk cluster config set iceberg_target_lag_ms 300000

    You can also set the lag target per topic using the redpanda.iceberg.target.lag.ms topic property.

    Increasing the lag target means Iceberg tables receive new data less frequently. Choose a lag value that balances file efficiency against how current your downstream data must be.

datalake_translator_flush_bytes and iceberg_target_lag_ms work best when tuned together. A high flush threshold combined with a short lag window may not improve file sizes if the lag window expires before enough data has accumulated.

To check the current values of key translation properties:

rpk cluster config get datalake_translator_flush_bytes
rpk cluster config get iceberg_target_lag_ms

To check topic-level overrides:

rpk topic describe <topic-name> -c

Optimize message size

Redpanda has validated 32 MiB as the maximum recommended message size for Iceberg-enabled topics. With large messages, each Parquet file contains fewer records because the flush threshold is reached sooner. This can reduce the efficiency of analytical queries that need to scan many records.

If query latency is a concern and your workload produces large messages, consider:

  • Reducing individual message sizes if your data model allows it.

  • Increasing datalake_translator_flush_bytes and iceberg_target_lag_ms to produce Parquet files with more records per file. See Tune flush threshold and lag target.

Turn off data file compression

By default, Redpanda compresses data files with zstd. Redpanda does not support other compression codecs. In Redpanda 26.2.3 and later, you can turn off compression with the write.parquet.compression-codec table property. Set the property to uncompressed or none on the table in your catalog, the same way as the Bloom filter properties. See Use Bloom filters for point lookups. Redpanda ignores letter case. A change applies to new data files only.

If you set any value other than uncompressed, none, or zstd, Redpanda logs a warning and uses zstd.

Compression uses more CPU. It also makes data files smaller in object storage. Before you turn off compression, test with a representative workload.

Size clusters for Iceberg workloads

When you enable Iceberg for any substantial workload and start translating topic data to the Iceberg format, you may see most of your cluster’s CPU utilization increase. If this additional workload overwhelms the brokers and causes the Iceberg table lag to exceed the configured target lag, Redpanda automatically increases the scheduling priority of Iceberg translation to help it catch up with incoming data. However, this does not substitute for adequate cluster resources.

You may need to increase the size of your Redpanda cluster to accommodate the additional workload. To ensure that your cluster is sized appropriately, contact the Redpanda Customer Success team.

Monitor translation performance

Use the following Iceberg metrics to understand whether translation is keeping pace with incoming data:

For metrics related to DLQ files, invalid records, and catalog commit failures, see Troubleshooting metrics.

If translation consistently lags despite available CPU headroom, the workload may be partition-bound. Each core translates its assigned partitions independently, so distributing data across more partitions allows more cores to contribute to translation and can improve total throughput.