Cloud

Query Iceberg Topics using Snowflake and Open Catalog

This guide walks you through querying Redpanda topics as Iceberg tables in Snowflake, with Amazon S3 as object storage and a catalog integration using Open Catalog.

After reading this page, you will be able to:

  • Configure AWS IAM credentials granting Open Catalog access to your S3 bucket

  • Integrate Redpanda Iceberg topics with Snowflake using Open Catalog

Prerequisites

  • rpk or familiarity with the Redpanda Cloud API to use secrets in your cluster configuration. For rpk, see Install or Update rpk. For the Cloud API, you must authenticate using a service account.

  • A Snowflake account.

  • An Open Catalog account. To create an Open Catalog account, you require ORGADMIN access in Snowflake.

Authorize access to Open Catalog

You must create an AWS IAM policy and role that Open Catalog and Snowflake use to access the S3 bucket where your Iceberg data is stored. Redpanda writes Iceberg data and metadata files to the bucket using your cluster’s existing object storage credentials, so Redpanda does not need additional IAM configuration for its own S3 access. You finish configuring this role’s trust policy when you create the catalog and the external volume, using values that Open Catalog and Snowflake generate for you.

Create an IAM policy

Create an IAM policy with the following S3 permissions, scoped to your cluster’s storage bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<bucket-name>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<bucket-name>"
    }
  ]
}

Replace <bucket-name> with the name of your cluster’s object storage bucket. You use this same IAM policy for both the Open Catalog role and the Snowflake external volume.

Create an IAM role

Create an IAM role and attach the IAM policy you created. Open Catalog and Snowflake each need this role’s ARN before they can generate the IAM user and external ID values you use to finish configuring the trust policy, so create the role with a temporary trust relationship for now:

  1. In the AWS IAM console, create a new role.

  2. For the trusted entity type, select AWS account. Under An AWS account, select This account.

  3. Attach the IAM policy you created.

  4. Note the role’s ARN (<iam-role-arn>). You update this role’s trust policy after Open Catalog and Snowflake generate the values you need, when you create the catalog and the external volume.

Create a catalog in Open Catalog

Create the catalog that Redpanda’s Iceberg data and metadata are registered to, with your Tiered Storage S3 bucket configured as external storage:

  1. In Open Catalog, create a new catalog.

  2. For S3 role ARN, enter <iam-role-arn>, the ARN of the IAM role you created.

  3. Configure the catalog’s external storage location to point to your Tiered Storage S3 bucket.

    Your Open Catalog account must be in the same AWS region as your S3 bucket.

  4. On the Open Catalog home page, in the Catalogs pane, select the catalog you created. Under Storage Details, copy the IAM user arn (<open-catalog-iam-user-arn>).

  5. If you didn’t specify an external ID when you created the IAM role, Open Catalog generates one for you (<open-catalog-external-id>). Record this value.

For complete steps, see the Open Catalog documentation.

Update the IAM role trust policy for Open Catalog

Update the trust policy for the IAM role you created, using the IAM user ARN and external ID that Open Catalog generated when you created the catalog:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<open-catalog-iam-user-arn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<open-catalog-external-id>"
        }
      }
    }
  ]
}

After you update the trust policy, Open Catalog can assume the role to read and write Iceberg data and metadata in your S3 bucket.

Create a Snowflake external volume

Create a Snowflake external volume using the same IAM role you created for Open Catalog. Snowflake provisions its own IAM user to assume the role, so you add a second statement to the role’s trust policy rather than replacing the statement that trusts Open Catalog:

  1. In Snowflake, run CREATE EXTERNAL VOLUME, pointing STORAGE_AWS_ROLE_ARN to the IAM role you created:

    CREATE OR REPLACE EXTERNAL VOLUME <iceberg-external-volume-name>
  STORAGE_LOCATIONS = (
    (
      NAME = '<storage-location-name>'
      STORAGE_PROVIDER = 'S3'
      STORAGE_BASE_URL = 's3://<bucket-name>/'
      STORAGE_AWS_ROLE_ARN = '<iam-role-arn>'
      STORAGE_AWS_EXTERNAL_ID = '<external-volume-external-id>'
    )
  )
  ALLOW_WRITES = TRUE;

    Use your own values for the following placeholders:

    • <iceberg-external-volume-name>: Provide a name for your external volume in Snowflake.

    • <storage-location-name>: Provide a name for the storage location.

    • <external-volume-external-id>: Choose an external ID for the volume’s trust relationship, distinct from the external ID Open Catalog uses. If you don’t set STORAGE_AWS_EXTERNAL_ID, Snowflake generates one for you.

  2. Run DESC EXTERNAL VOLUME to retrieve the IAM user ARN that Snowflake generated for the volume:

    DESC EXTERNAL VOLUME <iceberg-external-volume-name>;

    Record the STORAGE_AWS_IAM_USER_ARN value from the output (<external-volume-iam-user-arn>).

  3. Add a new statement to the IAM role’s trust policy for the volume’s IAM user ARN and external ID, alongside the existing statement that trusts Open Catalog:

    Add this as a new statement in the trust policy’s Statement array. If you replace the existing trust policy instead, Open Catalog loses access to the bucket. 
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<open-catalog-iam-user-arn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<open-catalog-external-id>"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<external-volume-iam-user-arn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<external-volume-external-id>"
        }
      }
    }
  ]
}

  4. Run SYSTEM$VERIFY_EXTERNAL_VOLUME to confirm Snowflake can access the bucket:

    SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME('<iceberg-external-volume-name>');

For complete steps, see the Snowflake documentation.

Set up catalog integration using Open Catalog

To integrate Iceberg-enabled topics with Open Catalog, create a service connection and configure catalog roles.

Create a new Open Catalog service connection for Redpanda

To create a new service connection to integrate the Iceberg-enabled topics into Open Catalog:

  1. In Open Catalog, select Connections, then + Connection.

  2. In Configure Service Connection, provide a name. Open Catalog creates a new principal with this name.

  3. Make sure Create new principal role is selected.

  4. Enter a name for the principal role. Then, click Create.

After you create the connection, get the client ID and client secret. Save these credentials to add to your cluster configuration in a later step.

Create a catalog role

Grant privileges to the principal created in the previous step:

  1. In Open Catalog, select Catalogs, and select your catalog.

  2. On the Roles tab of your catalog, click + Catalog Role.

  3. Give the catalog role a name.

  4. Under Privileges, select CATALOG_MANAGE_CONTENT. This provides full management privileges for the catalog. Then, click Create.

  5. On the Roles tab of the catalog, click Grant to Principal Role.

  6. Select the catalog role you just created.

  7. Select the principal role you created earlier. Click Grant.

Update cluster configuration

To configure your Redpanda cluster to enable Iceberg on a topic and integrate with Open Catalog:

  1. Store the Open Catalog client secret in your cluster using rpk or the Data Plane API.

  2. Edit your cluster configuration to set the iceberg_enabled property to true, and set the catalog integration properties listed in the example below using rpk or the Control Plane API. For example, to use rpk cluster config set, run:

    rpk cluster config set \
  iceberg_enabled=true \
  iceberg_catalog_type=rest \
  iceberg_rest_catalog_endpoint=https://<snowflake-orgname>-<open-catalog-account-name>.snowflakecomputing.com/polaris/api/catalog \
  iceberg_rest_catalog_authentication_mode=oauth2 \
  iceberg_rest_catalog_client_id=<open-catalog-connection-client-id> \
  iceberg_rest_catalog_client_secret='${secrets.<open-catalog-client-secret-name>}' \
  iceberg_rest_catalog_warehouse=<open-catalog-name>

# Optional properties:
# iceberg_translation_interval_ms_default=1000
# iceberg_catalog_commit_interval_ms=1000

    Use your own values for the following placeholders:

    • <snowflake-orgname> and <open-catalog-account-name>: Your Open Catalog account URI is composed of these values.

      In Snowflake, navigate to Admin, then Accounts. Click the ellipsis near your Open Catalog account name, and select Manage URLs. The Current URL contains <snowflake-orgname> and <open-catalog-account-name>.

    • <open-catalog-connection-client-id>: The client ID of the service connection you created in an earlier step.

    • <open-catalog-client-secret-name>: The name of the secret you created in the previous step. You must pass the secret name to the ${secrets.<secret-name>} placeholder, not the secret value itself.

    • <open-catalog-name>: The name of your catalog in Open Catalog.

    Successfully updated configuration. New configuration version is 2.

  3. Enable the integration for a topic by configuring the topic property redpanda.iceberg.mode. This mode creates an Iceberg table for the topic consisting of two columns: one for the record metadata including the key, and another binary column for the record’s value. See Enable Iceberg integration for more details on Iceberg modes.

    Use any of the following to set redpanda.iceberg.mode:

    • rpk. See the following examples to run rpk topic commands.

    • The Cloud UI. Navigate to Topics to create a new topic and specify redpanda.iceberg.mode in Additional Configuration, or edit an existing topic under the topic’s Configuration tab.

    • The Data Plane API to create a new topic or update a property for an existing topic. Specify the key-value pair for redpanda.iceberg.mode in the request body.

      The following examples show how to use rpk to create a new topic or alter the configuration for an existing topic, setting the Iceberg mode to key_value.

      Create a new topic and set redpanda.iceberg.mode:
      rpk topic create <topic-name> --topic-config=redpanda.iceberg.mode=key_value
      Set redpanda.iceberg.mode for an existing topic:
      rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=key_value

  4. Produce to the topic. For example,

    echo "hello world\nfoo bar\nbaz qux" | rpk topic produce <topic-name> --format='%k %v\n'

The topic appears as a table in Open Catalog.

  1. In Open Catalog, select Catalogs, then open your catalog.

  2. Under your catalog, you will see the redpanda namespace (or the namespace you configured with iceberg_default_catalog_namespace), and a table with the name of your topic. The namespace and the table are automatically added for you.

Query Iceberg table in Snowflake

To query the topic in Snowflake, you must create a catalog integration so that Snowflake has access to the table data and metadata.

Configure catalog integration with Snowflake

  1. Run the CREATE CATALOG INTEGRATION command in Snowflake:

    CREATE CATALOG INTEGRATION <catalog-integration-name>
  CATALOG_SOURCE = POLARIS
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'redpanda'
  REST_CONFIG = (
    CATALOG_URI = '<open-catalog-uri>'
    WAREHOUSE = '<open-catalog-name>'
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_CLIENT_ID = '<open-catalog-connection-client-id>'
    OAUTH_CLIENT_SECRET = '<open-catalog-connection-client-secret>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  REFRESH_INTERVAL_SECONDS = 30
  ENABLED = TRUE;

    Use your own values for the following placeholders:

  2. Run the following command to verify that the catalog is integrated correctly:

    SELECT SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('<catalog-integration-name>');
    # Example result for redpanda.iceberg.mode=key_value
+-----------------------------------------------------------------------+
| SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('<catalog_integration_name>') |
+-----------------------------------------------------------------------+
| [{"namespace":"redpanda","name":"<table_name>"}]                      |
+-----------------------------------------------------------------------+

Create Iceberg table in Snowflake

After creating the catalog integration, you must create an externally-managed table in Snowflake. You must run your Snowflake queries against this table.

In your Snowflake database, run the CREATE ICEBERG TABLE command. The following example also specifies that the table should automatically refresh metadata:

CREATE ICEBERG TABLE <table-name>
  CATALOG = '<catalog-integration-name>'
  EXTERNAL_VOLUME = '<iceberg-external-volume-name>'
  CATALOG_TABLE_NAME = '<topic-name>'
  AUTO_REFRESH = TRUE

Use your own values for the following placeholders:

  • <table-name>: Provide a name for your table in Snowflake.

  • <catalog-integration-name>: The name of the catalog integration you configured in an earlier step.

  • <iceberg-external-volume-name>: The name of the external volume you configured using the Tiered Storage bucket.

  • <topic-name>: The name of the table in your catalog, which is the same as your Redpanda topic name.

Query the Iceberg table

To verify that Snowflake has successfully created the table containing the topic data, run the following:

SELECT * FROM <table-name>;

Your query results should look like the following:

# Example for redpanda.iceberg.mode=key_value with 3 records produced to topic

+--------------------------------------------------------------------------------------------------------------+------------+
|                                                   REDPANDA                                                   |   VALUE    |
+--------------------------------------------------------------------------------------------------------------+------------+
| { "partition": 0, "offset": 0, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "68656C6C6F"} | 776F726C64 |
| { "partition": 0, "offset": 1, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "666F6F"}     | 626172     |
| { "partition": 0, "offset": 2, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "62617A" }    | 717578     |
+--------------------------------------------------------------------------------------------------------------+------------+

Manage access for query engine users

Redpanda manages the permissions between Redpanda and Open Catalog. To grant your Snowflake users or other query engines read access to the Iceberg tables, use Open Catalog access control to assign catalog privileges. For example, you can grant TABLE_READ_DATA to a read-only role rather than the CATALOG_MANAGE_CONTENT privilege used by the Redpanda service principal.

Suggested reading