Query Iceberg Topics using Snowflake and Open Catalog

In the AWS IAM console, create a new role.

For the trusted entity type, select AWS account. Under An AWS account, select This account.

Attach the IAM policy you created.

Note the role’s ARN (<iam-role-arn>). You update this role’s trust policy after Open Catalog and Snowflake generate the values you need, when you create the catalog and the external volume.

In Open Catalog, select Connections, then + Connection.

In Configure Service Connection, provide a name. Open Catalog creates a new principal with this name.

Make sure Create new principal role is selected.

Enter a name for the principal role. Then, click Create.

In Open Catalog, select Catalogs, and select your catalog.

On the Roles tab of your catalog, click + Catalog Role.

Give the catalog role a name.

Under Privileges, select CATALOG_MANAGE_CONTENT. This provides full management privileges for the catalog. Then, click Create.

On the Roles tab of the catalog, click Grant to Principal Role.

Select the catalog role you just created.

Select the principal role you created earlier. Click Grant.

Edit your cluster configuration to set the iceberg_enabled property to true, and set the catalog integration properties listed in the example below. You must restart your cluster if you change this configuration for a running cluster. You can run rpk cluster config edit to update these properties:

iceberg_enabled: true
iceberg_catalog_type: rest
iceberg_rest_catalog_endpoint: https://<snowflake-orgname>-<open-catalog-account-name>.snowflakecomputing.com/polaris/api/catalog
iceberg_rest_catalog_authentication_mode: oauth2
iceberg_rest_catalog_client_id: <open-catalog-connection-client-id>
iceberg_rest_catalog_client_secret: <open-catalog-connection-client-secret>
iceberg_rest_catalog_warehouse: <open-catalog-name>

# Optional
iceberg_translation_interval_ms_default: 1000
iceberg_catalog_commit_interval_ms: 1000

You must restart your cluster so that the configuration changes take effect.

Enable the integration for a topic by configuring the topic property redpanda.iceberg.mode. This mode creates an Iceberg table for the topic consisting of two columns: one for the record metadata including the key, and another binary column for the record’s value. See Enable Iceberg integration for more details on Iceberg modes.

rpk topic create <topic-name> --topic-config=redpanda.iceberg.mode=key_value

rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=key_value

Produce to the topic. For example,

echo "hello world\nfoo bar\nbaz qux" | rpk topic produce <topic-name> --format='%k %v\n'

In Open Catalog, select Catalogs, then open your catalog.

Under your catalog, you will see the redpanda namespace (or the namespace you configured with iceberg_default_catalog_namespace), and a table with the name of your topic. The namespace and the table are automatically added for you.

Run the CREATE CATALOG INTEGRATION command in Snowflake:

CREATE CATALOG INTEGRATION <catalog-integration-name>
  CATALOG_SOURCE = POLARIS
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'redpanda'
  REST_CONFIG = (
    CATALOG_URI = '<open-catalog-uri>'
    WAREHOUSE = '<open-catalog-name>'
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_CLIENT_ID = '<open-catalog-connection-client-id>'
    OAUTH_CLIENT_SECRET = '<open-catalog-connection-client-secret>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  REFRESH_INTERVAL_SECONDS = 30
  ENABLED = TRUE;

Run the following command to verify that the catalog is integrated correctly:

SELECT SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('<catalog-integration-name>');

# Example result for redpanda.iceberg.mode=key_value
+-----------------------------------------------------------------------+
| SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('<catalog_integration_name>') |
+-----------------------------------------------------------------------+
| [{"namespace":"redpanda","name":"<table_name>"}]                      |
+-----------------------------------------------------------------------+