Cloud

Allow Pipeline Traffic Through External Firewalls

Many managed services, such as Snowflake, MongoDB Atlas, and managed Elasticsearch, reject connections from IP addresses that aren’t on their network allowlist. Add your cluster’s public egress addresses to that allowlist so your pipelines can reach the service. This page applies to BYOC clusters. For Dedicated or Serverless clusters, contact Redpanda support for the addresses that your pipelines connect from.

After reading this page, you will be able to:

  • Identify the public IP addresses that pipelines on a BYOC cluster connect from

  • Add those addresses to the network allowlist of an external service

  • Confirm which address a running pipeline connects from

This page is about the firewall on the external service. To let pipelines reach private destinations that your cluster’s own firewall blocks, such as a database in a peered network, see Configure Egress for Redpanda Connect Pipelines.

Prerequisites

  • A BYOC cluster on AWS, GCP, or Azure.

  • Access to the cloud account or project where the cluster runs, to look up its network resources.

  • Permission to change the network allowlist of the external service.

How your pipelines reach the internet

Pipelines on a BYOC cluster reach public endpoints through network address translation (NAT) in the cluster’s network, so an external service sees the NAT’s public address instead of the pipeline’s own address. Which addresses you allowlist depends on how the cluster was set up:

Cluster setup Public egress addresses

AWS, default egress

One NAT gateway, each with its own Elastic IP address, in every availability zone the cluster uses. A multi-AZ cluster has up to three addresses. Pipelines can run in any of these zones, so allowlist all of them.

AWS, customer-managed VPC

The public addresses of the NAT gateway or other egress path in the VPC that you provide. Redpanda creates no NAT gateways in a customer-managed VPC, so you look up and manage these addresses yourself.

GCP, default egress

One reserved static IP address on the cluster’s Cloud NAT gateway.

Azure, default egress

One NAT gateway in every availability zone the cluster uses, each with a public IP prefix that covers two addresses (a /31). Allowlist every prefix.

Centralized egress on AWS, GCP, or Azure

The public addresses of the NAT or firewall in your hub network, which you manage. See Configure Centralized Egress with AWS Transit Gateway for AWS, Configure Centralized Egress with GCP VPC Peering for GCP, and Configure Centralized Egress with Azure Firewall for Azure.

These addresses stay the same for the life of the cluster unless its network is rebuilt.

Find the egress addresses

  • AWS

  • GCP

  • Azure

  • Centralized egress or customer-managed VPC

  1. In the AWS console, open the VPC service in the account and region where the cluster runs.

  2. Select NAT gateways and filter by the VPC that the cluster uses.

  3. Note the Primary public IPv4 address of every NAT gateway in that VPC.

  1. In the Google Cloud console, open Network services > Cloud NAT in the project and region where the cluster runs.

  2. Select the NAT gateway for the cluster’s VPC. It is named after the VPC and region.

  3. Note the reserved IP address listed under NAT IP addresses.

  1. In the Azure portal, open the resource group that holds the cluster’s network.

  2. Open each NAT gateway for the cluster. They are named ngw-<prefix>-<region>-<network-id>-<zone>, one per availability zone.

  3. Under Outbound IP, note the public IP prefix of each gateway. Each prefix is a /31 that covers two addresses.

Use the public addresses of the NAT gateway, firewall, or appliance in your hub network, or in the VPC that you provide.

Add the addresses to the external service

Add every address you found to the network allowlist of the service, as a single-address CIDR block (for example, 203.0.113.10/32). For an Azure public IP prefix, add the /31 prefix as shown in the portal. The setting is usually called an IP access list, network policy, or allowlist. For example:

  • Snowflake: create or update a network policy that allows the addresses.

  • MongoDB Atlas: add the addresses to the project’s IP access list.

After the service accepts the change, restart any pipeline that failed to connect.

Confirm the address a pipeline uses

To check which address a running pipeline connects from, deploy a short-lived pipeline that asks a public echo service for its source address and logs the answer:

input:
  generate:
    count: 1
    mapping: root = ""

pipeline:
  processors:
    - http:
        url: https://checkip.amazonaws.com
        verb: GET
    - log:
        message: 'Pipeline egress address: ${! content().string().trim() }'

output:
  drop: {}

The pipeline logs the address once and stops. On a multi-zone AWS or Azure cluster, this shows only the address of the zone where the pipeline ran, so use the console steps above to find the full list. Delete the pipeline when you’re done.

Next steps