Connect
Also in Cloud

gcp_cloud_storage

Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix.

Introduced in version 3.43.0.

Metadata

This input adds the following metadata fields to each message:

  • gcs_key

  • gcs_bucket

  • gcs_last_modified

  • gcs_last_modified_unix

  • gcs_content_type

  • gcs_content_encoding

  • All user defined metadata

You can access these metadata fields using function interpolation.

Credentials

By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in Google Cloud Platform.

  • Common

  • Advanced

input:
  label: ""
  gcp_cloud_storage:
    bucket: "" # No default (required)
    prefix: ""
    credentials_json: ""
    scanner:
      to_the_end: {}
input:
  label: ""
  gcp_cloud_storage:
    bucket: "" # No default (required)
    prefix: ""
    credentials_json: ""
    scanner:
      to_the_end: {}
    delete_objects: false

Fields

bucket

The name of the bucket from which to download objects.

Type: string

credentials_json

The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses Application Default Credentials. For more information about creating service account credentials, see Google’s service account documentation.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Requires version 4.33.0 or later.

Type: string

Default: ""

delete_objects

Whether to remove objects from the Cloud Storage bucket once the messages read from them are delivered without error. An object is kept if delivery fails.

Type: bool

Default: false

prefix

Optional path prefix, if set only objects with the prefix are consumed.

Type: string

Default: ""

scanner

The scanner used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the csv scanner processes individual rows in a CSV file without loading the entire file in memory.

Requires version 4.25.0 or later.

Type: scanner

Default:

to_the_end: {}