Cloud
Also in Self-Managed

parquet_decode

Decodes Parquet files into a batch of structured messages.

This processor uses https://github.com/parquet-go/parquet-go, which is itself experimental. As a result, how this processor functions can change outside of major version releases.

  • Common

  • Advanced

processor:
  label: ""
  parquet_decode:
    handle_logical_types: v1
processor:
  label: ""
  parquet_decode:
    handle_logical_types: v1

Fields

handle_logical_types

Set to v2 to enable enhanced decoding of logical types, or keep the default value (v1) to ignore logical type metadata when decoding values.

In Parquet format, logical types are represented using standard physical types along with metadata that provides additional context. For example, UUIDs are stored as a FIXED_LEN_BYTE_ARRAY physical type, but the schema metadata identifies them as UUIDs. By enabling v2, this processor uses the metadata descriptions of logical types to produce more meaningful values during decoding.

Each version enables the logical type handling for that version and all earlier versions, which allows handling of new logical types to be introduced without breaking existing pipelines. When creating new pipelines, Redpanda recommends that you use the newest available version.

Type: string

Default: v1

Option Summary

v1

No special handling of logical types

v2

  • TIMESTAMP - decodes as an RFC3339 string describing the time. If the isAdjustedToUTC flag is set to true in the parquet file, the time zone will be set to UTC. If it is set to false the time zone will be set to local time.

  • UUID - decodes as a string, i.e. 00112233-4455-6677-8899-aabbccddeeff.

 
# Examples:
handle_logical_types: v2

Examples

Reading Parquet Files from AWS S3

In this example we consume files from AWS S3 as they’re written by listening onto an SQS queue for upload events. We make sure to use the to_the_end scanner which means files are read into memory in full, which then allows us to use a parquet_decode processor to expand each file into a batch of messages. Finally, we write the data out to local files as newline delimited JSON.

input:
  aws_s3:
    bucket: TODO
    prefix: foos/
    scanner:
      to_the_end: {}
    sqs:
      url: TODO
  processors:
    - parquet_decode: {}

output:
  file:
    codec: lines
    path: './foos/${! meta("s3_key") }.jsonl'