ollama_chat
| Ollama connectors are currently only available on BYOC GCP clusters. |
|
When Redpanda Connect runs a data pipeline with a Ollama processor in it, Redpanda Cloud deploys a GPU-powered instance for the exclusive use of that pipeline. As pricing is based on resource consumption, this can have cost implications. |
Generates responses to messages in a chat conversation using the Ollama API and external tools.
| This component is available only in GPU-enabled pipelines. |
-
Common
-
Advanced
processor:
label: ""
ollama_chat:
model: "" # No default (required)
prompt: "" # No default (optional)
image: "" # No default (optional)
response_format: text
max_tokens: 0 # No default (optional)
temperature: 0 # No default (optional)
save_prompt_metadata: false
history: "" # No default (optional)
tools: []
runner:
context_size: 0 # No default (optional)
batch_size: 0 # No default (optional)
server_address: "" # No default (optional)processor:
label: ""
ollama_chat:
model: "" # No default (required)
prompt: "" # No default (optional)
system_prompt: "" # No default (optional)
image: "" # No default (optional)
response_format: text
max_tokens: 0 # No default (optional)
temperature: 0 # No default (optional)
num_keep: 0 # No default (optional)
seed: 0 # No default (optional)
top_k: 0 # No default (optional)
top_p: 0 # No default (optional)
repeat_penalty: 0 # No default (optional)
presence_penalty: 0 # No default (optional)
frequency_penalty: 0 # No default (optional)
stop: [] # No default (optional)
save_prompt_metadata: false
history: "" # No default (optional)
max_tool_calls: 3
tools: []
runner:
context_size: 0 # No default (optional)
batch_size: 0 # No default (optional)
gpu_layers: 0 # No default (optional)
threads: 0 # No default (optional)
use_mmap: false # No default (optional)
server_address: "" # No default (optional)
cache_directory: "" # No default (optional)
download_url: "" # No default (optional)This processor sends prompts to your chosen Ollama large language model (LLM) and generates text from the responses using the Ollama API and external tools.
By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the server_address field. You can download and install Ollama from the Ollama website.
For more information, see the Ollama documentation and examples.
Fields
cache_directory
If server_address is not set, download the Ollama binary to this directory and use it as a model cache.
Type: string
# Examples:
cache_directory: /opt/cache/connect/ollamadownload_url
If server_address is not set, download the Ollama binary from this URL. The default value is the official Ollama GitHub release for this platform.
Type: string
frequency_penalty
A number between -2.0 and 2.0 that the processor sends to the Ollama server as the frequency_penalty model option. Positive values penalize each token in proportion to how often it has already appeared in the text, which makes the model less likely to repeat the same line verbatim. A value of 0 is not sent, so the server’s default applies.
Type: float
history
Include historical messages in a chat request. You must use a Bloblang query to create an array of objects in the form of [{"role": "", "content":""}] where:
-
roleis the sender of the original messages, eithersystem,user,assistant, ortool. -
contentis the text of the original messages.
Type: string
image
An optional image to submit along with the prompt value. The result should be a byte array.
Type: string
# Examples:
image: 'root = this.image.decode("base64") # decode base64 encoded image'max_tokens
The maximum number of tokens to predict and output. Limiting the amount of output means that requests are processed faster and have a fixed limit on the cost.
Type: int
max_tool_calls
The maximum number of sequential calls you can make to external tools to retrieve additional information to answer a prompt.
Type: int
Default: 3
model
The name of the Ollama model to use. For a full list of models, see the Ollama website.
Type: string
# Examples:
model: llama3.1
# ---
model: gemma2
# ---
model: qwen2
# ---
model: phi3num_keep
Specify the number of tokens from the initial prompt to retain when the model resets its internal context. By default, this value is set to 4. Use -1 to retain all tokens from the initial prompt.
Type: int
presence_penalty
A number between -2.0 and 2.0 that the processor sends to the Ollama server as the presence_penalty model option. Positive values penalize every token that has already appeared in the text, regardless of how often, which makes the model more likely to move on to new topics. A value of 0 is not sent, so the server’s default applies.
Type: float
prompt
The user prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string.
This field supports interpolation functions.
Type: string
repeat_penalty
Sets how strongly to penalize repetitions. A higher value, for example 1.5, will penalize repetitions more strongly. A lower value, for example 0.9, will be more lenient.
Type: float
response_format
The format of the response the Ollama model generates. If specifying JSON output, then the prompt should specify that the output should be in JSON as well.
Type: string
Default: text
Options: text, json
runner
Options for the model runner that are used when the model is first loaded into memory.
Type: object
runner.context_size
Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process.
Type: int
runner.gpu_layers
Sets the number of layers to offload to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically.
Type: int
runner.threads
Sets the number of threads to use during response generation. For optimal performance, set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads.
Type: int
runner.use_mmap
Map the model into memory. Set to true to load only the necessary parts of the model into memory. This setting is only supported on Unix systems.
Type: bool
save_prompt_metadata
Set to true to save the prompt value to a metadata field (@prompt) on the corresponding output message. If you use the system_prompt field, its value is also saved to an @system_prompt metadata field on each output message.
Type: bool
Default: false
seed
Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt.
Type: int
# Examples:
seed: 42server_address
The address of the Ollama server to use. Leave this field blank and the processor starts and runs a local Ollama server, or specify the address of your own local or remote server.
Type: string
# Examples:
server_address: http://127.0.0.1:11434stop[]
Sets the stop sequences to use. When this pattern is encountered, the LLM stops generating text and returns the final response.
Type: array<string>
system_prompt
The system prompt to submit along with the user prompt.
This field supports interpolation functions.
Type: string
temperature
The temperature of the model. Increasing the temperature makes the model answer more creatively.
Type: int
tools[]
The external tools the LLM can invoke, such as functions, APIs, or web browsing. You can build subpipelines of processors that include definitions of these tools, and the specified LLM can choose when to invoke them to help answer a prompt.
Type: array<object>
Default: []
tools[].description
A description of this tool. The LLM uses it to decide whether to invoke the tool.
Type: string
tools[].parameters.properties
The parameters the LLM can provide when it invokes this tool, keyed by parameter name.
Type: object
tools[].parameters.properties.enum[]
The values this parameter is limited to. Leave empty to accept any value.
Type: array<string>
Default: []
tools[].parameters.required[]
The names of the parameters the LLM must provide when it invokes this tool.
Type: array<string>
Default: []
tools[].processors[]
The processors to run when the LLM invokes this tool. They receive a message whose payload is the tool call arguments as a JSON object, and their output is returned to the LLM as the tool result.
Type: array<processor>
Examples
Use Llava to analyze an image
This example fetches image URLs from stdin and has a multimodal LLM describe the image.
input:
stdin:
scanner:
lines: {}
pipeline:
processors:
- http:
verb: GET
url: "${!content().string()}"
- ollama_chat:
model: llava
prompt: "Describe the following image"
image: "root = content()"
output:
stdout:
codec: linesUse subpipelines as tool calls
This example allows llama3.2 to execute a subpipeline as a tool call to get more data.
input:
generate:
count: 1
mapping: |
root = "What is the weather like in Chicago?"
pipeline:
processors:
- ollama_chat:
model: llama3.2
prompt: "${!content().string()}"
tools:
- name: GetWeather
description: "Retrieve the weather for a specific city"
parameters:
required: ["city"]
properties:
city:
type: string
description: the city to lookup the weather for
processors:
- http:
verb: GET
url: 'https://wttr.in/${!this.city}?T'
headers:
# Spoof curl user-ageent to get a plaintext text
User-Agent: curl/8.11.1
output:
stdout: {}