๐ŸŽ New User? Get 20% off your first purchase with code NEWUSER20 ยท โšก Instant download ยท ๐Ÿ”’ Secure checkout Register Now โ†’
Menu

Categories

Running Ollama Models via REST API in Production Scripts

Running Ollama Models via REST API in Production Scripts
You run Ollama for model inference on your own hardware. The CLI works for quick tests, but production scripts need deterministic timeouts, structured output, and error codes. The ollama rest api gives you that control. This guide walks through the endpoints, request/response formats, error handling, and performance settings that matter when you integrate Ollama into cron jobs, CI pipelines, or monitoring workflows. By the end you will know how to call each endpoint with curl or Python, tune timeouts and context windows, handle streaming and blocking responses, and expose the API safely when automation runs on separate hosts. All examples assume Ollama 0.1.x or later; older builds may lack certain parameters.

Ollama API Basics: Endpoints and Authentication

Ollama listens on localhost:11434 by default. Change the bind address with the OLLAMA_HOST environment variable. For a systemd service, add Environment="OLLAMA_HOST=0.0.0.0:11434" to the service unit file under [Service], then systemctl daemon-reload && systemctl restart ollama. In Docker, map the port and set -e OLLAMA_HOST=0.0.0.0:11434. Binding to 0.0.0.0 exposes the API to your network; use this only behind a firewall or reverse proxy.

The main endpoints are /api/tags (list installed models), /api/generate (single completion), /api/chat (multi-turn conversations), and /api/embeddings (vector representations). Ollama has no built-in authentication. It is a local service meant to run on the same host as your scripts. If you must expose it remotely, front it with nginx or Traefik, enforce TLS, and add HTTP basic auth or header-based tokens. A minimal nginx location block:

location /ollama/ {
    proxy_pass http://127.0.0.1:11434/;
    proxy_set_header Authorization $http_authorization;
    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.htpasswd;
}

List available models with:

curl -s http://localhost:11434/api/tags | jq -r '.models[].name'

You will see names like llama3.2:3b or mistral:7b. Use these exact strings in the model field of subsequent requests. The Ollama API documentation lists every parameter; this guide focuses on the subset you actually tune in production.

Generating Completions with /api/generate

POST JSON to /api/generate with at minimum model and prompt. Set stream to false for a single blocking response or true for newline-delimited JSON chunks. The options object takes temperature (0.0โ€“2.0, lower is more deterministic), num_ctx (context window size in tokens), and num_predict (max tokens to generate; -1 means no limit beyond the model's max).

Request Format and Key Parameters

Blocking request:

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:3b",
    "prompt": "Explain TCP window scaling in one sentence.",
    "stream": false,
    "options": {
      "temperature": 0.3,
      "num_predict": 50
    }
  }'

Response is a single JSON object with response (the generated text), done (true), total_duration and load_duration in nanoseconds, and prompt_eval_count / eval_count for token counts. Extract the text with jq -r .response.

Streaming request returns one JSON object per line. Each line has response (a token or fragment), done (false until the last line), and cumulative timing. Use curl --no-buffer to see output immediately:

curl --no-buffer -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2:3b","prompt":"List three Linux syscalls.","stream":true}'

Each line is valid JSON; concatenate response fields to build the full answer. Use blocking mode in cron jobs or CI steps where you wait for the result. Use streaming when you feed output to a user interface or log partial results as they arrive.

Parsing the JSON Response

Python example with timeout and error handling:

import requests
import json

url = "http://localhost:11434/api/generate"
payload = {
    "model": "mistral:7b",
    "prompt": "What is the default TCP keepalive interval on Linux?",
    "stream": False,
    "options": {"temperature": 0.2}
}

try:
    r = requests.post(url, json=payload, timeout=30)
    r.raise_for_status()
    print(r.json()["response"])
except requests.exceptions.Timeout:
    print("Request timed out after 30s")
except requests.exceptions.ConnectionError:
    print("Cannot reach Ollama; is the service running?")
except requests.exceptions.HTTPError as e:
    print(f"HTTP {e.response.status_code}: {e.response.text}")

Set timeout to twice your expected inference time. A 3B model on CPU might take 10โ€“20 seconds for 100 tokens; a 13B model can exceed a minute. If Ollama runs out of memory mid-generation, it kills the request and you get a broken connection; the Python snippet catches that as ConnectionError. For more on HTTP request patterns see API Basics: REST & JSON Explained.

Chat Format Requests with /api/chat

The /api/chat endpoint takes a messages array instead of a single prompt. Each message has role (system, user, or assistant) and content. Use this for multi-turn conversations or when you want a system prompt that sets behavior separate from the user input. The stream and options parameters work identically to /api/generate.

curl -X POST http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:3b",
    "messages": [
      {"role": "system", "content": "You are a log parser. Output only JSON."},
      {"role": "user", "content": "Parse: 2024-01-15 ERROR database timeout"}
    ],
    "stream": false
  }'

The response JSON contains message with role and content, plus the same timing fields. If you send conversation history, append each user and assistant turn to the messages array in chronological order. Ollama re-evaluates the entire context each call (it does not persist state), so long histories increase latency and token usage.

Prefer /api/chat when you define output format or tone in a system message and want to keep that instruction separate from variable user input. For single-shot prompts with no conversation, /api/generate is simpler and skips the role wrapping. A practical use case: your monitoring script sends error snippets to Ollama with a system prompt that enforces structured JSON output, then parses that JSON to file a ticket with severity and component fields extracted by the model.

Embeddings and Utility Endpoints

The /api/embeddings endpoint returns a vector representation of your input text. POST JSON with model and prompt; the response contains an embedding array of floats. Use this for semantic search, clustering log messages, or finding similar incidents in a knowledge base.

curl -X POST http://localhost:11434/api/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"nomic-embed-text","prompt":"kernel panic on boot"}' \
  | jq -r '.embedding | length'

The nomic-embed-text model returns 768-dimensional vectors. Store these in a vector database like pgvector in PostgreSQL or use cosine similarity in your script to rank results. For details on indexing and querying vectors see PostgreSQL Administration & Performance Tuning.

The /api/pull endpoint downloads a model. Use it in provisioning scripts to ensure a model is present before your automation runs:

curl -X POST http://localhost:11434/api/pull \
  -d '{"name":"llama3.2:3b","stream":false}'

Check running models and memory usage with /api/ps:

curl -s http://localhost:11434/api/ps | jq '.models[] | {name, size_vram}'

This shows which models are loaded and how much VRAM each consumes. Use /api/show to retrieve a model's Modelfile and parameters:

curl -X POST http://localhost:11434/api/show \
  -d '{"name":"mistral:7b"}' | jq -r '.parameters'

These endpoints are less common in day-to-day inference scripts but valuable for orchestration, health checks, and automated model management in CI/CD pipelines.

Error Handling and Timeout Tuning

Ollama returns HTTP 404 if the model is not found, 500 on out-of-memory or model load failures. It does not provide structured error codes; check the response body for strings like out of memory or model not found. When Ollama crashes due to OOM, systemd restarts it (if you run it as a service), but your script must detect the failure and retry after a delay.

Common API Errors and HTTP Status Codes

A 404 with "error":"model 'typo:7b' not found" means you requested a model that is not pulled. Validate model names against /api/tags before making requests. A 500 with out of memory in the body means Ollama tried to load the model or generate tokens and exhausted available RAM or VRAM. Lower num_ctx or switch to a smaller model. If the connection drops mid-request, Ollama likely crashed; your script sees a connection reset or incomplete response.

Example retry logic in bash with exponential backoff:

attempt=0
max_attempts=3
while [ $attempt -lt $max_attempts ]; do
  if curl -f -s --max-time 120 -X POST http://localhost:11434/api/generate \
       -d '{"model":"llama3.2:3b","prompt":"test","stream":false}' > /tmp/result.json; then
    break
  fi
  attempt=$((attempt + 1))
  sleep $((2 ** attempt))
done

The -f flag makes curl return non-zero on HTTP errors; --max-time 120 sets a total timeout of two minutes. Adjust based on your model size and hardware.

Setting Request and Inference Timeouts

Ollama can take 30 seconds or more to return the first token on CPU, especially for larger models. Set curl --max-time or the timeout parameter in your HTTP library higher than you think necessary; 120 seconds is safer for 7B+ models on CPU. The curl man page explains --max-time vs --connect-timeout; use --max-time to cover the entire request duration.

Set num_predict in options to limit generated tokens and prevent runaway generation. A value of 200 caps output at roughly 200 tokens, stopping the model early. This bounds both latency and memory usage. If you hit repeated timeouts, check Ollama logs (journalctl -u ollama if running as systemd) for load errors or hardware limits. For more on tuning inference performance see Ollama Production Hardening: Security and Performance Audit.

Performance: Model Context and Concurrency Limits

The num_ctx option sets the context window in tokens; many models default to 2048. Larger context windows let the model see more prompt or conversation history but consume more VRAM or RAM. If you hit CUDA out of memory errors, lower num_ctx to 1024 or 512. A 7B model with 4096 context can require 12GB+ of VRAM depending on quantization.

num_ctx and GPU Memory Trade-offs

Example request with reduced context:

curl -X POST http://localhost:11434/api/generate \
  -d '{
    "model": "mistral:7b",
    "prompt": "Summarize this log.",
    "stream": false,
    "options": {"num_ctx": 1024}
  }'

Check current VRAM usage with nvidia-smi or /api/ps. If you run multiple models or large batches, you may exhaust memory even with a modest context. The OLLAMA_MAX_LOADED_MODELS environment variable controls how many models Ollama keeps in memory simultaneously; the default is 1. Set it higher if you have enough VRAM and want to avoid reload delays:

Environment="OLLAMA_MAX_LOADED_MODELS=2"

in your systemd unit, then restart. Each loaded model holds VRAM until evicted, so only raise this if you frequently switch between a small set of models and have 16GB+ VRAM.

Handling Concurrent Requests

Ollama queues requests; only one inference runs per model at a time by default. If you send three requests to the same model in parallel, the second and third wait for the first to finish. This serialization prevents memory thrashing but limits throughput. If you need concurrency, run multiple Ollama instances on different ports (each with OLLAMA_HOST set to a unique bind) or load multiple copies of the model (expensive in VRAM). For most scripts, serialize requests in your code or use a job queue like Redis or a simple lock file. Running two 7B models concurrently requires double the VRAM; plan hardware accordingly. For broader performance tuning strategies see Linux Performance Tuning.

Integrating the API into CI/CD and Monitoring

Production scripts call the Ollama API to analyze logs, validate commit messages, or generate summaries for alerting. Set OLLAMA_HOST in your environment if Ollama runs on a separate server. Always use curl --fail so non-2xx HTTP responses return a non-zero exit code, stopping your pipeline or cron job on errors.

Example: Commit Message Analysis in Jenkins

A Jenkins pipeline step that checks whether a commit message references a ticket ID:

pipeline {
  agent any
  environment {
    OLLAMA_HOST = 'http://ollama-server:11434'
  }
  stages {
    stage('Validate Commit') {
      steps {
        script {
          def commit_msg = sh(script: 'git log -1 --pretty=%B', returnStdout: true).trim()
          def prompt = "Does this commit message mention a JIRA ticket (e.g., PROJ-1234)? Answer yes or no only. Message: ${commit_msg}"
          def response = sh(
            script: """
              curl --fail -s --max-time 60 -X POST \${OLLAMA_HOST}/api/generate \\
                -H 'Content-Type: application/json' \\
                -d '{"model":"llama3.2:3b","prompt":"${prompt}","stream":false,"options":{"temperature":0.1}}' \\
                | jq -r .response
            """,
            returnStdout: true
          ).trim().toLowerCase()
          if (!response.contains('yes')) {
            error("Commit message does not reference a ticket ID")
          }
        }
      }
    }
  }
}

This fails the build if the model does not detect a ticket reference. Set temperature low for deterministic yes/no answers. Escape quotes in the prompt or pass it via a file to avoid shell quoting issues in real pipelines.

Example: Log Anomaly Detection in a Cron Job

A bash script that runs every hour, summarizes recent syslog entries, and posts the summary to Slack:

#!/bin/bash
set -euo pipefail

OLLAMA_HOST="${OLLAMA_HOST:-http://localhost:11434}"
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

# Tail last 100 lines, filter ERRORs
LOGS=$(tail -100 /var/log/syslog | grep -i error || true)

if [ -z "$LOGS" ]; then
  exit 0
fi

PROMPT="Summarize the following syslog errors in two sentences. Mention any critical services or repeated issues:\n\n$LOGS"

SUMMARY=$(curl --fail -s --max-time 120 -X POST "$OLLAMA_HOST/api/generate" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"mistral:7b\",\"prompt\":$(jq -n --arg p "$PROMPT" '$p'),\"stream\":false,\"options\":{\"temperature\":0.3,\"num_predict\":100}}" \
  | jq -r .response) || {
    echo "Ollama request failed" >&2
    exit 1
  }

curl -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d "{\"text\":\"Syslog anomaly summary:\n$SUMMARY\"}"

Use jq -n --arg to safely embed multi-line log text in JSON. The || { ... } block catches curl failures and prevents the script from posting garbage to Slack. Adjust the grep pattern and log path to match your system. For more robust log parsing consider structured logs and journalctl filters.

Security and Network Exposure Considerations

Ollama binds to localhost:11434 by default, which is safe; only processes on the same host can connect. If you set OLLAMA_HOST=0.0.0.0:11434, anyone on your network can send inference requests and consume your GPU or CPU. Never expose Ollama directly to the internet; it has no authentication, rate limiting, or input validation beyond basic JSON parsing.

For remote access from trusted clients, front Ollama with nginx and enforce HTTP basic auth plus TLS. Example nginx configuration:

server {
  listen 443 ssl;
  server_name ollama.internal.example.com;

  ssl_certificate /etc/nginx/ssl/ollama.crt;
  ssl_certificate_key /etc/nginx/ssl/ollama.key;

  location / {
    proxy_pass http://127.0.0.1:11434;
    proxy_set_header Host $host;
    proxy_read_timeout 180s;

    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.htpasswd;

    limit_req zone=ollama_limit burst=5 nodelay;
  }
}

http {
  limit_req_zone $binary_remote_addr zone=ollama_limit:10m rate=10r/m;
}

Generate .htpasswd with htpasswd -c /etc/nginx/.htpasswd username. The limit_req directive caps clients to 10 requests per minute with a burst of 5, preventing a single user from monopolizing the GPU. Set proxy_read_timeout higher than your longest expected inference. For stricter isolation, use Tailscale or an SSH tunnel instead of exposing nginx publicly. If you run Ollama in Docker, do not publish port 11434 on 0.0.0.0; bind it to 127.0.0.1 or omit the port mapping and use Docker networks to connect containers. For a complete self-hosted setup with Traefik and TLS see Rocky Linux 9 + Docker + Traefik: Self-Hosted Platform.

Checklist for Production Ollama API Integration

  • Validate model names against /api/tags before making requests; catch 404 errors if a model is missing.
  • Set --max-time or timeout to at least twice your expected inference duration; 120 seconds for 7B models on CPU.
  • Use stream: false in CI/CD steps and cron jobs where you need the full response before proceeding; use stream: true for user-facing applications.
  • Lower num_ctx to 1024 or 512 if you hit out-of-memory errors; monitor VRAM with nvidia-smi or /api/ps.
  • Set num_predict to cap generated tokens and prevent runaway inference; 200 is a safe limit for summaries.
  • Run Ollama behind nginx with HTTP basic auth and TLS if you expose it beyond localhost; never bind to 0.0.0.0 on the public internet.
  • Implement retry logic with exponential backoff for transient failures; log errors to journalctl -u ollama for debugging.
  • Serialize requests in your scripts or use a job queue if concurrency is low; running multiple models concurrently requires proportional VRAM.
  • For remote access from automation, prefer Tailscale or SSH tunnels over public nginx endpoints.

If you are new to REST API integration patterns, Mastering CURL: The Complete Guide to Command-Line HTTP Requests covers timeouts, retries, and authentication in depth. For broader context on running Ollama in production see Ollama Fundamentals.

Frequently Asked Questions

Does the Ollama REST API require authentication?

No, Ollama has no built-in authentication. It is designed to run on localhost. If you expose it remotely, front it with nginx or Traefik and add HTTP basic auth, API key headers, or mutual TLS. Never bind Ollama to 0.0.0.0 on a public-facing server without a reverse proxy enforcing access control.

How do I increase the timeout for slow model inference?

In curl, use --max-time 120 or higher to set a total request timeout in seconds. In Python requests.post(..., timeout=120). Adjust based on model size and hardware; 7B models on CPU can take 60+ seconds for longer prompts. Always set a timeout to prevent scripts from hanging indefinitely if Ollama crashes.

Can I run multiple inference requests in parallel with Ollama?

Ollama queues requests per model; only one runs at a time by default. Concurrent requests to the same model serialize. To achieve parallelism, run multiple Ollama instances on different ports or load multiple models (each consumes separate VRAM). For most scripts, serialize requests in your code or use a job queue.

What HTTP status code does Ollama return when a model is not found?

Ollama returns 404 with a JSON body like {"error":"model 'name' not found"}. Check the response body for the error string. Use curl --fail to make curl exit non-zero on 4xx/5xx responses, stopping your script immediately. Always validate model names against /api/tags before making inference requests.

How do I stream responses from /api/generate in a bash script?

Set "stream": true in your JSON payload and use curl --no-buffer. Each line is a JSON object with a response field containing a token or fragment. Parse with jq -r .response per line, or concatenate in a loop. The last line has "done": true. Example: curl --no-buffer -X POST ... | while read -r line; do echo "$line" | jq -r .response; done.

Share this article:
Bas van den Berg
About the Author

Bas van den Berg

IT Administrator, Security Architect, Infrastructure Security Specialist, Technical Author

Bas van den Berg is an experienced IT Administrator and Security Architect specializing in the design, protection, and long-term operation of secure IT infrastructures.

With a strong background in system administration and cybersecurity, he has worked extensively with enterprise environments, focusing on access control...

IT Administration Security Architecture Network Security System Hardening Access Control

Stay Updated

Subscribe to our newsletter for the latest tutorials, tips, and exclusive offers.