Promtail

Ship logs from Promtail to Parseable. Follow the setup steps and practical guidance for using Promtail in your Parseable observability workflow.

Ship logs from Promtail to Parseable using the HTTP client.

Overview

Integrate Promtail with Parseable to:

Kubernetes Native - Designed for Kubernetes log collection
Service Discovery - Automatic target discovery
Label Extraction - Rich labeling capabilities
Pipeline Stages - Powerful log processing

Prerequisites

Promtail installed
Parseable instance accessible
Kubernetes cluster (for k8s deployments)

Promtail Configuration

Basic Configuration

Create promtail-config.yaml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://parseable:8000/api/v1/ingest
    headers:
      Authorization: Basic YWRtaW46YWRtaW4=
      X-P-Stream: promtail-logs
    batchwait: 1s
    batchsize: 1048576

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

Kubernetes Pod Logs

server:
  http_listen_port: 9080

positions:
  filename: /run/promtail/positions.yaml

clients:
  - url: http://parseable:8000/api/v1/ingest
    headers:
      Authorization: Basic YWRtaW46YWRtaW4=
      X-P-Stream: k8s-logs

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    pipeline_stages:
      - cri: {}

JSON Log Parsing

scrape_configs:
  - job_name: json-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: app
          __path__: /var/log/app/*.json
    pipeline_stages:
      - json:
          expressions:
            level: level
            message: message
            timestamp: timestamp
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: RFC3339

Multi-line Logs

scrape_configs:
  - job_name: multiline
    static_configs:
      - targets:
          - localhost
        labels:
          job: java-app
          __path__: /var/log/java/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<level>\w+) (?P<message>.*)'
      - labels:
          level:

Running Promtail

Docker

docker run -d \
  --name promtail \
  -v $(pwd)/promtail-config.yaml:/etc/promtail/config.yaml \
  -v /var/log:/var/log:ro \
  grafana/promtail:latest \
  -config.file=/etc/promtail/config.yaml

Docker Compose

version: '3.8'
services:
  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yaml
      - /var/log:/var/log:ro
    command: -config.file=/etc/promtail/config.yaml

Kubernetes DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      serviceAccountName: promtail
      containers:
        - name: promtail
          image: grafana/promtail:latest
          args:
            - -config.file=/etc/promtail/config.yaml
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Pipeline Stages

Common Stages

Stage	Description
`json`	Parse JSON logs
`regex`	Extract with regex
`labels`	Add labels
`timestamp`	Parse timestamps
`multiline`	Handle multi-line logs
`drop`	Drop matching logs
`output`	Set log content

Example Pipeline

pipeline_stages:
  - json:
      expressions:
        level: level
        msg: message
        ts: timestamp
  - labels:
      level:
  - timestamp:
      source: ts
      format: RFC3339Nano
  - output:
      source: msg

Configuration Options

Parameter	Description
`clients[].url`	Parseable endpoint
`clients[].headers`	HTTP headers
`clients[].batchwait`	Max wait before sending
`clients[].batchsize`	Max batch size in bytes

Best Practices

Use Service Discovery - Leverage Kubernetes SD
Add Labels - Include relevant metadata
Parse Structured Logs - Extract fields from JSON
Handle Multi-line - Configure for stack traces
Monitor Promtail - Check metrics endpoint

Troubleshooting

Logs Not Appearing

Check Promtail logs for errors
Verify file paths are correct
Check permissions on log files
Verify Parseable endpoint is accessible

Label Issues

Check relabel_configs syntax
Verify source_labels exist
Test with promtool

Next Steps

Configure Fluent Bit as alternative
Set up alerts for log patterns
Create dashboards for monitoring

Promtail

On this page