Parseable Auto Instrumentation

Learn about Parseable Auto Instrumentation in Parseable, including core concepts, configuration steps, and practical guidance for building an effective…

Parseable Auto Instrumentation, or PAI, is a Kubernetes operator that automatically collects logs, metrics, traces, and Kubernetes events from a cluster and exports them to Parseable, with zero manual OpenTelemetry configuration required.

It is useful when you want cluster-wide observability, but do not want every team to maintain its own collector configuration. You define the Parseable OTLP endpoint, target datasets, namespace selectors, and trace instrumentation settings in one ParseableConfig custom resource. PAI then reconciles the OpenTelemetry Collector and Instrumentation resources for you.

PAI depends on the OpenTelemetry Operator. The OpenTelemetry Operator manages OpenTelemetryCollector and Instrumentation custom resources. PAI sits one level above it and generates those resources with Parseable-specific OTLP exporters, datasets, headers, authentication, and cleanup behavior.

How PAI works

ParseableConfig
  |
  v
PAI operator
  |
  | creates and reconciles
  v
OpenTelemetry Operator resources
  |
  +-- pai-log-collector                 DaemonSet collector
  +-- pai-metrics-events-collector      Deployment collector
  +-- pai-traces                        Deployment collector
  +-- pai-instrumentation-collector-v1  Instrumentation CR
  +-- pai-agent                         DaemonSet for language detection
  |
  v
Parseable datasets

The important thing to understand is that PAI does not bypass OpenTelemetry. It uses OpenTelemetry as the collection layer. Logs, metrics, traces, and events are collected or received by OpenTelemetry Collectors, then exported to Parseable over OTLP HTTP.

For traces, PAI creates an in-cluster trace collector called pai-traces. Auto-instrumented application pods send traces to the stable pai-traces-collector service. That collector owns the Parseable ingestion endpoint, API key, target dataset, OTLP encoding, and exporter headers, so application pods do not need Parseable credentials.

What PAI creates

Once you apply a ParseableConfig, PAI creates the resources needed for the enabled signals.

Resource	Type	Purpose
`pai-log-collector`	OpenTelemetry Collector DaemonSet	Reads pod logs from every node and collects kubelet resource metrics when enabled
`pai-metrics-events-collector`	OpenTelemetry Collector Deployment	Collects cluster metrics, Prometheus scrape metrics, and Kubernetes events
`pai-traces`	OpenTelemetry Collector Deployment	Receives traces from instrumented workloads and exports them to Parseable
`pai-instrumentation-collector-v1`	OpenTelemetry `Instrumentation` CR	Tells the OpenTelemetry Operator how to inject SDKs into application pods
`pai-agent`	DaemonSet	Detects application runtimes, including distroless containers
`<namespace>-pai-collector`	ClusterRole	Gives collector service accounts the permissions required to read Kubernetes telemetry

You do not need to manage these resources directly. PAI owns their lifecycle. If the ParseableConfig changes, PAI reconciles them. If the ParseableConfig is deleted, PAI cleans them up.

Prerequisites

Before installing PAI, make sure you have:

A Kubernetes cluster running v1.26 or later
Helm v3
A Parseable OTLP HTTP ingestion endpoint
A Parseable API key
OpenTelemetry Operator v0.99.0 or later installed in the cluster

PAI uses the v1beta1 OpenTelemetryCollector API, so older OpenTelemetry Operator versions will not work. The Instrumentation resource is still created with the OpenTelemetry Operator's v1alpha1 API.

Install PAI

Install the OpenTelemetry Operator

If the OpenTelemetry Operator is not already installed, install it first:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace otel-operator --create-namespace \
  --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true

Install the PAI operator

Install the operator into its own namespace:

helm repo add parseable https://charts.parseable.com
helm repo update
helm install pai parseable/pai -n pai-system --create-namespace

Check that the operator pod is running:

kubectl get pods -n pai-system

Create a Parseable credentials secret

For new setups, use an API key. PAI reads the apiKey value from the secret and sends it to Parseable as the x-api-key header.

kubectl create secret generic parseable-creds \
  --from-literal=apiKey=<YOUR_PARSEABLE_API_KEY> \
  -n pai-system

PAI still supports basic authentication for existing deployments. In that case, set authType: basic or omit authType, and create the secret with username and password.

Apply a ParseableConfig

Create a ParseableConfig resource. This is the main PAI configuration object. The example below enables pod logs, cluster metrics, Kubernetes events, and trace auto-instrumentation for selected namespaces.

apiVersion: observability.parseable.com/v1alpha1
kind: ParseableConfig
metadata:
  name: production
  namespace: pai-system
spec:
  target:
    endpoint: https://<YOUR_PARSEABLE_OTLP_ENDPOINT>
    authType: apiKey
    encoding: json
    credentialsSecret:
      name: parseable-creds
      namespace: pai-system
    headers:
      X-P-Environment: production

  logs:
    podLogs:
      enabled: true
      targetDataset: pai-logs
      namespaceSelector:
        mode: include
        namespaces:
          - app-namespace

  metrics:
    clusterMetrics:
      targetDataset: pai-cluster-metrics
      namespaceSelector:
        mode: include
        namespaces:
          - app-namespace
      k8sCluster:
        enabled: true
      kubelet:
        enabled: true

  events:
    enabled: true
    targetDataset: pai-events
    namespaceSelector:
      mode: include
      namespaces:
        - app-namespace

  traces:
    targetDataset: pai-traces
    namespaceSelector:
      mode: include
      namespaces:
        - app-namespace
    instrumentation:
      languages:
        - java
        - python
        - nodejs
        - dotnet
      detectionTimeout: "1m"

Apply it:

kubectl apply -f parseableconfig.yaml

What each signal means

Logs

PAI collects Kubernetes pod logs through the OpenTelemetry filelog receiver. The log collector runs as a DaemonSet, so every node can read the pod log files available on that node. Records are enriched with Kubernetes metadata such as namespace, pod, container, and node.

Use spec.logs.podLogs.namespaceSelector when you only want logs from specific namespaces, or when you want to exclude noisy namespaces. You can also use spec.logs.files to tail host log directories, such as audit logs or custom application logs written to a mounted path.

Metrics

PAI supports built-in cluster metrics and Prometheus-style scrape configs. Built-in metrics come from receivers such as k8s_cluster, kubelet scraping, and kube-state-metrics when enabled. These give you Kubernetes object state, pod and container resource usage, node conditions, and similar operational signals.

For application metrics, use spec.metrics.scrapeConfigs. PAI creates Prometheus scrape pipelines inside the OpenTelemetry Collector and exports those metrics into the dataset you choose.

Traces

For traces, PAI uses OpenTelemetry auto-instrumentation. You list the languages you want PAI to instrument under spec.traces.instrumentation.languages. PAI then creates an OpenTelemetry Instrumentation resource and annotates matching workloads for SDK injection.

The supported instrumentation images in PAI currently cover Java, Python, Node.js, and .NET. PAI also runs an agent that detects workload languages, including workloads that use distroless images where normal image-name checks are not enough.

Instrumented applications send traces to the in-cluster pai-traces-collector service. The trace collector then exports the data to the Parseable dataset configured in spec.traces.targetDataset.

Events

PAI collects Kubernetes events through the k8sobjects receiver in watch mode. Events are exported as the logs signal and stored in the dataset configured under spec.events.targetDataset.

This helps correlate scheduling failures, restarts, image pull errors, and other cluster activity with the rest of your telemetry.

Control collection scope

Namespace selector

Most Kubernetes-aware signal configs support namespaceSelector.

namespaceSelector:
  mode: include
  namespaces:
    - app-namespace
    - payments

Use include to collect only from the listed namespaces. Use exclude to collect from every namespace except the listed namespaces. If you omit the selector, PAI collects from all namespaces for that signal.

For cluster metrics, namespace filtering applies to pod-scoped metrics. Node-scoped metrics are not filtered by namespace because nodes do not belong to a namespace.

Workload selector for traces

Use spec.traces.workloadSelector when you want trace auto-instrumentation only for workloads with specific labels.

spec:
  traces:
    targetDataset: pai-traces
    workloadSelector:
      mode: include
      matchLabels:
        observability: enabled

This is helpful when you want to roll out tracing gradually instead of instrumenting every workload in a namespace at once.

Custom headers

You can set custom headers globally under spec.target.headers, or per signal under the relevant signal config.

spec:
  target:
    headers:
      X-P-Environment: production

  traces:
    targetDataset: pai-traces
    headers:
      X-P-Environment: production-traces

Signal-level headers override global headers with the same key. PAI-generated headers such as authentication, X-P-Stream, X-P-Log-Source, and X-P-Tenant take precedence over custom headers.

Encoding

spec.target.encoding controls the OTLP HTTP wire format. Use json unless you know your Parseable deployment supports protobuf ingestion for the relevant signal.

spec:
  target:
    encoding: json

Add host logs or application metrics

Host file logs

Use spec.logs.files to tail log files from a host path on every node.

spec:
  logs:
    files:
      - name: audit-logs
        hostPath: /var/log/app/audit
        targetDataset: pai-audit-logs

PAI creates a separate filelog pipeline for each entry.

Prometheus scrape configs

Use spec.metrics.scrapeConfigs when an application exposes Prometheus metrics.

spec:
  metrics:
    scrapeConfigs:
      - name: application-metrics
        uri: /metrics
        port: 8080
        targetDataset: pai-application-metrics
        namespaceSelector:
          mode: include
          namespaces:
            - app-namespace
        podSelector:
          app.kubernetes.io/name: my-app

PAI discovers matching pods, scrapes the configured path and port, then exports the metrics to the selected Parseable dataset.

Verify the setup

Check that the PAI operator and generated collectors are running:

kubectl get pods -n pai-system
kubectl get opentelemetrycollector -n pai-system
kubectl get instrumentation -n pai-system

Check the ParseableConfig status:

kubectl get parseableconfig production -n pai-system -o yaml

Look at status.conditions for reconciliation errors. For traces, status.workloads shows which workloads were processed, which language was detected, and whether instrumentation was applied.

Once telemetry is flowing, open Parseable and look for the datasets you configured, such as pai-logs, pai-cluster-metrics, pai-events, and pai-traces.

Troubleshooting

No collectors are created: Check the ParseableConfig status and confirm the OpenTelemetry Operator is installed. PAI requires the v1beta1 OpenTelemetryCollector CRD.
Authentication errors: Confirm spec.target.authType matches the secret keys. apiKey requires an apiKey key. Basic auth requires username and password.
Logs are missing from a namespace: Check spec.logs.podLogs.namespaceSelector. In include mode, only listed namespaces are collected.
Metrics are missing: Confirm the relevant metric source is enabled. For built-in metrics, check spec.metrics.clusterMetrics. For app metrics, check spec.metrics.scrapeConfigs, the pod labels, port, and metrics path.
Traces are missing: Check that the workload namespace and labels match your trace selectors, then inspect status.workloads to confirm language detection and instrumentation.

Use collector logs when you need more detail:

# Metrics and events
kubectl logs -n pai-system deployment/pai-metrics-events-collector-collector --all-containers --tail=200

# Traces
kubectl logs -n pai-system deployment/pai-traces-collector --all-containers --tail=200

The log collector runs as a DaemonSet, so select the pai-log-collector pod on the affected node:

kubectl get pods -n pai-system
kubectl logs -n pai-system <PAI_LOG_COLLECTOR_POD_NAME> --all-containers --tail=200

Pause collection

Set spec.paused: true to stop collection without deleting the ParseableConfig.

spec:
  paused: true

PAI removes collectors, instrumentation, and the agent while the config is paused. Set it back to false to resume collection.

Uninstall

Delete the ParseableConfig first so PAI can clean up generated resources.

kubectl delete parseableconfig production -n pai-system
helm uninstall pai -n pai-system
kubectl delete namespace pai-system

For a full cluster cleanup, including OpenTelemetry resources created during testing, use the cleanup script from the PAI repository.

curl -sL https://raw.githubusercontent.com/parseablehq/pai/refs/heads/main/scripts/cleanup.sh | bash

Parseable Auto Instrumentation

On this page