# Architecture (/architecture)
This document outlines the overall architecture of the Parseable Observability Platform, detailing the flow of MELT data from ingestion to storage and querying.
This document is organized into specific sections for each sub-system like ingestion, query, search, and index. To understand the specific decisions and trade-offs, refer the [design choices document](/design-choices).
## Overview
Parseable is shipped as a single unified binary (or container image if you prefer). This includes the Prism and Parseable DB. There is no additional dependency to run Parseable.
The binary can be run in different modes. You’d generally run standalone mode to test and experience Parseable on your laptop, or a small testing server.
As you move to a production setup, we recommend running the distributed mode. Here each node has a specific role, i.e. ingestion, query or search.
## Ingestion
Parseable ingestion nodes follow a shared-nothing architecture, meaning each node independently handles the entire ingestion pipeline. In production, you typically place a load balancer in front of two or more ingestion nodes, allowing ingestion requests to be distributed across nodes seamlessly.
When a node receives an ingestion request (via HTTP or Kafka), it first validates the request, then converts the payload into an Apache Arrow-based file format. During this process, it also performs auto schema detection, enabling Parseable to intelligently classify logs and generate structured schemas on the fly. This makes it easy for users to filter, search, and analyze across diverse log types with minimal upfront configuration.
The Arrow files are temporarily staged in a dedicated local disk area. Once the disk write completes, the ingestion node acknowledges the request with a success response.
To ensure data durability during staging, we recommend attaching a small, reliable disk (such as NFS, Azure Files, or EFS) to each ingestion node.
A background job then reads the staged Arrow files, converts them into highly compressed Parquet files, and uploads them to S3 or any configured object store. During this transformation, the ingestion node also generates query metadata, which significantly enhances performance during log searches and queries.
## Query
Distributed Query nodes are Enterprise only features available in cloud and enterprise versions and cannot be deployed with OSS versions.
Query node is primarily responsible responding to query API. The query workflow starts when someone calls the query API with (a PostgreSQL compatible) SQL query, and a start and end timestamp. The query node looks up the metadata locally first, falling back to object store only if not found.
Based on metadata, the node identifies the relevant parquet files and uses the object store API to get these files. Here again, this only happens if the files are not already present locally. If the files are to be downloaded from object storage - this adds to latency and hence the occasional cold queries.
Another node called Prism node responds to all the role, user management, dataset management API. In Parseable OSS, a query node also serves as the leader node.
## Design choices
### Low latency writes
Ingested data is staged on local disk upon successful return by Parseable API. Data is then asynchronously committed to object store like S3. This ensures low latency, high throughput ingestion. To ensure data durability, we recommend using a small, reliable storage (EFS, Azure Files, NFS or equivalent) attached to the ingesting nodes. This ensures that data is not lost in case of a node failure.
### Atomic ingestion
Each ingestion batch received via API is concurrently appended to the same file within a one-minute window. When converted from Arrow to Parquet, entries are reordered to ensure the latest data appears first.
### Efficient storage
Parseable stores heavily compressed Parquet format to one of the most cost efficient storage, i.e. object storage. This leads to significant cost savings, especially for large datasets.
### Smart caching
Frequently accessed logs are cached in memory and NVMe SSDs on query nodes for faster access. The system prioritizes recent data, manages cache eviction automatically, and minimizes object store API calls using Parseable manifest files and Parquet footers.
### Index on demand
By default data is stored in columnar Parquet files, allowing fast aggregations, filtering numerical columns and SQL queries. Parseable allows indexing specific chunks of data, on demand - to allow text search on log data as and when needed.
### Stateless high availability
High availability (HA) is ensured through a distributed mode in which multiple ingestion and query servers operate independently.
### Object storage first
There is no separate consensus layer, eliminating complex coordination and reducing operational overhead. Object storage manages all concurrency control. See [AWS S3 configuration](/storage/awss3) for detailed setup instructions.
### SQL for querying
We chose SQL as the query language for Parseable because it is widely used and understood, making it easier for users to interact with the system. SQL allows users to filter, aggregate, and join data from multiple sources. SQL is also very well supported by modern LLMs to generate queries from plain text.
## Trade-offs
### Staged writes
Staging data locally on the ingestor node for at least a minute, leads to a minor lag in querying the data. We trades immediate persistence for low latency ingestion.
### Occasional Cold Queries
The query layer fetches indexes from object storage (e.g., S3) and uses intelligent caching to accelerate future access. During the initial cache warm-up, some queries may access data directly from cold storage, resulting in higher latencies.
### Timed queries
A query call requires start and end timestamp. This ensures data is queried across a fixed, definite set of files. Parseable ensures query response includes the staging and committed data on object storage as required.
# Benchmarks (/benchmarks)
Performance is a core requirement for observability platforms. As telemetry data volumes grow exponentially, it is critical for observability solutions to deliver high performance, at crucial times when teams need to troubleshoot and resolve issues quickly.
Parseable's telemetry data lake architecture is designed from the ground up for high performance ingestion and querying of telemetry data at scale.
We use modern storage formats, efficient indexing techniques, and optimized query execution strategies to deliver fast performance, additionally local NVMe and memory caching further accelerates query response times.
## ClickBench
[ClickBench](https://benchmark.clickhouse.com/) is an industry-standard benchmark for analytical databases. While originally designed for clickstream data, it has been widely adopted to evaluate performance of databases for various analytical workloads, including observability data.
Parseable is a top performer on the ClickBench. [View Parseable on ClickBench](http://logg.ing/clickbench)
Read more about our performance philosophy in our post: [Performance is Table Stakes](https://www.parseable.com/blog/performance-is-table-stakes).
## Real world performance
Understanding Parseable's performance with a real-world scenario of ingesting and querying 100 TB of telemetry data per day.
Test Results at a Glance
| Metric | Result |
| --------------------- | ------------------------------------ |
| **Ingestion Rate** | **2 TB/hour** (sustained) |
| **Query Latency** | **Millisecond-range** (even at load) |
| **Monthly Cost** | **\~$11,810** (for a 4-node cluster) |
| **Compression Ratio** | **Up to 90%** |
For more details, see our blog post: [The Economics and Physics of 100 TB Telemetry Data Per Day](https://www.parseable.com/blog/the-economics-and-physics-of-100-tb-telemetry-data-per-day).
# Design Choices (/design-choices)
This document outlines our key design choices, ensuring durability, scalability, and efficiency for modern observability workloads. This page also covers the technical trade offs in Parseable.
If you have a specific use case or need a feature tailored to your observability needs, let us know at [sales@parseable.com](mailto:sales@parseable.com). We ship fast and most of such requests can be done in a matter of days.
### Highlights
#### Low latency writes
Ingested data is staged on local disk upon successful return by Parseable API. Data is then asynchronously committed to object store like S3. This ensures low latency, high throughput ingestion. To ensure data durability, we recommend using a small, reliable storage (EFS, Azure Files, NFS or equivalent) attached to the ingesting nodes. This ensures that data is not lost in case of a node failure.
#### Atomic ingestion
Each ingestion batch received via API is concurrently appended to the same file within a one-minute window. When converted from Arrow to Parquet, entries are reordered to ensure the latest data appears first.
#### Efficient storage
Parseable stores heavily compressed Parquet format to one of the most cost efficient storage, i.e. object storage. This leads to significant cost savings, especially for large datasets.
#### Smart caching
Frequently accessed logs are cached in memory and NVMe SSDs on query nodes for faster access. The system prioritizes recent data, manages cache eviction automatically, and minimizes object store API calls using Parseable manifest files and Parquet footers.
#### Index on demand
By default data is stored in columnar Parquet files, allowing fast aggregations, filtering numerical columns and SQL queries. Parseable allows indexing specific chunks of data, on demand - to allow text search on log data as and when needed.
#### Stateless high availability
High availability (HA) is ensured through a distributed mode in which multiple ingestion and query servers operate independently.
#### Object storage first
There is no separate consensus layer, eliminating complex coordination and reducing operational overhead. Object storage manages all concurrency control.
#### SQL for querying
We chose SQL as the query language for Parseable because it is widely used and understood, making it easier for users to interact with the system. SQL allows users to filter, aggregate, and join data from multiple sources. SQL is also very well supported by modern LLMs to generate queries from plain text.
### Trade-offs
#### Staged writes
Staging data locally on the ingestor node for at least a minute, leads to a minor lag in querying the data. We enable immediate persistence for low latency ingestion.
#### Occasional Cold Queries
The query layer fetches indexes from object storage (e.g., S3) and uses intelligent caching to accelerate future access. During the initial cache warm-up, some queries may access data directly from cold storage, resulting in higher latencies.
#### Timed queries
A query call requires start and end timestamp. This ensures data is queried across a fixed, definite set of files. Parseable ensures query response includes the staging and committed data on object storage as required.
# Features (/features)
import { IconSparkles, IconBellRinging, IconDashboard, IconSql, IconGauge, IconLock, IconBrain, IconUserCircle, IconDatabase, IconRefresh, IconActivityHeartbeat } from '@tabler/icons-react';
Explore the main capabilities available in Parseable, from AI-native investigation and APM to dashboards, alerting, PromQL, access control, APIs, and automated Kubernetes instrumentation. Use this page as a feature map for moving from telemetry collection to investigation, operations, and production management.
} title="Agent Observability">
Debug AI agent runs with traces, prompts, tool calls, costs, and runtime behavior in one view.
} title="PromQL">
Native PromQL support with Grafana Prometheus data source for seamless visualization.
} title="APM">
Analyze service health, latency, errors, and traces from one application performance workflow.
} title="Alerting">
Set up real-time alerts to notify you of important events or anomalies in your telemetry data.
} title="Dashboards">
Create customizable dashboards to visualize your telemetry data and gain insights at a glance.
} title="OpenID Integration">
Seamlessly integrate with OpenID providers for secure authentication and single sign-on capabilities.
} title="AI Native">
AI-powered observability with Keystone Q\&A, dataset summarization, and natural language to SQL conversion.
} title="SQL Editor">
Write and execute SQL queries to analyze your telemetry data with a powerful query editor.
} title="Role-Based Access Control">
Secure your data with fine-grained access controls and user permissions management.
} title="Log IQ">
Transform unstructured logs into structured JSON data for easier querying, searching, and visualization.
} title="Retention Policies">
Manage your data lifecycle with customizable retention policies to optimize storage costs and compliance.
} title="Smart Cache">
Optimize query performance with intelligent caching mechanisms for frequently accessed data.
} title="Multi-tenancy">
Run tenant-aware deployments with isolated request handling, storage layout, and access control.
} title="PAI">
Parseable Auto Instrumentation, or PAI, is a Kubernetes operator that automatically collects logs, metrics, traces, and events from a cluster and exports them to Parseable, with zero manual configuration required.
# Quickstart (/get-started)
## Choose how to run Parseable
### Executable Binary \[toc]
### Parseable Cloud \[toc]
### Docker Image \[toc]
### Self Host \[toc]
Run Parseable directly from the single binary.
Create a managed workspace.
Start Parseable in a local container.
Run Parseable for production workloads.
Follow one of these guides to start an instance. For a local binary or Docker instance, continue below to send your first event. For Cloud, use the ingestion endpoint and credentials shown in your workspace instead of the local values.
```bash
curl -fsSL https://logg.ing/install | bash
```
For Windows users
```bash
powershell -NoProfile -Command "irm https://logg.ing/install-windows | Out-String | iex"
```
The local instance listens on port `8000`. Send one JSON event to the `demo` dataset:
```bash
curl --location 'http://localhost:8000/api/v1/ingest' \
--header 'X-P-Stream: demo' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data '[{"message":"hello from Parseable"}]'
```
The `X-P-Stream` header selects the dataset. The authorization header uses the local default `admin:admin` credentials.
## Verify it worked
Open [the local console](http://localhost:8000), sign in with `admin` / `admin`, and select `demo` in the log explorer. You should see the event with `message` set to `hello from Parseable`.
If it does not appear, confirm that Parseable is still running, then check the URL, credentials, and `X-P-Stream` header in the command above. For a production deployment, use the [self-hosted installation guide](/self-hosted/installation).
## Next steps
Send data from [OpenTelemetry](/ingest-data/otel), [a logging agent](/ingest-data/logging-agents), or [your application](/ingest-data/programming-languages). Then [query logs](/user-guide/sql-editor), [build a dashboard](/user-guide/dashboards), or [set an alert](/user-guide/alerting).
# Home (/)
Parseable is an open source, column-oriented data lake built for observability. It brings logs, metrics, traces, and agent telemetry into one system, with SQL and PromQL queries, dashboards, alerting, APM, and AI workflows on top.
Telemetry does not all behave the same way. Logs can be wide and change shape from event to event; metrics can have millions of label combinations; and traces need both individual span lookup and service-level analysis. Parseable stores these signals on a shared, open data lake foundation while letting each be ingested and queried in the way it needs.
## How Parseable works
Parseable treats growing telemetry as a data engineering problem. Data moves through three stages:
1. **Ingest:** The data lake engine accepts events at high volume, stages them locally, and forwards them in batches. Staging helps absorb ingestion spikes while producing efficiently sized objects for storage.
2. **Store:** Events are compressed into columnar Apache Parquet files on object storage. Queries can read only the columns they need, and other Parquet-compatible tools can use the same data.
3. **Query:** A separate query engine runs SQL and PromQL. It caches frequently accessed data on local disk, so hot queries do not need to fetch every file from object storage. Ingestion and query capacity can scale independently.
Parseable is written in Rust for predictable performance without garbage collection pauses. Apache Arrow provides an in-memory columnar format for vectorized execution; Parquet is the open format at rest. This design supports wide events, high-cardinality metrics, and growing volumes without keeping every byte on always-hot storage. Read the [architecture](/architecture) and [key concepts](/key-concepts) for the deeper model.
You can use the self-hosted core as a single binary, run a managed [Cloud workspace](/quickstart/cloud), or choose [Enterprise](/editions/compare) for advanced deployments. See the [edition comparison](/editions/compare) for details.
```bash
powershell -NoProfile -Command "irm https://logg.ing/install-windows | Out-String | iex"
```
The [ingestion overview](/ingestion) explains the available protocols. Start with the source you already have:
Send telemetry through the OpenTelemetry Collector.
Forward logs from lightweight agents and edge workloads.
Route, transform, and send logs with Vector.
Collect logs from Docker containers.
Send logs and telemetry from Kubernetes clusters.
Ingest Prometheus metrics with remote write.
Send telemetry directly from application code.
Observe agents, LLM gateways, runtimes, and coding assistants.
## Work with each signal
* **Logs:** Work with events that range from unstructured text to JSON with changing fields. Parseable can read selected columns from wide or sparse events, while [LogIQ](/user-guide/log-iq) can extract fields from verbose log strings. [Send OTLP logs](/ingest-data/otel/logs), then use the [log explorer](/user-guide/logs) or [SQL editor](/user-guide/sql-editor).
* **Metrics:** Store numeric time series identified by labels. Parseable stores label values as column data for high-cardinality metrics and provides native PromQL. Send metrics with [OpenTelemetry](/ingest-data/otel/metrics) or [Prometheus remote write](/ingest-data/prometheus), then [explore metrics](/user-guide/metrics) or [query with PromQL](/user-guide/promql).
* **Traces:** Connect spans by trace ID for specific trace lookup and service-level analysis, including latency and error rates. Parseable supports both patterns on the same columnar data. [Send OTLP traces](/ingest-data/otel/traces), then use [traces](/user-guide/traces) or [APM](/user-guide/apm).
* **Agent telemetry:** Bring activity from AI agents and LLM infrastructure into the same observability foundation. Follow the [AI infrastructure guides](/ingest-data/ai-agents) and [agent observability](/user-guide/agent-observability) to inspect it.
## Keep working
* **Investigate and share:** [Build dashboards](/user-guide/dashboards), [set alerts](/user-guide/alerting), or [connect an integration](/integrations).
* **Deploy and manage:** Follow the [self-hosted installation guide](/self-hosted/installation), then review [configuration](/self-hosted/configuration) and [access control](/user-guide/rbac).
* **Use developer tools:** Find endpoints in the [API reference](/api), or use the [MCP server](/mcp) to investigate telemetry with an AI client.
* **Find a cause:** If data is missing, check the [ingestion overview](/ingestion), your source guide above, and [API key setup](/user-guide/api-keys).
## Read with an AI agent
Use [the full documentation as text](/llms-full.txt), or **Copy Markdown** on any page. Individual pages also have `.mdx` text endpoints, such as [/docs/pb-cli/install.mdx](/pb-cli/install.mdx). For terminal workflows, see [pb CLI for agents](/pb-cli/agents).
# Ingestion Reference (/ingestion-reference)
## Ingestion HTTP headers
You can use HTTP headers to control how data is ingested and processed.
Field name starting with `@` is replaced with `_`. In case of key collision, the event is rejected.
When an incoming event has a field with a data type different from the existing schema, the field is automatically renamed with a type suffix (e.g., `body_timestamp_utf8`, `span_kind_int64`). This prevents ingestion failures where field types may vary between events.
### Required headers
| Header | Description | Example | Possible Values |
| --------------- | ------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------- |
| `X-P-Stream` | Target dataset name. Creates the dataset if it doesn't exist. | `nginx-logs` | Valid dataset name |
| `X-API-Key` | API key used to authorize the request | `px_api_key` | Valid Parseable API key |
| `X-P-Tenant-ID` | Target tenant ID. | `tenant-123` | Valid tenant ID - only applicable for multi-tenant setups (for example Parseable Cloud) |
| `Content-Type` | Content type of the request body | `application/json` | `application/json`,`application/protobuf` |
### Optional headers
| Header | Description | Example |
| ----------------- | -------------------------------------------------------------------- | --------------------------------- |
| `X-P-Tag-{field}` | Add custom tags/metadata to events. Replace `{field}` with tag name. | `X-P-Tag-environment: production` |
## Reference
Choose a protocol below, or browse [Integrations](/integrations) for source-specific guides to databases, containers, cloud providers, streaming platforms, CI/CD tools, and security sources.
* **HTTP ingestion:** [JSON ingestion API](/api/v1/ingestLogsWithHeaders).
* **OpenTelemetry:** [Logs](/ingest-data/otel/logs), [metrics](/ingest-data/otel/metrics), [traces](/ingest-data/otel/traces), and [OpenTelemetry Collector](/ingest-data/logging-agents/otel-collector).
* **Prometheus:** [Remote write](/ingest-data/prometheus).
* **Agents and instrumentation:** [Telemetry agents](/ingest-data/logging-agents), [Parseable Auto Instrumentation](/ingest-data/auto-instrumentation), and [eBPF](/ingest-data/zero-instrumentation).
* **Application sources:** [Programming languages](/ingest-data/programming-languages) and [AI infrastructure](/ingest-data/ai-agents).
In case of otel-metrics, there may be key collision in the attributes key, as the attributes are present in different hierarchical levels in the otel metrics event. In such scenarios, the key values are overridden by the last available attribute.
## Log processing
When agents like Fluent Bit or Vector send logs to Parseable, they typically send the entire log line in a single field (usually named `log` or `message`). With log extraction enabled, Parseable can automatically parse this raw log line and extract structured fields using regex patterns.
**The flow:**
1. Agent sends JSON log data with the raw log line in a field (e.g., `{"log": "192.168.1.1 - - [10/Jan/2026:12:00:00] ..."}`)
2. Agent sets `X-P-Extract-Log` header to the field name containing the raw log (e.g., `log`)
3. Agent sets `X-P-Log-Source` header to the log format name (e.g., `nginx_access`)
4. Parseable reads the value from the specified field, applies the matching regex pattern, and adds all extracted fields to the event
### Headers for log extraction
| Header | Description | Example | Possible Values |
| ----------------- | -------------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------- |
| `X-P-Log-Source` | Log format name - Parseable applies regex patterns to extract fields | `nginx_access`, `syslog_log` | See [Supported log source formats](#supported-log-source-formats) |
| `X-P-Extract-Log` | Field name in the JSON payload that contains the raw log line | `log`, `message` | Any valid field name |
### Supported log source formats
| # | Format Name |
| -- | ----------------------- |
| 1 | `access_log` |
| 2 | `alb_log` |
| 3 | `block_log` |
| 4 | `candlepin_log` |
| 5 | `choose_repo_log` |
| 6 | `cloudvm_ram_log` |
| 7 | `cups_log` |
| 8 | `dpkg_log` |
| 9 | `elb_log` |
| 10 | `engine_log` |
| 11 | `env_logger_log` |
| 12 | `error_log` |
| 13 | `esx_syslog_log` |
| 14 | `haproxy_log` |
| 15 | `java` |
| 16 | `katello_log` |
| 17 | `klog` |
| 18 | `kubernetes_log` |
| 19 | `lnav_debug_log` |
| 20 | `nextflow_log` |
| 21 | `nginx_access` |
| 22 | `openam_log` |
| 23 | `openamdb_log` |
| 24 | `openstack_log` |
| 25 | `page_log` |
| 26 | `parseable_server_logs` |
| 27 | `postgres` |
| 28 | `postgresql_log` |
| 29 | `procstate_log` |
| 30 | `proxifier_log` |
| 31 | `rails_log` |
| 32 | `redis_log` |
| 33 | `s3_log` |
| 34 | `simple_rs_log` |
| 35 | `snaplogic_log` |
| 36 | `sssd_log` |
| 37 | `strace_log` |
| 38 | `sudo_log` |
| 39 | `syslog_log` |
| 40 | `tcf_log` |
| 41 | `tcsh_history` |
| 42 | `uwsgi_log` |
| 43 | `vmk_log` |
| 44 | `vmw_log` |
| 45 | `vmw_py_log` |
| 46 | `vmw_vc_svc_log` |
| 47 | `vpostgres_log` |
| 48 | `web_robot_log` |
| 49 | `xmlrpc_log` |
| 50 | `zookeeper` |
| 51 | `zookeeper_log` |
## Flattening
Nested JSON objects are automatically flattened. For example, the following JSON object
```json
{
"foo": {
"bar": "baz"
}
}
```
will be flattened to
```json
{
"foo_bar": "baz"
}
```
before it gets stored. While querying, this field should be referred to as foo\_bar. For example, select `foo_bar` from ``. The flattened field will be available in the schema as well.
## Batching and Compression
Wherever applicable, we recommend enabling the log agent's compression and batching features to reduce network traffic and improve ingestion performance. The maximum payload size in Parseable is 10 MiB (10485760 Bytes). The payload can contain single log event as a JSON object or multiple log events in a JSON array. There is no limit to the number of batched events in a single call.
## Timestamp
Correct time is critical to understand the proper sequence of events. Timestamps are important for debugging, analytics, and deriving transactions. We recommend that you include a timestamp in your log events formatted in RFC3339 format.
Parseable uses the event-received timestamp and adds it to the log event in the field `p_timestamp`. This ensures there is a time reference in the log event, even if the original event doesn't have a timestamp.
## Staging
Staging in Parseable refers to the process of storing log data on locally attached storage before it is pushed to a long term and persistent store like S3 or something similar. Staging acts as a buffer for incoming events and allows a stable approach to pushing events to the persistent store.
Once an HTTP call is received on the Parseable server, events are parsed and converted to Arrow format in memory. This Arrow data is then written to the staging directory (defaults to `$PWD/staging`). Every minute, the server converts the Arrow data to Parquet format and pushes it to the persistent store. We chose a minute as the default interval, so there is a clear boundary between events, and the prefix structure on S3 is predictable.
The query flow in Parseable allows transparent access to the data in the staging directory. This means that the data in the staging directory is queryable in real-time. As a user, you won't see any difference in the data fetched from the staging directory or the persistent store.
The staging directory can be configured using the `P_STAGING_DIR` environment variable, as explained in the environment vars section.
## Planning for Production
When planning for the production deployment of Parseable, the two most important considerations from a staging perspective are:
Storage size: Ensure that the staging area has sufficient capacity to handle the anticipated log volume. This prevents data loss due to disk space exhaustion. To calculate the storage size, consider the average log event size, the expected log volume for 5-10 minutes. This is done as under high loads, the conversion to Parquet and subsequent push to S3 may lag behind.
Local storage redundancy: Data in staging has not been committed to persistent store, it is important to have the staging itself reliable and redundant. This way, the staging data is protected from data loss due to simple disk failures. If using AWS, choose from services like EBS (Elastic Block Store) or EFS (Elastic File System), and mount these volumes on the Parseable server. Similarly, on Azure chose from Managed Disks or Azure Files. If you're using a private cloud, a reliable mounted volume from a NAS or SAN can be used.
# Ingestion (/ingestion)
import {
IconActivityHeartbeat,
IconArrowsShuffle,
IconBrain,
IconCloud,
IconCode,
IconContainer,
IconCpu,
IconCpu2,
IconDatabase,
IconGauge,
IconGitBranch,
IconRoute,
IconServer,
IconShieldLock,
IconTransform,
} from '@tabler/icons-react';
## Ingestion guides \[toc]
Choose the ingestion path that matches the system sending data to Parseable.
### OpenTelemetry \[toc]
### Prometheus Remote Write \[toc]
### AI Infrastructure \[toc]
### Telemetry Agents \[toc]
### Containers \[toc]
### Streaming \[toc]
### Databases \[toc]
### Parseable Auto Instrumentation \[toc]
### Security \[toc]
### eBPF \[toc]
### GPU \[toc]
### CI/CD \[toc]
### Infrastructure \[toc]
### Cloud \[toc]
### Programming Languages \[toc]
} title="OpenTelemetry">
Send logs, metrics, and traces through OTLP and OpenTelemetry-compatible pipelines.
} title="Prometheus Remote Write">
Ingest Prometheus metrics with the remote write protocol.
} title="AI Infrastructure">
Observe agents, LLMs, gateways, runtimes, sandboxes, and coding assistants.
} title="Telemetry Agents">
Forward telemetry with collectors and agents such as Fluent Bit, Vector, Logstash, and OTel Collector.
} title="Containers & Orchestration">
Collect logs and telemetry from Docker, Kubernetes, ECS, EKS, GKE, and AKS.
} title="Streaming">
Route events through Kafka, Redpanda, RabbitMQ, NATS, Cribl, and managed streaming platforms.
} title="Databases">
Send or migrate telemetry from PostgreSQL, MySQL, MongoDB, Redis, SQL Server, and Elasticsearch.
} title="Parseable Auto Instrumentation">
Collect Kubernetes logs, metrics, traces, and events with Parseable-managed instrumentation.
} title="Security">
Bring security events from Auth0, Falco, Trivy, and related sources into Parseable.
} title="eBPF">
Capture kernel-level observability signals with eBPF-based tooling.
} title="GPU">
Collect GPU metrics and accelerator telemetry.
} title="CI/CD">
Capture telemetry from GitHub Actions, Jenkins, GitLab CI, CircleCI, ArgoCD, and Terraform.
} title="Infrastructure">
Collect telemetry from infrastructure services such as Traefik and Windows Server.
} title="Cloud">
Ingest logs and events from AWS, Azure, Google Cloud, and Fly.io.
} title="Programming Languages" className="col-span-2">
Send telemetry directly from application code in Python, JavaScript, Go, Java, Rust, .NET, PHP, and Ruby.
You can ingest data to Parseable in JSON format over HTTP(s). This means that any log agent, metric collector, or tracing library that can send data over HTTP in JSON format can be used to ingest data into Parseable. For log data you can use the HTTP output plugins of all the logging agents/shippers. You can also directly integrate Parseable with your application via the [ingestion API](/api/v1/ingestLogsWithHeaders).
Parseable also supports OTel native data ingestion via Protobuf over HTTP. OpenTelemetry-compatible agents and libraries can send logs, metrics, and traces to the signal-specific OTLP endpoints: `/v1/logs`, `/v1/metrics`, and `/v1/traces`. Parseable derives the telemetry signal type from the endpoint path and uses `X-P-Stream` to choose the destination dataset.
## Detailed reference
For HTTP headers, schema behavior, log processing, batching, timestamps, staging, and production planning, see the [Ingestion Reference](/ingestion-reference).
# MCP Server (/mcp)
import { Step, Steps } from 'fumadocs-ui/components/steps';
Parseable MCP Server lets any MCP-compatible AI client work with your existing Parseable deployment. Instead of copying logs, schemas, alert details, or query output into a chat by hand, the client gets a clear tool surface for the work you would normally do in Prism or through APIs, such as finding datasets, checking schemas, running SQL or PromQL, reviewing alerts, and checking access information.
It does not ingest telemetry, store query results, or create another observability backend. The MCP server is a bridge. Your AI client talks to the MCP server, the MCP server calls Parseable with the API key you provide, and Parseable still enforces the same RBAC permissions that apply everywhere else.
## How it fits
```text
AI client
|
| MCP tool call
v
Parseable MCP Server
|
| Parseable API request with X-API-Key
v
Parseable
```
This is useful when you want an AI client to help with an investigation without copying logs, metrics, alert definitions, or schema details into the chat by hand. You can ask for recent errors in a dataset, inspect the fields available in a stream, run a bounded SQL query, check a PromQL expression, or review alert configuration from the same place where you are already working.
## Prerequisites
You need a Parseable Cloud account or a self-hosted Parseable instance that the MCP server can reach. You also need a Parseable API key with the right RBAC permissions for the datasets and actions you want the AI client to use.
For local setup, make sure Node.js and `npm` are available on the machine where your MCP client runs. For remote setup, use an MCP client that supports Streamable HTTP connections and custom headers.
## Connection options
| Option | How it works | Best for |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Local stdio | The AI client starts `@parseable/parseable-mcp-server` as a local subprocess and passes credentials through local configuration. | Claude Desktop, Claude Code, Cursor, VS Code, and other local MCP clients. |
| Remote HTTP | The AI client connects to an MCP endpoint over Streamable HTTP and sends the Parseable API key in request headers. | Hosted MCP deployments or clients that prefer a remote connector. |
Local setup is usually the easiest place to start. Remote setup is useful when a team wants one MCP endpoint to manage upgrades and network access centrally.
## Local setup
### Run the setup command
Run the installer from the machine where your MCP client is installed:
```bash
npx -y @parseable/parseable-mcp-server@latest init
```
The setup flow asks which client you want to configure, whether you are using Parseable Cloud or a self-hosted instance, and which API key should be used.
### Choose the Parseable target
For Parseable Cloud, the setup writes a hosted MCP connection for clients that support it. For self-hosted Parseable, provide the base URL of your Parseable instance, for example:
```text
https://parseable.example.com
```
Use the URL that your machine can actually reach. If your Parseable instance is behind a private network, the local MCP client must be on that same network or have access through your usual tunnel or VPN.
### Add the API key
Paste a Parseable API key when the setup prompts for it. The key should have only the permissions needed for the work you want the AI client to perform. For example, a read-only investigation workflow should not use a key that can change alerts or manage users.
After setup is complete, restart the MCP client so it can discover the new Parseable tools.
You can also run setup in a scripted form. For a self-hosted Parseable instance:
```bash
npx -y @parseable/parseable-mcp-server@latest init \
--client claude-desktop \
--mode self-hosted \
--url https://parseable.example.com \
--api-key "$PARSEABLE_API_KEY"
```
For Parseable Cloud:
```bash
npx -y @parseable/parseable-mcp-server@latest init \
--client cursor \
--mode cloud \
--api-key "$PARSEABLE_API_KEY"
```
The interactive setup shows the clients supported by the version you are installing.
## Manual local configuration
If you prefer to edit your MCP client configuration directly, add a local server entry like this:
```json
{
"mcpServers": {
"Parseable": {
"command": "npx",
"args": ["-y", "@parseable/parseable-mcp-server@latest"],
"env": {
"PARSEABLE_URL": "https://parseable.example.com",
"PARSEABLE_API_KEY": "your-api-key"
}
}
}
}
```
Some MCP clients use `servers` instead of `mcpServers`. Use the shape expected by your client.
## Remote HTTP setup
In remote mode, the client connects to an MCP endpoint and sends credentials as headers. Use HTTPS for remote connections.
For a self-hosted Parseable target:
```json
{
"mcpServers": {
"parseable": {
"type": "http",
"url": "https://mcp.example.com/mcp",
"headers": {
"X-Parseable-URL": "https://parseable.example.com",
"X-API-Key": "your-parseable-api-key"
}
}
}
}
```
For Parseable Cloud:
```json
{
"mcpServers": {
"parseable": {
"type": "http",
"url": "https://mcp.parseable.com/mcp",
"headers": {
"X-Parseable-Mode": "cloud",
"X-API-Key": "your-parseable-cloud-api-key"
}
}
}
}
```
Do not send credentials to an MCP endpoint you do not trust. The MCP server uses the supplied API key to call Parseable on your behalf.
## What the tools can do
Parseable MCP Server exposes 27 built-in tools around the same objects you already work with in Parseable. Some tools only read data, while a few can change alert state or create alert resources. The exact result still depends on the API key. If the key cannot access a dataset or action in Parseable, the MCP tool cannot access it either.
| Group | Tool | What it does |
| --------------------- | --------------------- | ------------------------------------------------------------------------------------------ |
| Datasets and events | `list_datasets` | Lists the datasets the API key can access. |
| | `get_dataset_info` | Shows dataset metadata such as retention, time range, and dataset configuration. |
| | `get_dataset_schema` | Returns the fields and types available in a dataset. |
| | `get_dataset_stats` | Returns dataset-level statistics such as event count and storage size. |
| | `sample_events` | Fetches a small, bounded sample of recent events from a dataset. |
| Queries | `query_sql` | Runs a read-only SQL query against a dataset with query controls applied. |
| | `query_promql` | Runs an instant or range PromQL query against a metrics dataset. |
| | `explain_query` | Explains a SQL query plan without executing the query itself. |
| Alerts | `list_alerts` | Lists alerts along with state, severity, tags, and related metadata. |
| | `get_alert` | Returns the full configuration for a specific alert. |
| | `list_alert_tags` | Lists the tags currently used across alerts. |
| | `enable_alert` | Enables an existing alert. |
| | `disable_alert` | Disables an existing alert. |
| | `evaluate_alert` | Evaluates an alert immediately. This can send real notifications if the condition matches. |
| | `create_alert` | Creates a new alert through a guided flow that collects the required alert fields. |
| Alert targets | `list_alert_targets` | Lists configured notification targets such as Slack, webhook, or Alertmanager targets. |
| | `get_alert_target` | Returns the full configuration for a specific alert target. |
| | `create_alert_target` | Creates a new notification target for alert delivery. |
| Access review | `list_users` | Lists users visible to the API key. |
| | `get_user_roles` | Shows the roles assigned to a specific user. |
| | `list_roles` | Lists the roles configured in Parseable. |
| | `get_role` | Shows the permissions attached to a specific role. |
| | `get_default_role` | Shows the default role assigned to new users. |
| Cluster and retention | `ping` | Checks whether the MCP server can reach Parseable and returns basic connectivity details. |
| | `get_cluster_status` | Shows node status for a distributed Parseable deployment. |
| | `get_cluster_metrics` | Returns high-level cluster metrics for ingest, query, and storage. |
| | `get_retention` | Shows the retention policy for a dataset. |
## Common workflows
### Investigate logs
A typical log investigation starts by listing datasets, inspecting the schema, and sampling a few recent records. After that, the AI client can help write a SQL query for a specific time window, run it, and explain the rows that came back.
Example prompts:
* `Show errors from the checkout dataset during the last 30 minutes.`
* `Which service produced the most 5xx responses today?`
* `Sample recent events from api-logs so I can understand the fields.`
### Query metrics
For metrics, use PromQL through the MCP server. You can ask for an instant query when you need the current value, or a range query when you want to inspect how a series changed over time.
Example prompts:
* `Run a range query for request latency over the last hour.`
* `Show the current error rate for the payments service.`
* `Explain this PromQL query before running it.`
### Work with alerts
The MCP server can help inspect existing alerts and alert targets. It can also help create alerts through a guided flow where the client collects the dataset, condition, evaluation window, severity, tags, and notification target before creation.
Be careful with alert evaluation. Evaluating an alert can trigger real notifications, so review the action before confirming it in your MCP client.
### Review access
You can ask the client to list users, inspect roles assigned to a user, and compare those roles with the default role. This is useful when an API key or user can see less data than expected.
## Query controls
SQL queries must include an explicit time window and are restricted to read-only use. The MCP server blocks common data-changing SQL operations and applies a row limit when the query does not provide one.
`sample_events` is also bounded, so it is meant for understanding the shape of a dataset rather than exporting data. Results remain subject to Parseable RBAC and the configured query limits.
For long investigations, keep time windows narrow first. Increase the range only after you know which dataset, service, or field is relevant.
## Security notes
* Use API keys with the smallest set of roles needed for the workflow, and store them in your MCP client configuration or a secrets manager. Do not commit them to source control.
* Use HTTPS for remote MCP and Parseable connections, and do not connect an AI client to an MCP endpoint you do not operate or trust.
* Review state-changing actions such as alert creation, alert enablement, alert disablement, and alert evaluation before confirming them.
* Rotate the API key in Parseable if it is exposed.
Parseable MCP Server does not store telemetry data, query history, result sets, credentials, or user preferences. Parseable remains the system that authenticates the key and enforces permissions.
## Troubleshooting
* If the tools do not appear in your client, restart the MCP client after running setup.
* If the local server does not start, check that `PARSEABLE_URL` and `PARSEABLE_API_KEY` are present in the generated client configuration.
* If you get `401` or `403`, check the API key first. `401` usually means the key is missing, invalid, or rotated. `403` means the key is valid but does not have permission for that dataset or action.
* If a dataset is missing, check the dataset name and the roles assigned to the API key.
* If PromQL does not work, confirm that you are querying a metrics dataset and that your Parseable deployment supports PromQL.
* If a query times out, reduce the time range or result size before increasing timeouts.
* If a remote connection fails, check the MCP endpoint URL, HTTPS configuration, request headers, and network reachability.
# SQL Reference (/query)
Parseable offers a PostgreSQL compatible SQL query interface to query log data. Users can choose to use the filter interface directly without having to deal with SQL at all. However for more complex queries and advanced users, Parseable offers a SQL query interface.
You can specify the query and the relevant time range for which you want this query to be run. The response is inclusive of both the start and end timestamps.
The filter interface is quite self explanatory, with options to filter by specific columns and values and also by time range. In this document, we'll cover more about the SQL API and its capabilities.
Check out the [Query API](/api/v1/queryLogStream).
## How does it work?
After parsing and creating the execution plan for a query, the Parseable query server uses the data manifest file to filter out the relevant Parquet files. The data manifest file is a JSON file that contains the specific column metadata for a whole day. The querier uses this file to filter the relevant Parquet files based on the query filters and the time range.
Only the relevant Parquet file paths are then added as a data source to custom table provider. Datafusion then efficiently reads the files via the GetRange S3 API, pulling only the very specific data needed for the query. This ensures that only the relevant data is read from the storage, reducing the query time and cost.
## Supported functions
Parseable supports a wide range of SQL functions - Aggregate, Window and Scalar functions. Refer the Apache Datafusion documentation for the complete list of supported functions and their usage.
* [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html)
* [Window Functions](https://datafusion.apache.org/user-guide/sql/window_functions.html)
* [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
## Query with regular expressions
This section provides examples of how to use regular expressions in Parseable queries.
* Match regular expression (Case Sensitive)
```sql
SELECT * FROM frontend where message ~ 'failing' LIMIT 9000;
```
* Match regular expression (Case Insensitive)
```sql
SELECT * FROM frontend where message ~* 'application' LIMIT 9000;
```
* Does not match regular expression (Case Sensitive)
```sql
SELECT * FROM frontend where message !~ 'started' LIMIT 9000;
```
* Does not match regular expression (Case Insensitive)
```sql
SELECT * FROM frontend where message !~* 'application' LIMIT 9000;
```
* Matches the beginning of the string (Case Insensitive)
```sql
SELECT * FROM frontend where message ~* '^a' LIMIT 9000;
```
* Matches the end of the string
```sql
SELECT * FROM frontend where message ~ 'failing$' LIMIT 9000;
```
* Matches numeric type data
```sql
SELECT * FROM frontend where uuid ~ '[0-9]' LIMIT 9000;
```
* Matches numeric type data (two digits)
```sql
SELECT * FROM frontend where uuid ~ '[0-9][0-9]' LIMIT 9000;
```
* Matches numeric type data (two digits)
```sql
SELECT * FROM frontend where uuid ~ '[0-9][0-9]' LIMIT 9000;
```
* Replace every instance of the numeric type data with the symbol \*
```sql
SELECT REGEXP_REPLACE(uuid,'[0-9]','*','g') FROM frontend LIMIT 9000;
```
* When a Regex is run against a string, the REGEXP\_MATCHES() function compares the two and returns the string that matches the pattern as a set.
```sql
SELECT REGEXP_MATCH(email,'@(.*)$') FROM frontend where email is not null LIMIT 10;
```
* Postgres regex numbers only: Use the REGEXP\_REPLACE() function to extract only the numbers from a string in PostgreSQL.
```sql
SELECT REGEXP_REPLACE(email,'\\D','','g') FROM frontend where email is not null LIMIT 10;
```
* Postgres regex split: SPLIT\_PART() function can split a string into many parts. To divide a string into several pieces, we must pass the String, the Delimiter, and the Field Number.
```sql
SELECT SPLIT_PART(email,'@',1) FROM frontend where email is not null LIMIT 10 -- return before @ from email;
SELECT SPLIT_PART(email,'@',2) FROM frontend where email is not null LIMIT 10 -- return after @ from email;
```
* Postgres Regex Remove Special Characters: Using the REGEXP\_REPLACE() function, all Special Characters from a supplied text can be eliminated.
```sql
SELECT REGEXP_REPLACE(email, '[^\\w]+','','g') FROM frontend where email is not null LIMIT 10;
```
* Functions and Operators in pattern matching: Like and other POSIX regular expressions are supported.
```sql
SELECT * FROM frontend where email LIKE '%test%' LIMIT 10;
SELECT * FROM frontend where email ~ '^test' LIMIT 10;
```
## Case sensitivity
Dataset column names are case sensitive. For example, if you send a log event like
```json
{
"foo": "bar",
"Foo": "bar"
}
```
Parseable will create two columns, `foo` and `Foo` in the schema. So, while querying, please refer to the fields as `foo` and `Foo` respectively. While querying, unquoted identifiers are converted to lowercase. To query column names with uppercase letters, they must be passed in double quotes. For example, when sending a query via the REST API, the following JSON payload will apply the `WHERE` condition to the column `Foo`:
```json
{
"query":"select * from dataset where \"Foo\"=bar",
"startTime":"2023-02-14T00:00:00+00:00",
"endTime":"2023-02-15T23:59:00+00:00"
}
```
If you're querying Parseable via Grafana UI (via the [data source plugin](https://github.com/parseablehq/parseable-datasource)), you can use the following query to query the column `Foo`:
```sql
SELECT * FROM dataset WHERE "Foo" = 'bar'
```
## Query analysis
In some cases, you may want to understand the query performance. To view the detailed query execution plan, use the `EXPLAIN ANALYZE` keyword in the query. For example, the following query will return the query execution plan and time taken per step.
```json
{
"query": "EXPLAIN ANALYZE SELECT * FROM frontend LIMIT 100",
"startTime": "2023-03-07T05:28:10.428Z",
"endTime": "2023-03-08T05:28:10.428Z"
}
```
## Response fields information with query results
To get the query result fields as a part of query API response, add the query parameter `fields=true` to the API call, e.g. `http://localhost:8000/api/v1/query?fields=true`.
For example, for a query like `select count(*) as count from app1`, with the query parameter added will respond like this:
```json
{
"fields": [
"count"
],
"records": [
{
"count": 2
}
]
}
```
# Release notes (/release-notes)
## v3.0
August 9, 2026
Parseable 3.0 focuses on making the platform safer, easier to operate, and better suited to teams running logs, metrics, traces, agents, runtimes, and model traffic in one production environment. It is a hardening release for existing deployments and a broader platform release for teams adopting Parseable across AI infrastructure and day-to-day observability workflows.
#### Security and operational hardening
This release tightens several areas that affect authentication, authorization, query safety, and data isolation. It includes fixes and improvements around API key security, SQL injection protection in list condition expressions, SSRF protection for alert targets, path traversal protection for local filesystem filters, dataset authorization, OAuth sync, OIDC redirects, and dependency and runtime hardening across the backend and Prism UI.
It also improves ingestion and trace endpoint behavior, alert query metadata, and OpenTelemetry metric handling, including exemplar and metric field fixes.
#### Tools for teams and agents
Parseable 3.0 adds more ways to work with telemetry outside the browser. The Parseable MCP Server gives MCP-compatible clients such as Claude, Cursor, VS Code, Codex, and other agentic tools a safer interface for inspecting datasets, schemas, logs, metrics, traces, alerts, and cluster health.
The `pb` command line interface also expands the terminal workflow for querying, tailing, inspecting profiles, and automating common telemetry tasks.
#### AI infrastructure observability
This release expands Parseable's AI infrastructure coverage across agent frameworks, sandboxes and runtimes, LLMs, and gateways. Agent framework integrations help teams inspect runs, model calls, tool calls, workflow steps, and failures. Runtime and sandbox integrations add visibility into execution, retries, logs, traces, and metrics. LLM and gateway integrations help with model latency, errors, token usage, routing, and provider behavior.
Together with Agent Observability in Prism, these integrations make it easier to follow an AI workload across the application, model, tool, runtime, and infrastructure layers.
#### Governance, Prism, and Kubernetes operations
Multi tenancy adds a clearer workspace boundary for shared deployments, keeping users, roles, API keys, datasets, dashboards, alerts, queries, and storage paths scoped to the right tenant.
Prism continues to improve the product surfaces used during investigations, including Agent Observability, APM service maps, dashboards, and reusable templates from the public `parseablehq/dashboards` repository.
On Kubernetes, Parseable 3.0 improves the installation and operations story through Helm chart updates and Parseable Auto Instrumentation (PAI). PAI uses the OpenTelemetry Operator and OpenTelemetry Collector to collect logs, metrics, traces, and Kubernetes events with less manual collector wiring, while still keeping the generated resources inspectable as normal Kubernetes objects.
For a detailed walkthrough, read the [Parseable 3.0 release recap](https://www.parseable.com/blog/parseable-3-release-recap). For the complete changelog, contributors, and tagged release assets, see the [GitHub v3.0.0 release](https://github.com/parseablehq/parseable/releases/tag/v3.0.0).
## v2.9
June 13, 2026
This release is focused on making Parseable easier to use for AI agents and day-to-day observability workflows. If you are already using Parseable for logs, metrics, or traces, the console should now feel more consistent across the product. If you are building agentic applications, you now get a dedicated way to understand how your agents behave in production.
#### Introducing Agent Observability
You can now inspect AI agent sessions directly in Parseable. This helps when you want to understand what prompt was used, which LLM calls were made, which tools were called, how many tokens were consumed, where latency came from, and whether errors or unusual sentiment showed up in the flow.
Instead of looking at disconnected logs or traces, you can follow the agent workflow in one place and see what happened step by step. This should make debugging slower responses, failed tool calls, or unexpected answers much easier.
#### Log context
Logs now include a **Log context** view in Prism. From any log row, open the row actions menu and choose **Show context** to see the records around that exact line. This helps when an error log does not explain enough on its own, because you can quickly check what happened before and after it, adjust the surrounding time window, jump back to the referenced line, and apply filters while staying in the same investigation.
#### Prism UI refresh
We have refreshed Prism across Logs, Metrics, Traces, Datasets, and the Home overview. The aim is simple: moving between observability views should feel familiar, with cleaner layouts getting in the way less often.
Alongside this, the release includes fixes and improvements around ingestion, API key ingestion, Helm configuration, S3 IRSA support, metric pruning, and field stats aggregation.
For a detailed walkthrough, read the [Parseable v2.9.0 release blog](https://www.parseable.com/blog/parseable-v290-release). For the complete changelog, contributors, and tagged release assets, see the [GitHub v2.9.0 release](https://github.com/parseablehq/parseable/releases/tag/v2.9.0).
## v2.8
May 30, 2026
### Hot Tier Improvements
Hot tier now has a more concurrent and tunable sync model. The refactor separates latest-data sync from historic-data sync, gives each stream its own background tasks, and makes object-store downloads safer through ranged, chunked downloads with partial-file staging.
**What's new:**
* Historic data sync and latest data sync now run as separate hot-tier tasks.
* Each stream gets its own hot-tier background sync tasks, improving concurrency across streams.
* Hot-tier file downloads are parallelized and chunked, with partial-file staging before final rename.
* Operators can tune hot-tier object-store access with `P_HOT_TIER_DOWNLOAD_CHUNK_SIZE`, `P_HOT_TIER_DOWNLOAD_CONCURRENCY`, `P_HOT_TIER_FILES_PER_STREAM_CONCURRENCY`, `P_HOT_TIER_LATEST_MINUTES`, and `P_HOT_TIER_HISTORIC_SYNC_MINUTES`.
### Metrics and PromQL Improvements
Metrics workflows get better series handling, better read behavior, and a smoother PromQL experience in the console. This release adds deterministic per-series hashing during ingestion, improves Parquet layout for metric reads, and expands PromQL support in Prism.
**What's new:**
* OpenTelemetry metrics known fields now include quantile value fields.
* Metrics ingestion now writes a stable per-series hash, helping Parseable identify physical metric series consistently.
* Metrics batches are sorted by metric name before Parquet write, improving pruning for metric-name based reads.
* Prism adds PromQL querying, PromQL fixes, metric row filtering and copy actions, improved no-data views, and tooltip fixes in Metrics Explore.
### Query and Ingestion Reliability
Parseable now surfaces more ingestion failures instead of hiding them behind panics or failed background operations. Query streaming and DataFusion execution tuning were also improved for larger workloads and multi-partition result delivery.
**What's new:**
* Streaming query merge was redesigned for more stable multi-partition result delivery.
* Ingestion and staging operations now return and propagate errors instead of relying on unwraps or hidden failures.
* Query execution adds tuning for target partitions, Parquet filtering, page index use, metadata sizing, predicate cache size, and execution batch sizing.
### Tracing and Observability Improvements
Tracing is easier to enable in local and development environments, and operators get richer visibility into storage and hot-tier paths. Parseable can now export application traces directly to stdout in JSON format when the OpenTelemetry endpoint is configured for stdout.
**What's new:**
* Set `OTEL_EXPORTER_OTLP_ENDPOINT=stdout` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=stdout` to export application traces to stdout.
* Tracing enablement and stdout behavior are simpler and clearer, including endpoint and header handling.
* Hot-tier, object-store, startup sync, and storage paths now include richer tracing and metrics for operator visibility.
### Console and User Experience Improvements
Prism received reliability and usability updates across Logs, Metrics, Traces, SQL, Alerts, and Errors. The changes reduce visual inconsistencies, improve filtering and copy workflows, and make empty or error states more consistent.
**What's new:**
* Empty-state handling is more consistent, including preventing flashes before query loaders in Logs, Metrics, and Traces.
* Trace views now align start times more consistently and show timestamps in bubble chart tooltips.
* Traces and metrics tables now support include, exclude, filter, and copy actions where relevant.
* SQL autocomplete quote handling, sidebar collapse behavior, timezone changes, Errors page dataset reset, and non-exception error details were improved, with expanded Playwright coverage across core console workflows.
## v2.7
May 8, 2026
### APM Support
APM is now a dedicated workflow inside Parseable. With this release, a new **Services** view is available in the sidebar, giving teams a service-level entry point for application performance monitoring across traces, errors, request volume, and latency.
Click into any service to open a scoped APM view with **Insights**, **Traces**, and **Errors** tabs, so teams can move from service health to request-level debugging without switching tools.
What's new:
* Browse all traced services from a dedicated Services page.
* View requests, error percentage, and latency per service.
* Drill into service-specific Insights, Traces, and Errors.
* Open services and APM routes through deep links from alerts, runbooks, or external tools.
### Ingestion Traces
Ingestion is now a fully traced workflow inside Parseable. The server emits OpenTelemetry spans across the ingestion path, from API intake to schema inference, record-batch conversion, staging, flush, and object-store upload.
This gives operators deeper visibility into ingest performance, background sync cycles, and bottlenecks across high-throughput telemetry pipelines.
What's new:
* Trace ingestion end to end with OpenTelemetry spans across the full ingest path.
* Export ingestion traces to any OpenTelemetry collector with standard configuration.
* Tune actix server behavior with new CLI args and matching `P_ACTIX_*` environment variables.
* Improve high-volume ingest with batched schema inference, parallel record processing, optimized byte-size counting, and safer sync-cycle guards.
### PromQL Support
Parseable now includes built-in PromQL support for metrics streams through Prometheus-compatible HTTP APIs under `/prometheus/api/v1`. Existing Grafana dashboards and Prometheus-compatible tooling can point to Parseable and query metrics without major changes.
This brings PromQL-native metric analysis into Parseable while keeping logs, traces, and metrics available in one observability platform.
What's new:
* Query metrics with PromQL through instant query, range query, series, labels, and label-values endpoints.
* Use existing Grafana Prometheus data sources with Parseable.
* Run PromQL functions, aggregations, binary operators, vector matching, and subqueries.
* Execute heavier queries safely with query splitting, result caching, distributed execution, memory guards, and cardinality analysis endpoints.
### Access Management and API Keys
Access Management is now the central place to manage users, roles, and API keys in Parseable. With this release, teams can create scoped API keys for programmatic access without relying on personal user credentials.
API keys are built for log forwarders, CLI tools, automation scripts, AI agents, dashboards, CI jobs, and other workloads where Basic Auth or OIDC is not the right fit. Each key inherits permissions from assigned roles and can be used through the `X-API-Key` header.
What's new:
* Create and manage API keys from Access Management.
* Assign one or more roles to every API key.
* Use dataset-scoped keys for ingestion agents, query agents, dashboards, and automation workflows.
* Revoke keys independently without affecting users or other workloads.
* Use API keys for supported ingestion and query workflows through the `X-API-Key` header.
## v2.5
December 8, 2025
**Parseable v2.5.0** is here! This major update significantly expands our platform capabilities, introducing a dedicated **Trace View**, a **Metrics View**, and powerful AI-native integrations connecting Keystone with the alerting ecosystem.
### 1. Trace View Implementation
We have unlocked a new dimension of observability with the introduction of the **Trace View**. This feature is designed to help you visualize the complete lifecycle of requests as they traverse your distributed systems, making it easier than ever to diagnose latency issues and bottlenecks.
* **Full Trace Visualization:** Users can now view comprehensive waterfall charts of traces, providing deep visibility into service interactions.
* **Enhanced Navigation:** We added intuitive breadcrumbs to the Traces Span detail sheet, allowing for seamless navigation through complex span hierarchies.
* **OpenTelemetry Standardization:** Trace field names for events and links have been updated to align strictly with OpenTelemetry standards, ensuring consistent data ingestion and better interoperability.
### 2. Metrics View
The Metrics experience has been overhauled to provide deeper insights and a more fluid user experience.
* **Data Consistency:** Attribute field names have been standardized across all metric types, simplifying queries and ensuring consistency across your data.
### 3. Keystone Integration with Alerts
We are bringing AI-native capabilities closer to your incident response workflow. Keystone is now tightly integrated with the alerting system.
* **Automated Investigation:** You can now initiate a Keystone analysis session directly from a triggered alert. This seamless handoff reduces the time between detecting an anomaly and understanding its root cause.
### 4. SQL Editor & Visualization
We have enhanced the SQL Editor to support more complex data exploration and reporting needs.
* **Table Chart Support:** We introduced the **'Table' chart type**, allowing you to visualize raw data rows directly within your dashboards for detailed reporting.
* **Multi-Dataset Expansion:** The SQL editor now supports up to **3 distinct datasets** simultaneously, enabling more complex joins and comparative analysis.
* **UX Enhancements:**
* Added a **"Clear All"** button to instantly reset GroupBy selections, speeding up exploratory data analysis.
* Improved state management for SQL editor tabs to prevent data loss when switching contexts.
* Refined the `dbName` logic to better handle complex subqueries.
### 5. Alerting Improvements
The alerting engine has been optimized for performance and better context.
* **State History:** Parseable now saves the history of alert states, providing a valuable audit trail of when alerts fired, resolved, or changed status.
* **Contextual Alerts:** You can now include the `metric_name` context when creating alerts, making notifications more informative at a glance.
* **Performance & Clarity:** We optimized the underlying queries used for alert creation to reduce load and removed redundant text from notification messages for cleaner, easier-to-read alerts.
### 6. Bug Fixes
* **SQL Editor:** Fixed an issue where error messages were not being displayed when a query failed, ensuring users now see the correct feedback.
* **Visualization Logic:** Refined query data checks in the `GaugeStyleSection` to correctly respect `rangeTo` values for accurate gauge rendering.
* **Recents Management:** The "Recents" list now automatically cleans itself by removing entries that are no longer valid.
* **Timezones:** Fixed an issue with URL parameter handling to ensure selected timezones persist correctly during navigation.
## v2.4.3
September 22, 2025
We're excited to announce Parseable v2.4.3. This release was focussed on improving the user experience of using SQL via our SQL editor.
### New SQL editor
The SQL editor has a new sidebar to showcase all the columns and their types for fast lookups. After the query is run and results are available, you can now use the chart view customization options and add these charts to dashboards directly.
AI assistant, saved queries are now rearranged to the right side of the screen for better visibility and access.
### Create alerts from Dashboard
Visualization gives you a good idea of patterns forming, and many times you want to be alerted when these patterns cross a certain threshold. You can now create alerts directly from the dashboard.
Click on the "three dots" button in a dashboard tile and select "Create Alert" from the dropdown menu. You will be redirected to the Alert Builder page, where your current filters and group-by options are automatically applied. Once satisfied with the configuration, click "Create" to activate your alert.
### Investigation flows
As you analyze the data in your dashboard, you may identify specific trends or anomalies that require further exploration. Our new investigation flows make it easy to dive deeper into the data. You can now drill down into the data from the Dashboard or a triggered Alert.
In a dashboard tile:
* Select the "Investigate" option under the "three dots" button in chart.
* You'll be redirected to the Explore Page of the dataset with the same filters and group by options as the chart.
From a triggered Alert:
* Select the "Investigate" option from the Triggered Alert page.
* You'll be redirected to the Explore Page of the dataset with the same filters and group by options as the chart along with the timeline of the alert.
## v2.4.1
August 18, 2025
We're thrilled to announce Parseable v2.4.1, packed with powerful new features that enhance monitoring, visualization, and data ingestion capabilities. This release introduces advanced alerting mechanisms, expanded dashboard visualization options, and native Protocol Buffer support.
### 1. Advanced Alert Types
Building on our alerting capabilities, v2.4.1 introduces three sophisticated alert types to help you proactively monitor your systems. [Learn more about Alerts](/user-guide/alerting)
**Threshold Alerts**
* Set precise boundaries for your metrics.
* Trigger alerts when values exceed defined limits.
* Example: Alert when response time > 500ms.
* Configure multiple conditions with "Filter" and "Group by".
**Anomaly Detection**
* Leverage machine learning to automatically identify unusual patterns.
* No manual threshold configuration required.
* Detects deviations from normal behavior patterns.
* Reduces alert fatigue by focusing on genuine anomalies.
**Forecasting Alerts**
* Predict future metric values based on historical trends.
* Alert on expected limit violations before they occur.
* Enable proactive capacity planning.
* Example: Alert when disk space is predicted to run out in 24 hours.
### 2. New Dashboard Visualization Types
Expand your data visualization capabilities with four new chart types. [Learn more about Dashboards](/user-guide/dashboards)
**Pie Charts**
* Visualize proportional data at a glance.
* Perfect for showing distribution across categories.
* Interactive legends with click-to-filter functionality
**Donut Charts**
* Modern alternative to pie charts with hollow center
* Display key metrics in the center space
* Ideal for showing completion percentages
**Gauge Charts**
* Real-time performance indicators
* Configurable color-coded segments for thresholds
* Support for custom ranges and units
* Perfect for KPI monitoring (CPU usage, memory utilization)
**Query Value**
* Display single, prominent metric values
* Ideal for highlighting critical KPIs
* Support for auto-formatting and precision control
* Color coding based on thresholds
**Enhanced Styling Options:**
* Custom color palettes for all chart types
* Configurable units and number formatting
* Threshold-based color coding
* Responsive design for all screen sizes
### 3. Protocol Buffer (Protobuf) Support
Native support for Protocol Buffer format significantly improves data ingestion efficiency:
**Key Benefits:**
* **Prometheus Compatibility Path**: Protobuf support is a crucial step towards enabling full Prometheus compatibility.
* **Reduced Bandwidth**: Up to 50% smaller payload sizes compared to JSON
* **Faster Processing**: Binary format enables quicker parsing
* **Type Safety**: Strong typing prevents data corruption
**Implementation:**
```bash
# Send protobuf encoded logs
curl -X POST https://parseable.example.com/api/v1/ingest \
-H "Content-Type: application/x-protobuf" \
-H "X-P-Stream: my-dataset" \
--data-binary @logs.pb
```
### 4. Enhanced Column Support
Parseable now supports up to **1000 columns** per dataset, a significant increase from previous limits:
* **Maximum Columns**: Support for up to 1000 columns per dataset.
* **Recommended Threshold**: Beyond 800 columns, consider splitting the dataset into multiple datasets based on schema similarity or query patterns for optimal performance.
* **Hard Limit**: The server will reject ingestion requests with more than 1000 columns.
### Upgrade Instructions
```bash
# Docker
docker pull parseable/parseable:v2.4.1
# Kubernetes
kubectl set image deployment/parseable parseable=parseable/parseable:v2.4.1
```
## v2.4.0
July 22, 2025
It’s that time again, a new Parseable release, and this one’s big. We’re moving up from v2.3.x to v2.4.0. Read on for all the highlights, new features, and tips to get the most out of v2.4.0
### 1. AI Powered Summarization
We're excited to introduce AI-powered summarization, a powerful new feature designed to simplify data analysis and debugging in Parseable.
**How It Works:**
* Select any dataset within Parseable.
* Click on `Summarize my data` to automatically generate a concise overview of your data.
* The AI identifies key patterns, anomalies, and potential faults in your dataset.
* Receive actionable recommendations and SQL queries to drill deeper into specific issues directly from the summary.
**Benefits:**
* Quickly gain insights without manually combing through extensive datasets.
* Reduce troubleshooting time by pinpointing anomalies and root causes effortlessly.
* Simplify collaboration by sharing clear, concise summaries across your team.
* Leverage AI-driven recommendations to proactively address issues before they escalate.
**Example:**
Imagine you’re troubleshooting elevated error rates in your logs. With AI-powered summarization, Parseable instantly highlights:
* An unusual spike in errors between specific timestamps.
* Affected services and hosts.
* Suggested SQL queries to drill down further, such as:
```sql
SELECT host, COUNT(*) as error_count
FROM logs
WHERE status='error' AND timestamp BETWEEN '2025-07-20T00:00:00' AND '2025-07-20T06:00:00'
GROUP BY host
ORDER BY error_count DESC;
```
### 2. User Groups
We’ve added **User Groups** to Parseable’s RBAC system to make access management enterprise ready.
**How It Works:**
* Create groups for teams, projects, or any logical set of users.
* Assign roles (Admin, Editor, Writer, Reader, Ingestor) to the entire group in one go.
* Users in a group instantly inherit all permissions assigned to that group.
**All previous RBAC rules remain unchanged:**
* Admin/Editor: Roles apply platform-wide (Editor is owner of all datasets but can’t manage users/groups or see cluster details).
* Writer/Reader: Assignable at the resource level—fine-grained access for specific datasets or resources.
* Ingestor: Can only be assigned at the dataset level (for managing ingest operations).
**Benefits:**
* No more repetitive, user-by-user role assignments.
* Simplifies audits and permission reviews.
**Example:**
Say your SRE team needs full write access to all logs, while support staff should only view error logs:
* Create an “SRE” group and give it Writer on all datasets.
* Create a “Support” group with Reader on the errors dataset.
* Onboard a new SRE? Just add them to the group—done.
### 3. Introducing Dashboards in Parseable
Dashboards are here! Now you can visualize your data in Parseable, your way.
**How It Works:**
* **Start from the Side Nav:** Head over to Dashboards in the sidebar and hit Create New.
* **Name & Tag:** Give your dashboard a name and a tag (for easy organizing), then click Create Dashboard.
* **Build Your Canvas:** Select your new dashboard to land on the dashboard canvas. Click Add Tiles to start building charts.
* **Chart Builder Flow:** Choose your chart type: Timeseries, Line, Bar, or Area.
* Select the dataset you want to visualize.
* See a live preview as you build.
* Pick the plot fields from your dataset, aggregate/group/sort on the x-axis, and set your y-axis (typically count, time, or number).
* Want a quick table instead? Just click Table in the top right, skip the chart setup!
* Style your chart as you like, then hit Create.
* **Save and Favorite:** Add as many tiles as you need, then save your dashboard. You can also mark dashboards as favorites for quick access later.
**Benefits:**
* Instantly visualize logs and metrics, no need to export or use external tools.
* Flexible charting with live previews and quick customization.
* Organize dashboards by tags, share them with your team, and pin your favorites.
**Example:**
* Want to monitor API error rates over time?
* Create a dashboard called “API Health,” tag it “ops.”
* Add a Timeseries chart for error counts from your logs.
* Save it, mark as favorite—and you’ve got a live, auto-updating view for your team.
### 4. All-New Onboarding Wizard
Getting started with Parseable just got a whole lot easier. With this release, we’ve overhauled the first-time user experience:
**How It Works:**
* Right after installation, you’re greeted by an onboarding wizard.
* In just 3 minutes, the wizard sets up everything you need to explore Parseable:
* **Demo Data:** A ready-made dataset of 5,000 events.
* **Pre-configured Alerts:** See how to catch spikes and anomalies out of the box.
* **Interactive Dashboards:** Instantly visualize and play with your sample data.
* **Practical Saved Filters:** Jump straight into real-world troubleshooting examples.
* **Example SQL Queries:** Learn how to slice and dice your data with practical queries.
### 5. Native GCP Object Store Support
Parseable now supports Google Cloud Storage (GCS) as a native object store backend.
**How It Works:**
* Set the following environment variables when starting Parseable:
```bash
GOOGLE_APPLICATION_CREDENTIALS=/parseable/svc/${GCS_CREDENTIALS_FILE:-key.json}
P_GCS_BUCKET=
```
* Parseable will use your specified GCS bucket for all data storage.
## v2.3.5
June 26, 2025
We’re excited to roll out Parseable **v2.3.5**, a release packed with features that make exploring, debugging, and managing your observability data simpler and more powerful than ever. Here’s everything new:
### 1. **Filters & Group By: Slice, Dice, and Save Your Logs**
**What’s new?**
You can now filter and group logs directly from the log explorer using a flexible, built-in filter panel. Not only can you **filter logs by any field** (e.g., `service=payments`, `env=prod`), you can also **group them by any field** to spot patterns and outliers fast.
**How it works:**
* **Build a filter:** Type in the filter panel just like a search box—e.g., `status=error AND region=us-west-2`.
* **Add a group by:** Instantly group results by `error_code` or `user_id` to see which issues are most frequent or which users are affected.
* **Save filter views:** Liked what you built? Click “Save to Library” and name the view (e.g., `Payments Errors Last Week`). Next time, just apply it from the Library to debug similar issues—no more rebuilding filters from scratch.
**Example:**
Troubleshooting an incident in the `payments` service?
* Filter: `service=payments AND status=error`
* Group by: `error_code`
* Result: A quick tally of error types, so you can spot if a particular error is spiking.
**Benefits:**
* Saved filters help you build a library of ready-to-use “debug recipes.”
* The panel is as easy as typing, no complex query syntax required.
* Perfect for SREs and devs who want fast, repeatable root cause analysis.
### 2. **AI-Powered Forecasts—Now Available for Filters**
We’ve extended our **AI-based ingestion forecasting** to work with *any* filter you apply. This means you can select a filtered view (e.g., just logs for a specific team or region) and get **automatic predictions** for future log volumes based on your selection.
**Benefits:**
* Filter logs to `env=prod AND region=eu-central-1`
* Instantly see a chart forecasting ingestion volume for just those logs
* Plan ahead for traffic spikes, storage needs, or incident response
* No more guessing how much your filtered logs will grow over time!
### 3. **Field Stats: Occurrence Counts & Drill Down Made Easy**
You can now see detailed stats for any field in your telemetry, right in the UI. For every field (like `status`, `endpoint`, `customer_id`), Parseable shows:
* **Count of unique values** (e.g., how many different status codes?)
* **Frequency/occurrence** of each value (e.g., how many times did `status=error` appear?)
* **Order and drill down**: Click any value to instantly filter logs to just that slice.
**Example:**
Investigating API failures?
* Click `status` → see all possible values (e.g., `200`, `404`, `500`) and how often each occurred
* Click on `500` → now you’re only seeing server errors, ready to debug further
* Combine with group by `endpoint` to see which routes are most affected
Here's a demo video showcasing the filters feature:
### 4. **Multi-Role Support for OIDC Users**
Teams often need flexible, fine-grained access control—now Parseable’s OIDC integration supports **assigning multiple roles to a single user**.
**Benefits:**
* Give a DevOps engineer both `admin` and `reader` roles for different datasets
* Grant a user `editor` + `reader` permissions, so they can review logs across datasets without manual role shuffling
* All roles are respected in UI and API, just assign them in your OIDC provider and Parseable does the rest
## v2.3.3
June 17, 2025
### Performance Improvements
* **Optimized Conversion Task Assignment**: Improved the conversion task system to assign a definite set of arrow files for each conversion task, preventing multiple tasks from processing the same files during heavy loads.
* **Parallel Object Store Sync**: Enhanced object store synchronization to run in parallel for each dataset, significantly improving performance under heavy loads compared to the previous sequential approach.
* **Full Core Utilization**: Increased thread allocation for conversion and sync task handlers from 2 threads to utilizing all available server cores, resulting in better overall performance of conversion and sync workflows.
* **Startup Recovery Process**: Added a conversion and sync task that runs at server startup to process any pending files that weren't processed due to unplanned server shutdowns, improving system resilience.
### Bug Fixes
* **Fixed Poison Errors**: Resolved issues with poison errors that occurred when write locks acquired by ingestion threads panicked.
## v2.3.2
June 3, 2025
### New Features
Today is a big milestone for us at Parseable. With the release of Parseable v2.3.2, we're not just adding new features, we're officially welcoming AI to the heart of your observability workflows. This release marks our very first step into AI-powered developer tools, making it even easier to explore, query, and anticipate your data. Whether you're a SQL cloud or a newcomer, or simply want to stay ahead of your growing telemetry, v2.3.2 brings smarter, more intuitive capabilities straight into your workflow.
### AI Assistant for SQL Editor
If you've ever stared at a blinking cursor in the SQL editor, wondering how to build the right query for your latest troubleshooting adventure, you're not alone. With Parseable v2.3.2, you now have an AI-powered sidekick right inside your SQL editor.
Here's what's new:
* **Query in Plain English**: Simply tell the assistant what you want, "show me all error logs for the last hour grouped by host," and it'll generate the SQL for you.
* **Multiple LLM Providers**: We know teams have preferences and constraints, so you can pick your favorite large language model (LLM) from the settings page. Out of the box, we support:
* OpenAI GPT
* Anthropic Claude
* **Plug-and-Play Configuration**: No extra setup headaches. Add your provider key, set your preferences, and you're ready to start querying.
This feature isn't just about convenience, it's about making data exploration accessible to everyone on your team, from SREs to product managers. The AI Assistant means faster queries, less context-switching, and more time for actual problem-solving.
### Chat with AI
At the bottom of your SQL editor, you'll now see a "Generate with AI" button. Use this to ask anything from:
* "Show me the number of 5xx errors grouped by host for the last hour"
* "Fix this query, it's giving a syntax error"
* "Summarize response statuses per environment tag"
The assistant understands context from your current datasets and query history, giving you results tailored to what you're working on. It also comes in handy when onboarding new team members. Junior engineers or analysts often need help crafting their first few queries. With the AI assistant, they can describe what they're looking for in plain language, like "Find the top 5 most common user agents in the backend table and how many times each appears?" and get an immediate, working query they can tweak and learn from.
Even seasoned users benefit when dealing with complex joins, window functions, or new datasets they haven't touched before. You can prompt the assistant with "Can you help me write a query that shows trends in status codes over time, grouped by host and tag-environment, and highlights spikes in error responses?" and get a scaffolded SQL with placeholders or editable sections you can refine.
In high-pressure situations, like an ongoing incident, the AI assistant acts as a query co-pilot. It reduces cognitive load, speeds up iteration, and lowers the risk of human error when time is critical. And because you can configure your preferred LLM in the settings, teams can balance performance, compliance, and cost based on their stack.
### Chat History
Every prompt and response is automatically saved. Click the "History" tab in the assistant panel to revisit, rerun, or refine past prompts. To make your workflow even smoother, Parseable also stores the full history of your AI interactions. Every prompt you've sent to the assistant, whether it's fixing a query, generating one from scratch, or just exploring a new dataset is saved alongside the responses. This means you can revisit past queries, reuse them in future investigations, or track how a particular issue evolved over time.
This is especially useful during incident retrospectives or recurring analytics, you'll never lose a good query again.
### The Library (Saved SQL, Now Smarter)
With Parseable 2.3.0, saved SQL queries live in the new Library. You can now:
* Save queries you want to reuse
* Edit queries with the help of AI
* Run them directly from the library pane
* Explain what a query does using the assistant
The library is your personal or team-wide knowledge base for telemetry insights.
### Failed Query? Let AI Fix It
Let's say you're investigating a sudden spike in latency reported by your alerting system. You jump into Parseable, type a rough query to check average response times, but something's off, either the field doesn't exist or the aggregation logic is incorrect. Instead of trial-and-error debugging, you can now ask the AI assistant:
"Fix with AI"
The assistant returns a corrected version using the correct percentile function and known schema fields, saving minutes or even hours of digging through docs and schema dumps.
This turns errors into learning moments and lets you recover faster in the middle of a high-pressure incident.
### Forecasting for Log Ingestion
Ever wish you could see into the future of your observability pipeline? Along with AI assisted SQL queries, now you can forecast the future ingestion load, with Parseable's new forecasting for log ingestion.
What this means for you:
* **Data-Driven Forecasts**: Our forecasting engine uses your recent log ingestion patterns to predict what's coming next. Enable it, and you'll see projected ingestion volumes right in your dashboards.
* **Smarter Planning**: Spot upcoming spikes before they happen. Plan capacity, staffing, and alerting based on real forecasts, not just yesterday's numbers.
* **Visualized Right Where You Need It**: Forecasts appear directly in the Parseable Explore UI, so you can compare historical and predicted loads at a glance.
Whether you're scaling infrastructure or just want to avoid surprises, forecasting helps you move from reactive to proactive.
## v2.3.1
### Highlights
* Global search datasets, saved filters, SQL snippets… anything Parseable tracks.
* 𝗣\_𝗠𝗔𝗫\_𝗙𝗟𝗔𝗧𝗧𝗘𝗡\_𝗟𝗘𝗩𝗘𝗟 env var to configure nested lists; default depth is 10, but you call configure as per your use-case.
## v2.3.0
### Enhancements
* SQL editor visualisation - SQL editor page allows slicing and dicing data exactly as you wish. We now added visualisation capabilities to the SQL editor. After running a query, you can either inspect the raw rows or quickly switch to a visual chart for better pattern recognition.
* Home chart improvements - Landing page shows a high level overview of your Parseable instance. We have now enhanced the overview section with deeper insights into total ingestion, compression and storage details.
* We’ve also improved the search API response in the sidebar, so you can quickly look up saved filters, alerts, datasets among others.
* Streaming response - We have now added support for streaming response for query. This improves responsiveness for queries that involve downloading larger data chunks. This is available as a flag for the query API. Refer to the Parseable docs on how to use this API.
### Enterprise updates
* Schema detection improvements: Parseable Enterprise edition supports automatic schema detection. The server identifies formats sent by a client when a header is set. Server also validates if the format is actually the same as set by header. This allows a consistent experience on the server side - so users can reliably build dashboards, alerts and other visualisations without being concerned about an offending client.
* In some cases where the format doesn’t match the header specified, we now added support for a column that marks this result. You can now decide to use the mismatched data or not based on the value of p\_format\_verified.
* LLM configuration across roles - As we proceed our journey of adding native Large Language Model (LLM) support to improve MTTR and leverage user’s time better. We have now added user specific actions to add LLM config.
* This means every user can have a separate LLM key and related config and use their config for their tasks with complete isolation from other user’s actions, history etc.
### Bug Fixes
* Timezone handling explore page - Fix timezone management on log explore page ensuring accurate log display aligned to user-selected timezones.
* Settings page access - Ensured the Settings page is consistently accessible to all users, addressing previous visibility inconsistencies.
* Loaders & error states - Added loading indicators and improved error messaging on user and dataset pages, resulting in a clearer user experience during data fetching and error scenarios.
* Distinct SQL query in Datasets API - Corrected SQL queries to accurately fetch distinct values, improving dataset API responses.
* Date stats deletion fix - Fixed deletion issues in date-level statistics, enhancing data consistency.
* Date level stats fix - Addressed inaccuracies in date-level statistics calculation, ensuring reliable data insights.
* Reject events when the dataset has too many fields - Implemented validation to reject events with more than 250 fields, improving platform stability.
* Allow invalid certificates - Allowed intra-cluster communication using invalid certificates, easing integration scenarios.
### Enterprise fixes
* ALB Log Regex Fix - Resolved regex parsing issues for ALB logs, ensuring accurate log ingestion.
* Create Index Form Validation - Fixed validation logic in the Create Index form, resolving form submission issues and improving user feedback during index creation.
# Slackbot (/slack-bot)
The Parseable Slackbot brings Parseable investigation into the Slack
conversation where an incident, alert, or debugging thread is already happening.
Instead of switching context to copy a query into a browser, a user can connect
their Parseable account from Slack and ask questions with `@parseable`.
The bot is available for Parseable OSS, Cloud, and Enterprise users. There is no
separate setup flow for each edition. Slackbot works through the Parseable MCP
path and is accessed from Slack chat. Slack is the place where the user asks the
question, MCP provides the tool
interface, and Parseable returns data from logs, metrics, traces, alerts, and
other supported resources.
## How It Works
```text
Slack user
|
| /parseable connect
| API key
v
Parseable Slackbot
|
| MCP request
v
Parseable MCP Server
|
| Parseable API request
v
Parseable Cloud, Enterprise, or OSS
```
The bot does not ingest telemetry or create a new data store. It calls Parseable
through MCP and returns the answer in Slack. Parseable still authenticates the
API key and enforces the permissions attached to that key.
This works across Parseable Cloud, Parseable Enterprise, and self-hosted OSS
deployments. The exact actions available in Slack depend on the Parseable API key
used during connection.
## What You Can Ask
Once connected, mention the bot in a Slack channel or thread and ask a question:
```text
@parseable show me errors from api-gateway in the last 10 minutes
```
You can use the bot for the same kind of investigation you would normally do in
Parseable or through the MCP Server:
* Find datasets and inspect schema.
* Query logs with a bounded time range.
* Run PromQL for metrics.
* Look up traces and service behavior.
* Inspect alerts and alert targets.
* Review users, roles, and access information when the API key allows it.
* Create or evaluate alerts when the connected key has those permissions and the
action is confirmed.
For read-only workflows, connect with a read-only Parseable API key.
## Prerequisites
Before you start, make sure you have:
* Access to the Slack workspace where the Parseable bot is installed.
* A Parseable Cloud, Enterprise, or OSS deployment.
* A Parseable API key with the permissions you want the bot to use.
## Connect From Slack
Run the connect command in Slack:
```text
/parseable connect
```
The bot asks for the details it needs to reach Parseable. For Parseable Cloud,
provide the API key for the workspace you want to query. For Enterprise or OSS,
provide the Parseable URL and API key if the bot asks for both.
After the connection is saved, ask a small test question:
```text
@parseable list datasets
```
If the bot returns the dataset list, the Slack workspace is connected to
Parseable and ready for investigation.
## Ask Questions in Context
The bot is most useful when the question already has context. You can ask inside
an alert thread, an incident channel, or a service-specific channel.
Examples:
```text
@parseable what changed in checkout errors during the last 30 minutes?
```
```text
@parseable query payment-api logs for 5xx errors in the last hour
```
```text
@parseable show request latency for frontend over the last 15 minutes
```
```text
@parseable summarize recent alerts for production
```
Keep the first question narrow. Start with a dataset, service name, error type,
or time window. Once the bot finds the right direction, ask follow-up questions
in the same thread.
## Permissions and Safety
Slackbot follows the same security model as the MCP Server. It does not get
special access to Parseable. If the connected API key cannot access a dataset,
the bot cannot access it either.
Use these rules for production work:
* Use least-privilege API keys.
* Prefer read-only keys for investigation channels.
* Use a separate key for workflows that need alert creation or alert evaluation.
* Rotate the key if it is shared in the wrong channel or exposed outside Slack.
* Avoid pasting API keys in public channels.
Slack permissions are separate from Parseable permissions. Slack scopes control
what Slackbot can do inside Slack. The connected Parseable API key controls what
it can access inside Parseable.
Slackbot uses the following bot token scopes:
| OAuth scope | Description |
| -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`app_mentions:read`](https://docs.slack.dev/reference/scopes/app_mentions:read) | View messages that directly mention `@Parseable` in conversations that the app is in. |
| [`channels:history`](https://docs.slack.dev/reference/scopes/channels:history) | View messages and other content in public channels that `Parseable` has been added to. |
| [`channels:read`](https://docs.slack.dev/reference/scopes/channels:read) | View basic information about public channels in a workspace. |
| [`chat:write`](https://docs.slack.dev/reference/scopes/chat:write) | Send messages as `@Parseable`. |
| [`commands`](https://docs.slack.dev/reference/scopes/commands) | Add slash commands, including `/parseable connect`. |
| [`groups:history`](https://docs.slack.dev/reference/scopes/groups:history) | View messages and other content in private channels that `Parseable` has been added to. |
| [`im:history`](https://docs.slack.dev/reference/scopes/im:history) | View messages and other content in direct messages that `Parseable` has been added to. |
| [`mpim:history`](https://docs.slack.dev/reference/scopes/mpim:history) | View messages and other content in group direct messages that `Parseable` has been added to. |
| [`team:read`](https://docs.slack.dev/reference/scopes/team:read) | View the name, email domain, and icon for workspaces that `Parseable` is connected to. |
| [`users:read`](https://docs.slack.dev/reference/scopes/users:read) | View people in a workspace. |
| [`users:read.email`](https://docs.slack.dev/reference/scopes/users:read.email) | View email addresses of people in a workspace. |
If the app is submitted to the Slack Marketplace, Slack reviews the requested
scopes before approval. After approval, Slackbot can only use the scopes Slack
has approved for that app.
## How This Differs From Slack Alert Targets
Slackbot is for asking questions and receiving answers in Slack.
Slack alert targets are different. They send alert notifications from Parseable
to a Slack channel when an alert fires. If you only want alert messages in Slack,
use the [Slack alerting integration](/integrations/alerting/slack). If you want
to ask questions from Slack, use Slackbot.
Both can be useful together. An alert can arrive in Slack, and the same thread
can continue with an `@parseable` question to inspect the logs, metrics, traces,
or alerts behind the incident.
## Related Pages
* [MCP Server](/mcp)
* [Alerts](/user-guide/alerting)
* [Slack alerting integration](/integrations/alerting/slack)
* [API keys](/user-guide/api-keys)
# Alertmanager (/alerting/alert-manager)
### Overview
[Alertmanager](https://github.com/prometheus/alertmanager) handles alerts sent by client applications such as the Parseable server. Alertmanager takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie.
Parseable alerts can be sent to Alertmanager to allow advanced alert management and notification routing.
### Prerequisites
* Parseable server installed and running. See [installation guide](/self-hosted/installation) for more details.
* Alertmanager installed and running. See [installation and setup guide](https://github.com/prometheus/alertmanager) for more details.
### Configuration
This is a snippet of Parseable alert configuration, to demonstrate the fields required to send alerts to Alertmanager.
```json
{
"name":"AlertManagerTarget",
"type": "alertManager",
"endpoint": "https://some.webhook.com",
"username": "username",
"password":"password",
"skipTlsCheck": true,
"notificationConfig": {
"interval": 3,
"times": 7
}
}
```
The `endpoint` field is the URL of the Alertmanager API endpoint. The `username` and `password` fields are the credentials for basic authentication. The `skipTlsCheck` field is a boolean to skip TLS certificate verification. The `notificationConfig` field is used to repeat the alert notification. If it is not set, Parseable will send the notification just once.
Once Parseable alerts are configured to be sent to Alertmanager, you can use Alertmanager to manage the alerts. For example, you can use Alertmanager to route alerts to different receivers, such as email, PagerDuty, or OpsGenie. You can also use Alertmanager to silence alerts.
# API Reference (/api)
import { Card, Cards } from 'fumadocs-ui/components/card';
This section is the quickest way to understand how data reaches Parseable and which HTTP endpoint fits your integration.
Most teams do not need every route listed here. In practice, you usually start with one ingestion path, send data into a dataset, and then use the query and dataset APIs only when you need direct control from your own tooling.
## Getting Started
Before you begin, make sure you have:
* A running instance of Parseable
* The base URL of your Parseable server
* Required authentication credentials (if authentication is enabled)
Most integrations fall into one of these paths:
* Use `/api/v1/ingest` when your application already has JSON events and you want to route them with headers.
* Use `/v1/logs`, `/v1/traces`, or `/v1/metrics` when your collector or SDK is already sending native OTLP data.
* Use `/v1/prometheus/write` when Prometheus or another compatible system is forwarding metrics through Remote Write.
* Use `/api/v1/query` and `/api/v1/logstream/{stream_name}` when you need SQL queries or direct dataset management from your own code.
If you are deciding where to start, the simplest rule is this:
* JSON events from an app or script: `/api/v1/ingest`
* OpenTelemetry logs: `/v1/logs`
* OpenTelemetry traces: `/v1/traces`
* OpenTelemetry metrics: `/v1/metrics`
* Prometheus Remote Write metrics: `/v1/prometheus/write`
## Ingestion APIs
Send JSON events and route them with the `X-P-Stream` header.
Send OpenTelemetry logs to Parseable using the native `/v1/logs` endpoint.
Send OpenTelemetry traces to Parseable using the native `/v1/traces` endpoint.
Send OpenTelemetry metrics to Parseable using the native `/v1/metrics` endpoint.
Send snappy-compressed Prometheus Remote Write payloads to `/v1/prometheus/write`.
## Query and Dataset APIs
Run SQL queries across datasets. The dataset is selected inside the `FROM` clause.
Send JSON events directly to an existing dataset by path.
Create a dataset before writing events directly to `/api/v1/logstream/{stream_name}`.
Delete a dataset and its associated data.
## Authentication
These endpoints require authentication. This reference uses API keys so new integrations can start with the recommended machine-to-machine flow.
```bash
X-API-Key:
```
You can find the setup flow in the [API Keys](/user-guide/api-keys) guide.
For ingestion endpoints, the other header to pay attention to is usually `X-P-Stream`. That header tells Parseable which dataset should receive the payload. For OTLP routes, Parseable identifies whether the payload is logs, traces, or metrics from the endpoint path.
## Need Help?
If you have any questions or run into issues, please refer to our [GitHub repository](https://github.com/parseablehq/parseable) or [join our community](https://discord.gg/parseable).
# Agentic Observability (/cookbook/agentic-observability)
## Agentic Observability
Learn how to monitor AI agents and LLM-powered applications with Parseable.
### Overview
As AI agents become more prevalent, observability becomes critical for understanding their behavior, debugging issues, and ensuring reliability.
### Key Metrics to Track
* **Token Usage** - Monitor input/output tokens per request
* **Latency** - Track response times for LLM calls
* **Error Rates** - Monitor failed requests and retries
* **Cost** - Track API costs across different models
### Best Practices
1. Log all LLM interactions with full context
2. Track tool calls and their outcomes
3. Monitor agent decision paths
4. Set up alerts for anomalous behavior
# Instrumentation (/cookbook/instrumentation)
## Instrumentation Guide
Learn how to properly instrument your applications for effective observability with Parseable.
### Structured Logging
Always use structured logging formats like JSON:
```json
{
"timestamp": "2024-01-15T10:30:00Z",
"level": "info",
"message": "User logged in",
"user_id": "user-123",
"ip_address": "192.168.1.1",
"duration_ms": 45
}
```
### Key Fields to Include
* **timestamp** - When the event occurred
* **level** - Log level (debug, info, warn, error)
* **message** - Human-readable description
* **correlation\_id** - For tracing requests across services
* **service** - Name of the service generating the log
### Best Practices
1. Use consistent field names across services
2. Include context with every log entry
3. Avoid logging sensitive data
4. Use appropriate log levels
5. Include timing information for performance analysis
### Integration Examples
See the [Logging Agents](/ingest-data/logging-agents) and [OpenTelemetry](/ingest-data/otel) sections for integration guides.
# Tool Calls (/cookbook/tool-calls)
## Tool Calls Observability
Track and analyze tool calls made by AI agents to understand their behavior and debug issues.
### What are Tool Calls?
Tool calls are function invocations made by AI agents to interact with external systems, APIs, or perform specific actions.
### Logging Tool Calls
```json
{
"timestamp": "2024-01-15T10:30:00Z",
"agent_id": "agent-123",
"tool_name": "search_database",
"input": {"query": "user data"},
"output": {"results": 42},
"duration_ms": 150,
"status": "success"
}
```
### Key Metrics
* **Call Frequency** - How often each tool is called
* **Success Rate** - Percentage of successful tool calls
* **Latency** - Time taken for tool execution
* **Error Types** - Common failure modes
### Best Practices
1. Log both input and output of tool calls
2. Include timing information
3. Track the agent context for each call
4. Set up alerts for high failure rates
# Compare editions (/editions/compare)
Parseable is available in three editions: the free, [self-hosted OSS](https://github.com/parseablehq/parseable) core, the [fully managed Cloud](https://telemetry.new/) offering, and Enterprise for mission-critical deployments. This page compares what's included in each.
| | OSS | Cloud | Enterprise |
| -------------- | ------------ | ----------------------------------------- | ----------------------------------- |
| Deployment | Self-hosted | Fully managed on Parseable infrastructure | Fully managed, BYOC, or On-Premises |
| Infrastructure | Self-managed | Shared infrastructure | Dedicated infrastructure |
| License | AGPLv3 | Commercial | Commercial |
## Platform features
| Feature | OSS | Cloud | Enterprise |
| --------------------------------------------------------------------- | :-----------------------: | :-------------------: | :--------: |
| [Threshold alerts with SQL](/user-guide/alerting/standard-threshold) | ✅ | ✅ | ✅ |
| [Dashboards](/user-guide/dashboards) | ✅ | ✅ | ✅ |
| [SQL query](/key-concepts/query) | ✅ | ✅ | ✅ |
| [OpenID Connect](/user-guide/openid) (SSO) | ✅ | ✅ | ✅ |
| Unlimited users | ✅ | ✅ | ✅ |
| Unlimited hosts | ⚠️ (Maximum 1 query node) | ✅ | ✅ |
| [Role based access control](/user-guide/rbac) | ⚠️ (Groups not supported) | ✅ | ✅ |
| Customizable [retention](/user-guide/retention) | ✅ | ✅ (365 days included) | ✅ |
| Customizable object [storage](/key-concepts/storage) | ✅ | ✅ (Parseable managed) | ✅ |
| [PromQL query](/key-concepts/query) | ❌ | ✅ | ✅ |
| [High availability cluster](/key-concepts/high-availability) | ❌ | ✅ | ✅ |
| [Smart cache](/user-guide/smart-cache) | ❌ | ✅ | ✅ |
| [AI agent observability](/user-guide/agent-observability) | ❌ | ✅ | ✅ |
| [Automatic error tracking](/user-guide/errors) | ❌ | ✅ | ✅ |
| APM with service view | ❌ | ✅ | ✅ |
| Natural language Q\&A with [Keystone](/user-guide/ai-native/keystone) | ❌ | ✅ | ✅ |
| [Anomaly detection alerts](/user-guide/alerting/anomaly) | ❌ | ✅ | ✅ |
| [Forecasting alerts](/user-guide/alerting/forecasting) | ❌ | ✅ | ✅ |
| Alerts set with PromQL | ❌ | ✅ | ✅ |
| [AI SQL generation](/user-guide/ai-native/text-to-sql) | ❌ | ✅ | ✅ |
| [Summarization](/user-guide/ai-native/summary) | ❌ | ✅ | ✅ |
| [Custom partitioning](/key-concepts/partitioning) | ❌ | ❌ | ✅ |
| Multi-tenant cluster | ❌ | ❌ | ✅ |
| Audit Logging | ❌ | ❌ | ✅ |
| Live Tailing | ❌ | ❌ | ✅ |
| Bring Your Own \[Cloud, LLM, OAuth] | ❌ | ❌ | ✅ |
| Data residency & sovereignty options | ❌ | ❌ | ✅ |
## Support
| Feature | OSS | Cloud | Enterprise |
| ------------------------------- | :-: | :---: | :--------: |
| Community (Slack, GitHub) | ✅ | ✅ | ✅ |
| Architecture and design support | ❌ | ✅ | ✅ |
| 24x7 product support via portal | ❌ | ✅ | ✅ |
| Dedicated SLA | ❌ | ✅ | ✅ |
| Ingestion agent management | ❌ | ❌ | ✅ |
# Parseable Enterprise (/enterprise)
## Core infrastructure for mission-critical deployments
Core infrastructure projects, especially those that deal with data and need to guarantee data protection and reliability of the highest order, take time to build. Over the last 3 years while building ParseableDB, we've seen a wide range of edge cases, bug reports, weird behavior, and more.
The time is right to create an enterprise-grade offering, built on the solid foundation of AGPLv3 ParseableDB. With a dedicated, well-defined Enterprise offering, we'll be able to serve serious, mission-critical users much better, while improving the sustainability of the OSS product. It helps us derive some value from our work without pulling the rug from under the community.
**Sustainable open source. No drama. No bait-and-switch. Just fast, efficient, observability.**
## Parseable OSS vs. Enterprise Edition
### Parseable OSS
* High performance
* Single binary with built-in Prism
* RBAC and OIDC integrations
* Regular updates and feature releases based on community feedback
* Perfect for individuals, smaller teams, and non-mission-critical deployments
### Enterprise Edition
**Everything in OSS, plus:**
* Hardened, stable builds
* Searching on demand
* Distributed query support
* Built-in schema detection for log formats
* AI Native summarization, alerts, and correlation (coming soon, talk to us for a preview)
* High touch onboarding, O11Y data modeling, and 24/7 support
* Predictable pricing model
## How to access Parseable Enterprise
Drop us a note at [sales@parseable.com](mailto:sales@parseable.com) to set up a quick discovery call and discuss your needs.
Want to try before you buy? You can check out the enterprise features at [demo.parseable.com](https://demo.parseable.com).
Whether you're an indie developer building the next big thing or an enterprise scaling observability across a global infrastructure, we've got you covered. Let's build something awesome together!
# Parseable Auto Instrumentation (/ingest-data/auto-instrumentation)
import { Step, Steps } from 'fumadocs-ui/components/steps';
Parseable Auto Instrumentation, or PAI, is a Kubernetes operator that automatically collects logs, metrics, traces, and Kubernetes events from a cluster and exports them to Parseable, with zero manual OpenTelemetry configuration required.
It is useful when you want cluster-wide observability, but do not want every team to maintain its own collector configuration. You define the Parseable OTLP endpoint, target datasets, namespace selectors, and trace instrumentation settings in one `ParseableConfig` custom resource. PAI then reconciles the OpenTelemetry Collector and `Instrumentation` resources for you.
PAI depends on the OpenTelemetry Operator. The OpenTelemetry Operator manages `OpenTelemetryCollector` and `Instrumentation` custom resources. PAI sits one level above it and generates those resources with Parseable-specific OTLP exporters, datasets, headers, authentication, and cleanup behavior.
## How PAI works
```text
ParseableConfig
|
v
PAI operator
|
| creates and reconciles
v
OpenTelemetry Operator resources
|
+-- pai-log-collector DaemonSet collector
+-- pai-metrics-events-collector Deployment collector
+-- pai-traces Deployment collector
+-- pai-instrumentation-collector-v1 Instrumentation CR
+-- pai-agent DaemonSet for language detection
|
v
Parseable datasets
```
The important thing to understand is that PAI does not bypass OpenTelemetry. It uses OpenTelemetry as the collection layer. Logs, metrics, traces, and events are collected or received by OpenTelemetry Collectors, then exported to Parseable over OTLP HTTP.
For traces, PAI creates an in-cluster trace collector called `pai-traces`. Auto-instrumented application pods send traces to the stable `pai-traces-collector` service. That collector owns the Parseable ingestion endpoint, API key, target dataset, OTLP encoding, and exporter headers, so application pods do not need Parseable credentials.
## What PAI creates
Once you apply a `ParseableConfig`, PAI creates the resources needed for the enabled signals.
| Resource | Type | Purpose |
| ---------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------- |
| `pai-log-collector` | OpenTelemetry Collector DaemonSet | Reads pod logs from every node and collects kubelet resource metrics when enabled |
| `pai-metrics-events-collector` | OpenTelemetry Collector Deployment | Collects cluster metrics, Prometheus scrape metrics, and Kubernetes events |
| `pai-traces` | OpenTelemetry Collector Deployment | Receives traces from instrumented workloads and exports them to Parseable |
| `pai-instrumentation-collector-v1` | OpenTelemetry `Instrumentation` CR | Tells the OpenTelemetry Operator how to inject SDKs into application pods |
| `pai-agent` | DaemonSet | Detects application runtimes, including distroless containers |
| `-pai-collector` | ClusterRole | Gives collector service accounts the permissions required to read Kubernetes telemetry |
You do not need to manage these resources directly. PAI owns their lifecycle. If the `ParseableConfig` changes, PAI reconciles them. If the `ParseableConfig` is deleted, PAI cleans them up.
## Prerequisites
Before installing PAI, make sure you have:
* A Kubernetes cluster running v1.26 or later
* Helm v3
* A Parseable OTLP HTTP ingestion endpoint
* A Parseable API key
* OpenTelemetry Operator v0.99.0 or later installed in the cluster
PAI uses the `v1beta1` `OpenTelemetryCollector` API, so older OpenTelemetry Operator versions will not work. The `Instrumentation` resource is still created with the OpenTelemetry Operator's `v1alpha1` API.
## Install PAI
### Install the OpenTelemetry Operator
If the OpenTelemetry Operator is not already installed, install it first:
```bash
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--namespace otel-operator --create-namespace \
--set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" \
--set admissionWebhooks.certManager.enabled=false \
--set admissionWebhooks.autoGenerateCert.enabled=true
```
### Install the PAI operator
Install the operator into its own namespace:
```bash
helm repo add parseable https://charts.parseable.com
helm repo update
helm install pai parseable/pai -n pai-system --create-namespace
```
Check that the operator pod is running:
```bash
kubectl get pods -n pai-system
```
### Create a Parseable credentials secret
For new setups, use an API key. PAI reads the `apiKey` value from the secret and sends it to Parseable as the `x-api-key` header.
```bash
kubectl create secret generic parseable-creds \
--from-literal=apiKey= \
-n pai-system
```
PAI still supports basic authentication for existing deployments. In that case, set `authType: basic` or omit `authType`, and create the secret with `username` and `password`.
### Apply a ParseableConfig
Create a `ParseableConfig` resource. This is the main PAI configuration object. The example below enables pod logs, cluster metrics, Kubernetes events, and trace auto-instrumentation for selected namespaces.
```yaml
apiVersion: observability.parseable.com/v1alpha1
kind: ParseableConfig
metadata:
name: production
namespace: pai-system
spec:
target:
endpoint: https://
authType: apiKey
encoding: json
credentialsSecret:
name: parseable-creds
namespace: pai-system
headers:
X-P-Environment: production
logs:
podLogs:
enabled: true
targetDataset: pai-logs
namespaceSelector:
mode: include
namespaces:
- app-namespace
metrics:
clusterMetrics:
targetDataset: pai-cluster-metrics
namespaceSelector:
mode: include
namespaces:
- app-namespace
k8sCluster:
enabled: true
kubelet:
enabled: true
events:
enabled: true
targetDataset: pai-events
namespaceSelector:
mode: include
namespaces:
- app-namespace
traces:
targetDataset: pai-traces
namespaceSelector:
mode: include
namespaces:
- app-namespace
instrumentation:
languages:
- java
- python
- nodejs
- dotnet
detectionTimeout: "1m"
```
Apply it:
```bash
kubectl apply -f parseableconfig.yaml
```
## What each signal means
### Logs
PAI collects Kubernetes pod logs through the OpenTelemetry `filelog` receiver. The log collector runs as a DaemonSet, so every node can read the pod log files available on that node. Records are enriched with Kubernetes metadata such as namespace, pod, container, and node.
Use `spec.logs.podLogs.namespaceSelector` when you only want logs from specific namespaces, or when you want to exclude noisy namespaces. You can also use `spec.logs.files` to tail host log directories, such as audit logs or custom application logs written to a mounted path.
### Metrics
PAI supports built-in cluster metrics and Prometheus-style scrape configs. Built-in metrics come from receivers such as `k8s_cluster`, kubelet scraping, and kube-state-metrics when enabled. These give you Kubernetes object state, pod and container resource usage, node conditions, and similar operational signals.
For application metrics, use `spec.metrics.scrapeConfigs`. PAI creates Prometheus scrape pipelines inside the OpenTelemetry Collector and exports those metrics into the dataset you choose.
### Traces
For traces, PAI uses OpenTelemetry auto-instrumentation. You list the languages you want PAI to instrument under `spec.traces.instrumentation.languages`. PAI then creates an OpenTelemetry `Instrumentation` resource and annotates matching workloads for SDK injection.
The supported instrumentation images in PAI currently cover Java, Python, Node.js, and .NET. PAI also runs an agent that detects workload languages, including workloads that use distroless images where normal image-name checks are not enough.
Instrumented applications send traces to the in-cluster `pai-traces-collector` service. The trace collector then exports the data to the Parseable dataset configured in `spec.traces.targetDataset`.
### Events
PAI collects Kubernetes events through the `k8sobjects` receiver in watch mode. Events are exported as the logs signal and stored in the dataset configured under `spec.events.targetDataset`.
This helps correlate scheduling failures, restarts, image pull errors, and other cluster activity with the rest of your telemetry.
## Control collection scope
### Namespace selector
Most Kubernetes-aware signal configs support `namespaceSelector`.
```yaml
namespaceSelector:
mode: include
namespaces:
- app-namespace
- payments
```
Use `include` to collect only from the listed namespaces. Use `exclude` to collect from every namespace except the listed namespaces. If you omit the selector, PAI collects from all namespaces for that signal.
For cluster metrics, namespace filtering applies to pod-scoped metrics. Node-scoped metrics are not filtered by namespace because nodes do not belong to a namespace.
### Workload selector for traces
Use `spec.traces.workloadSelector` when you want trace auto-instrumentation only for workloads with specific labels.
```yaml
spec:
traces:
targetDataset: pai-traces
workloadSelector:
mode: include
matchLabels:
observability: enabled
```
This is helpful when you want to roll out tracing gradually instead of instrumenting every workload in a namespace at once.
### Custom headers
You can set custom headers globally under `spec.target.headers`, or per signal under the relevant signal config.
```yaml
spec:
target:
headers:
X-P-Environment: production
traces:
targetDataset: pai-traces
headers:
X-P-Environment: production-traces
```
Signal-level headers override global headers with the same key. PAI-generated headers such as authentication, `X-P-Stream`, and `X-P-Tenant` take precedence over custom headers.
### Encoding
`spec.target.encoding` controls the OTLP HTTP wire format. Use `json` unless you know your Parseable deployment supports protobuf ingestion for the relevant signal.
```yaml
spec:
target:
encoding: json
```
## Add host logs or application metrics
### Host file logs
Use `spec.logs.files` to tail log files from a host path on every node.
```yaml
spec:
logs:
files:
- name: audit-logs
hostPath: /var/log/app/audit
targetDataset: pai-audit-logs
```
PAI creates a separate filelog pipeline for each entry.
### Prometheus scrape configs
Use `spec.metrics.scrapeConfigs` when an application exposes Prometheus metrics.
```yaml
spec:
metrics:
scrapeConfigs:
- name: application-metrics
uri: /metrics
port: 8080
targetDataset: pai-application-metrics
namespaceSelector:
mode: include
namespaces:
- app-namespace
podSelector:
app.kubernetes.io/name: my-app
```
PAI discovers matching pods, scrapes the configured path and port, then exports the metrics to the selected Parseable dataset.
## Verify the setup
Check that the PAI operator and generated collectors are running:
```bash
kubectl get pods -n pai-system
kubectl get opentelemetrycollector -n pai-system
kubectl get instrumentation -n pai-system
```
Check the `ParseableConfig` status:
```bash
kubectl get parseableconfig production -n pai-system -o yaml
```
Look at `status.conditions` for reconciliation errors. For traces, `status.workloads` shows which workloads were processed, which language was detected, and whether instrumentation was applied.
Once telemetry is flowing, open Parseable and look for the datasets you configured, such as `pai-logs`, `pai-cluster-metrics`, `pai-events`, and `pai-traces`.
## Troubleshooting
* **No collectors are created:** Check the `ParseableConfig` status and confirm the OpenTelemetry Operator is installed. PAI requires the `v1beta1` `OpenTelemetryCollector` CRD.
* **Authentication errors:** Confirm `spec.target.authType` matches the secret keys. `apiKey` requires an `apiKey` key. Basic auth requires `username` and `password`.
* **Logs are missing from a namespace:** Check `spec.logs.podLogs.namespaceSelector`. In `include` mode, only listed namespaces are collected.
* **Metrics are missing:** Confirm the relevant metric source is enabled. For built-in metrics, check `spec.metrics.clusterMetrics`. For app metrics, check `spec.metrics.scrapeConfigs`, the pod labels, port, and metrics path.
* **Traces are missing:** Check that the workload namespace and labels match your trace selectors, then inspect `status.workloads` to confirm language detection and instrumentation.
Use collector logs when you need more detail:
```bash
# Metrics and events
kubectl logs -n pai-system deployment/pai-metrics-events-collector-collector --all-containers --tail=200
# Traces
kubectl logs -n pai-system deployment/pai-traces-collector --all-containers --tail=200
```
The log collector runs as a DaemonSet, so select the `pai-log-collector` pod on the affected node:
```bash
kubectl get pods -n pai-system
kubectl logs -n pai-system --all-containers --tail=200
```
## Pause collection
Set `spec.paused: true` to stop collection without deleting the `ParseableConfig`.
```yaml
spec:
paused: true
```
PAI removes collectors, instrumentation, and the agent while the config is paused. Set it back to `false` to resume collection.
## Uninstall
Delete the `ParseableConfig` first so PAI can clean up generated resources.
```bash
kubectl delete parseableconfig production -n pai-system
helm uninstall pai -n pai-system
kubectl delete namespace pai-system
```
For a full cluster cleanup, including OpenTelemetry resources created during testing, use the cleanup script from the PAI repository.
```bash
curl -sL https://raw.githubusercontent.com/parseablehq/pai/refs/heads/main/scripts/cleanup.sh | bash
```
# Prometheus Remote Write (/ingest-data/prometheus)
Use this endpoint when Prometheus sends metrics directly to Parseable through the Prometheus Remote Write protocol. Parseable accepts the snappy-compressed protobuf payload sent by Prometheus, converts the samples to OpenTelemetry metrics, and writes them to the dataset named in `X-P-Stream`.
**Endpoint:** `POST /v1/prometheus/write`
| Header | Required | Description |
| ------------------ | -------- | ------------------------------------------------------------------------------- |
| `Authorization` | Yes | Bearer token for a Parseable API key with access to write to the target dataset |
| `Content-Encoding` | Yes | Must be `snappy` |
| `Content-Type` | Yes | `application/x-protobuf` |
| `X-P-Stream` | Yes | Target stream name. |
**Prometheus configuration:**
```yaml
remote_write:
- url: "http://:8000/v1/prometheus/write"
authorization:
type: Bearer
credentials:
headers:
X-P-Stream: prometheus-metrics
```
`X-P-Stream` is the metrics dataset that should receive the samples. Prometheus sets the remote write content headers for the protobuf payload, so the main values you usually configure are the URL, Bearer token, and `X-P-Stream`.
# eBPF (/ingest-data/zero-instrumentation)
[Tetragon](https://github.com/cilium/tetragon) is an open-source project from Cilium that provides runtime security, deep observability, and kernel-level transparency using eBPF. Tetragon monitors processes, syscalls, file and network activity in the kernel, correlating threats with network data to identify responsible binaries. It shares insights via JSON logs and a gRPC endpoint.
This Document will walk you through how to set up Tetragon Connector to extract eBPF logs on Parseable using Vector. It is used to build observability pipelines that collect, transform, and route logs, metrics, and traces.
## Prerequisites
* A Kubernetes with admin accesss.
* Kubectl installed on your machine.
## Deploy Tetragon
You can deploy Tetragon using Helm on your Kubernetes cluster using the commands:
```bash
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
```
## Deploy Vector
You can deploy Vector using Helm on your Kubernetes cluster. We have created a vector-tetragon-values.yaml file to configure Vector to collect logs from Tetragon and send them to Parseable.
```bash
helm repo add vector https://helm.vector.dev
wget https://www.parseable.com/blog/vector/vector-tetragon-values.yaml
helm install vector vector/vector --namespace vector --create-namespace --values vector-tetragon-values.yaml
```
It will take some time to deploy Vector. You can check the status using the command:
```bash
kubectl get pods -n vector
```
Now Vector is ready to send the events stored in `/var/run/cilium/tetragon/tetragon.log` file to the Parseable tetrademo dataset. Once this is done, you can verify the log events inPrism.
We have a blog post which guides you through the process of tracking sensitive file access using Tetragon and Parseable. You can read it here.
# Data modelling (/key-concepts/data-model)
import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';
Datasets are the primary unit of storage in Parseable. Any MELT (Metrics, Events, Logs, Traces) data is ingested into one or other dataset.
Think of a dataset as a logical grouping of similar telemetry data. Proper datasets planning ensures fast query and optimal storage compression.
Every dataset is identified by a unique name. A dataset has a assigned schema (can be dynamic i.e. inferred from incoming data or static i.e. explicitly defined). Role based access control, alerts, retention and notifications are supported at the dataset level.
## Mapping sources to datasets
As SREs and DevOps engineers, you often deal with multiple data sources generating telemetry data. These sources can include applications, infrastructure components, and third-party services. Each of these sources can produce different types of data with varying schemas.
When debugging or investigating issues, it's common to correlate data from multiple sources. For example, you might want to correlate application logs with infrastructure metrics to identify performance bottlenecks.
Hence it is crucial to thoughtfully map data sources to datasets in Parseable. This mapping ensures that related data is stored together, making it easier to query and analyze.
A data source is anything that generates data, i.e. agents like FluentBit, FluentD, Vector, LogStash, agents from the OTel ecosystem, or the application itself. While ingesting data, you'll need to specify the dataset name to which the data should be sent. This allows mapping sources to datasets. Technically it is possible to map any number of data sources any number of data sources. Parseable allows this to ensure flexibility for varying use cases.
But, it is important to critically think about mapping data sources to datasets. Too many unrelated columns in a dataset can lead to poor compression and slower query performance. On the other hand, too many datasets can lead to increased complexity in managing the data.
When deciding how to map sources to datasets, consider the following:
* **Schema similarity**: If the sources have similar schema, it is better to map them to a single dataset. This allows for better compression and faster query performance. Similar here means fields are matching for 80 percent or more of the events. If the schema is too different, it is better to create separate datasets for each source.
* **Query patterns**: If you frequently query across multiple sources, it is better to map them to a single dataset. This allows you to query the data easily without having to join multiple datasets.
* **Data retention**: If the sources have different data retention requirements, it is better to create separate datasets for each source. This allows you to set different retention policies for each dataset.
* **Data ownership**: If different teams own different sources, it is better to create separate datasets for each source. This allows you to set different access controls for each dataset and manage the data better.
Let's understand this with some examples:
* **Kubernetes infrastructure logs**: Kubernetes infrastructure logs (e.g. kubelet, kube-proxy, etc.) can be mapped to a single dataset. This allows you to query the logs across all the Kubernetes components easily. Since these logs have similar schema, they fit well into a single dataset.
* **Application logs with similar schema**: If you have logs from multiple applications that log in a common format for example [go-log](https://pkg.go.dev/log), you can create a single dataset for all of them. This allows you to query the logs across applications easily.
* **Application logs with different schemas**: If you have logs from multiple applications that have completely different schemas, you can create a separate dataset for each application. This allows you to enforce a specific schema for each dataset and query them independently.
* **Aggregated data**: If you have aggregated data (e.g. metrics, traces) that you want to store, you can create a separate dataset for that. This allows you to query the aggregated data separately from the raw logs.
Beyond 800 columns in a dataset, consider splitting the dataset into multiple datasets based on schema similarity or query patterns. Beyond 1000 columns, the server will reject the ingestion request with an error.
## Adding data to Parseable
### Identify the data source
Refer the ingestion guides for various data sources in the [Ingestion](/ingestion) section.
### Create or use an existing dataset
You can create a dataset using the "Create Dataset" button on Datasets page. You'll be prompted to enter the dataset name, schema type, and partition column.
You can set the schema type to be [static](#static) (schema has to be explicitly provided at the time of creation of dataset) or [dynamic](#dynamic) (let server infer the schema from the incoming data). Once set, the schema type cannot be changed. Read more about schema types in the [Dataset Schema](#dataset-schema) section.
Partition column is an optional field. If you want to partition the dataset based on a specific field, you can specify that field here. If you don't specify a partition field, Parseable will use the internal `p_timestamp` field as the partition field. Read more about partitioning in the [Partitioning](/key-concepts/partitioning) section.
## Dataset vs index
Traditional indices in systems like Elasticsearch are build to ingest textual data, index each field and allow for fast search and retrieval. This works well for pure search use cases, where you want to search for specific keywords or phrases in the data.
But applying this concept to huge volumes of observability data (logs, metrics, traces) is not practical. Observability data is often structured, semi-structured or unstructured, and indexing every field can lead to excessive storage costs for little to no performance gain.
Parseable datasets are designed to handle large volumes of observability data efficiently. They focus on optimal storage compression and fast retrieval via query, rather than indexing every field. This allows Parseable to handle high cardinality data, such as logs with many unique fields, without the performance and storage overhead of traditional indices.
## Dataset schema
Schema defines the fields in an event and their types. Parseable supports two types of schema - dynamic and static. You can choose the schema type while creating the dataset. Additionally, if you want to enforce a specific schema, you'll need to send that schema at the time of creating the dataset.
### Dynamic
Datasets by default have dynamic schema. This means you don't need to define a schema for a dataset. The Parseable server detects the schema from first event. If there are subsequent events (with new schema), it updates internal schema accordingly.
Log data formats evolve over time, and users prefer a dynamic schema approach, where they don't have to worry about schema changes, and they are still able to ingest events to a given dataset.
For dynamic schema, Parseable doesn't allow changing the type of an existing column whose type is already set. For example, if a column is detected as string in the first event, it can't be changed to int or timestamp in a later event. If you'd like to enforce a specific schema, please use static schema.
### Static
In some cases, you may want to enforce a specific schema for a dataset. You can do this by setting the static schema type while creating the dataset. This schema will be enforced for all the events ingested to the dataset. You'll need to provide the schema in the form of a JSON object with field names and their types, with the create dataset API call. The following types are supported in the schema: `string`, `int`, `float`, `datetime`,`date`, `boolean`.
## FAQ
Some of common questions related to datasets are answered below. If you have any other questions, please reach out to us on [Slack](http://logg.ing/community) or [GitHub Discussions](http://github.com/parseablehq/parseable/discussions).
A dataset is not equivalent to an index in Elasticsearch. While both are used to store and retrieve data, datasets in Parseable are designed specifically for observability data (logs, metrics, traces) and focus on optimal storage compression and fast retrieval via query. Datasets can handle high cardinality data without the performance and storage overhead of traditional indices. Refer to the [Dataset vs index](#dataset-vs-index) section for more details.
General principle is to create a new dataset if the data has a different schema or if you want to enforce a specific schema. If the data has similar schema and you want to query it together, you can use an existing dataset. Refer to the [Mapping sources to datasets](#mapping-sources-to-datasets) section for more details.
If in doubt, use static schema. Static schema allows you to enforce a specific schema for the dataset, which can help in better compression and faster query performance. Dynamic schema is useful when you don't want to worry about schema changes and want to ingest events without specifying the schema. Refer to the [Dataset Schema](#dataset-schema) section for more details.
Partitioning is useful when you want to optimize query performance for specific fields. If you frequently query on a specific field, you can partition the dataset based on that field. This allows Parseable to store the data in a way that makes it faster to retrieve the data for that field. Refer to the [Partitioning](/key-concepts/partitioning) section for more details.
We recommend keeping the number of columns in a dataset to a reasonable limit, ideally less than 800. Too many columns can lead to poor compression and slower query performance. If you have more than 800 columns, consider splitting the dataset into multiple datasets based on schema similarity or query patterns. Beyond 1000 columns, the server will reject the ingestion request with an error.
# High Availability (/key-concepts/high-availability)
Parseable supports a distributed, high-availability mode for production use cases where downtime is not an option. The distributed setup is designed to ensure fault tolerance and high availability for log ingestion.
The distributed setup consists of multiple ingestion and query server and a S3 (or other object store) bucket.
The Query servers use metadata stored in the object store to query the data. The query server uses the Parseable manifest file and the Parquet footers in tandem to ensure that the data is read in fewest possible object store API calls.
## Node Functions
* **Prism Node**: Handles the Parseable UI and all user requests except for query and search operations. The Prism node is designed to be as compute efficient as the querier node.
* **Querier Node**: Processes data queries and analytics. Queriers use metadata stored in the object store to efficiently read data with minimal API calls.
* **Ingestor Node**: Processes incoming log events. Each ingestor creates its own set of metadata and data files in the object storage system, allowing for simple scaling as workloads change.
* **Indexer Node**: Manages indexing and search functionality.
## High Availability
Parseable Enterprise builds upon the distributed architecture of Parseable OSS, enhancing it with an even more robust high availability framework. This feature allows you to independently scale the query nodes along with the independently scalable ingest nodes.
The high availability architecture in Parseable Enterprise consists of four specialized node types, each serving a distinct function within the cluster:
### Node specific Environment variables
| Node Type | Role | Scalability | Node specific Env var |
| -------------- | ------------------------------------------- | -------------------------------- | --------------------- |
| Prism (Leader) | Manages UI, dataset configuration, and RBAC | Single Node | - |
| Query | Handles data querying and analytics | Independently scalable | P\_QUERIER\_ENDPOINT |
| Ingest | Processes incoming log event | Independently scalable | P\_INGESTOR\_ENDPOINT |
| Index | Manages indexing and search | Single Node (multi node planned) | P\_INDEXER\_ENDPOINT |
Details of the environment variables are available in the [Environment Variables](/self-hosted/configuration).
Each node in the cluster generates and maintains its own NodeMetadata file containing domain name information, authentication tokens, and node-specific configuration. These metadata files are stored in the configured S3 bucket and serve as the foundation for inter-node communication.
For optimal performance, we recommend the following specifications for each node type:
| Node Type | vCPU | Memory |
| -------------- | ---- | ------ |
| Prism (leader) | 16 | 32 GiB |
| Query | 16 | 32 GiB |
| Ingest | 8 | 16 GiB |
| Index | 16 | 32 GiB |
The Prism node requires similar compute and storage resources as the querier node because it handles all user interface operations and administrative requests.
# Key Concepts (/key-concepts)
import { IconChartCohort,IconDirections,IconCloudDataConnection,IconBrandInertia,IconFileTypeSql,IconServerBolt
} from '@tabler/icons-react';
Dive deeper into the key concepts of Parseable to understand how to best leverage Parseable for your specific observability needs. Whether you're new to Parseable or looking to deepen your understanding, these concepts will provide a solid foundation for using Parseable effectively.
} title="Data modelling">
Learn when to create datasets, how to aggregate observability data, use schemas, and manage your data effectively.
} title="Partitioning">
Optimize query performance by partitioning datasets based on frequently filtered columns.
} title="Query">
Use PostgreSQL-compatible SQL to query your telemetry data with powerful filtering and aggregation.
} title="Storage">
Achieve storage efficiency and cost with object stores like S3, GCS, Azure Blob as the primary storage layer.
} title="High availability" className="col-span-2">
Ensure your Parseable deployment is resilient and always available with built-in high availability features.
Deploy across public or private clouds, containers, VMs, or bare metal environments with complete data security and privacy.
# Partitioning (/key-concepts/partitioning)
Partitioning allows splitting of telemetry based on specific columns and value pairs to improve query performance. The decision to choose specific columns for partitioning is based on the access patterns of the data. By partitioning log data, you can optimize query performance, reduce the amount of data scanned during queries, and improve storage efficiency.
## When to use partitioning?
Partitioning is useful when you have a clear understanding of the most common data access patterns for a given log dataset. More specifically, when the columns where users are most likely to `filter` or `group by` are well known.
Also, a relatively larger dataset (at least few TBs or more) is better suited for partitioning. For tiny datasets, the overhead of managing partitions might outweigh the benefits.
## Select a column for partitioning
The first step is to find which column users are most likely to `filter` or `group by`. Once you have this information, it is important to know the variance in the column values (i.e. the number of unique values in the column). For example, if you have a column `log_level` with only a few unique values like `ERR`, `WARN`, and `INFO`, it would be a good candidate for partitioning. But if there is a column called `log_message` where each log event has a unique message, partitioning on this column will in fact make things worse.
**Important**: In Parseable, you can select only one column per dataset for partitioning.
## How to set up partitioning?
You can specify a single column for partitioning while creating a dataset on the Parseable Console (Create Dataset >> Custom Partition Field). Once the dataset is created, Parseable will automatically create physical partitions based on the values in this column.
You can also edit the partition column for an existing dataset (Dataset >> Manage >> Info >> Custom Partition Field). Note that this will have effect on all the new data that is ingested into the dataset, and not the existing data.
## How does partitioning work?
When a dataset is created with partitioning enabled, Parseable will create physical partitions based on the values in the specified column. For example, if you have a column `log_level` with the values `ERR`, `WARN`, and `INFO`, Parseable will create three physical partitions, one for each value.
When a query is run with a filter on `log_level`, Parseable will only scan the relevant partition(s) and not the entire dataset. This significantly reduces the amount of data scanned during queries and improves query performance.
Let's understand this better with an example. Let's say you have log events with columns `timestamp`, `log_level`, `service_name`, `log_message` and `os`. If one of the most common data query patterns is to filter events by `log_level` (i.e., most queries are of the form `select * from logs where log_level = '...'`), then you should consider partitioning by the `log_level` column.
Physically on the storage (S3 bucket or disk), you'll see the data organized by the partition column values. For example, if you have partitioning on `log_level`, you'll see the data organized like this:
```sql
log_level=ERR
log_level=WARN
log_level=INFO
```
If you frequently filter by both `log_level` and another column like `os`, you'll need to decide which one provides the most query benefit as your partition column, since only one column can be used for partitioning.
## Best practices
* **Choose the right column:** Choose the column that is most frequently used in query filters.
* **Understand the data distribution:** Ensure that the column you choose has a good distribution of values with relatively low cardinality (few unique values).
* **Avoid high cardinality columns:** Partitioning on a column with high cardinality (i.e., many unique values) can lead to too many small partitions, which is inefficient.
* **Consider query patterns:** Select a partition column that aligns with your most common query filters.
* **Monitor and adjust:** Monitor the query performance and adjust the partition column as needed if query patterns change.
# Query (/key-concepts/query)
Parseable offers a SQL compatible query interface to query telemetry data. Users can choose to use the filter interface directly without having to deal with SQL at all. However for more complex queries and advanced users, Parseable offers a SQL query interface.
You can specify the query and the relevant time range for which you want this query to be run. The response is inclusive of both the start and end timestamps.
The filter interface is quite self explanatory, with options to filter by specific columns and values and also by time range. In this document, we'll cover more about the SQL API and its capabilities.
Check out the [Query API](/api/v1/queryLogStream).
## How does it work?
After parsing and creating the execution plan for a query, the Parseable query server uses the data manifest file to filter out the relevant Parquet files. The data manifest file is a JSON file that contains the specific column metadata for a whole day. The querier uses this file to filter the relevant Parquet files based on the query filters and the time range.
Only the relevant Parquet file paths are then added as a data source to custom table provider. Datafusion then efficiently reads the files via the GetRange S3 API, pulling only the very specific data needed for the query. This ensures that only the relevant data is read from the storage, reducing the query time and cost.
## Supported functions
Parseable supports a wide range of SQL functions - Aggregate, Window and Scalar functions. Refer the Apache Datafusion documentation for the complete list of supported functions and their usage.
* [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html)
* [Window Functions](https://datafusion.apache.org/user-guide/sql/window_functions.html)
* [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
## Query with regular expressions
This section provides examples of how to use regular expressions in Parseable queries.
* Match regular expression (Case Sensitive)
```sql
SELECT * FROM frontend where message ~ 'failing' LIMIT 9000;
```
* Match regular expression (Case Insensitive)
```sql
SELECT * FROM frontend where message ~* 'application' LIMIT 9000;
```
* Does not match regular expression (Case Sensitive)
```sql
SELECT * FROM frontend where message !~ 'started' LIMIT 9000;
```
* Does not match regular expression (Case Insensitive)
```sql
SELECT * FROM frontend where message !~* 'application' LIMIT 9000;
```
* Matches the beginning of the string (Case Insensitive)
```sql
SELECT * FROM frontend where message ~* '^a' LIMIT 9000;
```
* Matches the end of the string
```sql
SELECT * FROM frontend where message ~ 'failing$' LIMIT 9000;
```
* Matches numeric type data
```sql
SELECT * FROM frontend where uuid ~ '[0-9]' LIMIT 9000;
```
* Matches numeric type data (two digits)
```sql
SELECT * FROM frontend where uuid ~ '[0-9][0-9]' LIMIT 9000;
```
* Matches numeric type data (two digits)
```sql
SELECT * FROM frontend where uuid ~ '[0-9][0-9]' LIMIT 9000;
```
* Replace every instance of the numeric type data with the symbol \*
```sql
SELECT REGEXP_REPLACE(uuid,'[0-9]','*','g') FROM frontend LIMIT 9000;
```
* When a Regex is run against a string, the REGEXP\_MATCHES() function compares the two and returns the string that matches the pattern as a set.
```sql
SELECT REGEXP_MATCH(email,'@(.*)$') FROM frontend where email is not null LIMIT 10;
```
* Postgres regex numbers only: Use the REGEXP\_REPLACE() function to extract only the numbers from a string in PostgreSQL.
```sql
SELECT REGEXP_REPLACE(email,'\\D','','g') FROM frontend where email is not null LIMIT 10;
```
* Postgres regex split: SPLIT\_PART() function can split a string into many parts. To divide a string into several pieces, we must pass the String, the Delimiter, and the Field Number.
```sql
SELECT SPLIT_PART(email,'@',1) FROM frontend where email is not null LIMIT 10 -- return before @ from email;
SELECT SPLIT_PART(email,'@',2) FROM frontend where email is not null LIMIT 10 -- return after @ from email;
```
* Postgres Regex Remove Special Characters: Using the REGEXP\_REPLACE() function, all Special Characters from a supplied text can be eliminated.
```sql
SELECT REGEXP_REPLACE(email, '[^\\w]+','','g') FROM frontend where email is not null LIMIT 10;
```
* Functions and Operators in pattern matching: Like and other POSIX regular expressions are supported.
```sql
SELECT * FROM frontend where email LIKE '%test%' LIMIT 10;
SELECT * FROM frontend where email ~ '^test' LIMIT 10;
```
## Case sensitivity
Dataset column names are case sensitive. For example, if you send a log event like
```json
{
"foo": "bar",
"Foo": "bar"
}
```
Parseable will create two columns, `foo` and `Foo` in the schema. So, while querying, please refer to the fields as `foo` and `Foo` respectively. While querying, unquoted identifiers are converted to lowercase. To query column names with uppercase letters, they must be passed in double quotes. For example, when sending a query via the REST API, the following JSON payload will apply the `WHERE` condition to the column `Foo`:
```json
{
"query":"select * from dataset where \"Foo\"=bar",
"startTime":"2023-02-14T00:00:00+00:00",
"endTime":"2023-02-15T23:59:00+00:00"
}
```
If you're querying Parseable via Grafana UI (via the [data source plugin](https://github.com/parseablehq/parseable-datasource)), you can use the following query to query the column `Foo`:
```sql
SELECT * FROM dataset WHERE "Foo" = 'bar'
```
## Query analysis
In some cases, you may want to understand the query performance. To view the detailed query execution plan, use the `EXPLAIN ANALYZE` keyword in the query. For example, the following query will return the query execution plan and time taken per step.
```json
{
"query": "EXPLAIN ANALYZE SELECT * FROM frontend LIMIT 100",
"startTime": "2023-03-07T05:28:10.428Z",
"endTime": "2023-03-08T05:28:10.428Z"
}
```
## Response fields information with query results
To get the query result fields as a part of query API response, add the query parameter `fields=true` to the API call, e.g. `http://localhost:8000/api/v1/query?fields=true`.
For example, for a query like `select count(*) as count from app1`, with the query parameter added will respond like this:
```json
{
"fields": [
"count"
],
"records": [
{
"count": 2
}
]
}
```
# Storage (/key-concepts/storage)
Parseable is a purpose built telemetry datalake. This means telemetry data is persisted in inexpensive, infinitely scalable commodity storage such as Amazon S3, Google Cloud Storage, Azure Blob, or other S3‑compatible service (Tigris, MinIO, Wasabi, DigitalOcean Spaces, etc.).
## Architecture
Parseable uses Apache Arrow and Parquet as its underlying data structures, optimized for analytical workloads. This columnar format provides:
* **Compression efficiency**: Significantly reduced storage costs
* **Query performance**: Fast analytical queries over compressed data
* **Schema evolution**: Flexible data structure changes over time
* **Cross-platform compatibility**: Standard format readable by many tools
(SSE)
* **Customer-Managed Keys**: SSE-C for custom encryption keys
* **TLS in Transit**: Secure data transmission
* **Access Control**: Fine-grained permissions through cloud IAM
## Supported providers
Parseable supports multiple cloud storage providers and S3-compatible services:
### Cloud Providers
* **AWS S3**: Native integration with all AWS regions
* **Azure Blob Storage**: Full support for Azure storage accounts
* **Google Cloud Storage**: Compatible through S3 API
### S3-Compatible Services
* **MinIO**: Self-hosted object storage
* **Wasabi**: Cost-optimized cloud storage
* **DigitalOcean Spaces**: Developer-friendly object storage
* **Backblaze B2**: Affordable cloud storage
## Authentication Models
Parseable supports multiple authentication mechanisms to fit different deployment scenarios:
### Static Credentials
* Access keys and secret keys for direct authentication
* Suitable for development and simple deployments
* Requires careful credential management
### Dynamic Credentials
* **IAM Roles**: For AWS EC2/ECS deployments
* **Instance Metadata Service (IMDS)**: Automatic credential rotation
* **Container Credentials**: For containerized environments
* **Azure AD Integration**: Service principal authentication
### Security Features
* **Encryption at Rest**: Support for server-side encryption (SSE)
* **Customer-Managed Keys**: SSE-C for custom encryption keys
* **TLS in Transit**: Secure data transmission
* **Access Control**: Fine-grained permissions through cloud IAM
## Data Organization
Parseable organizes data in object storage using a hierarchical structure:
```
bucket/
├── streams/
│ ├── app-logs/
│ │ ├── year=2024/
│ │ │ ├── month=01/
│ │ │ │ ├── day=15/
│ │ │ │ │ └── data.parquet
│ └── system-logs/
└── metadata/
└── schemas/
```
# Integrations (/integrations)
import { IconAddressBook, IconMail, IconWebhook } from '@tabler/icons-react';
Parseable integrates with tools and platforms across visualization, authentication, alerting, application platforms, data platforms, and cloud providers.
***
## Visualization
}>
Create dashboards and alerts with Parseable as a data source.
}>
Self-service BI and analytics on your log data.
}>
Open-source data exploration and visualization.
}>
Enterprise analytics and visualization.
}>
Connect and query your data sources.
}>
Business intelligence and analytics.
***
## Authentication
}>
OpenID Connect authentication.
}>
Open-source identity provider.
}>
Open-source identity and access management.
}>
Enterprise identity management.
}>
Flexible authentication platform.
}>
Microsoft Azure Active Directory.
}>
Google identity provider.
}>
Lightweight Directory Access Protocol.
***
## Alerting
}>
Send alerts and notifications to Slack channels.
}>
Incident management and on-call scheduling.
}>
Alert and incident management.
}>
Alert notifications to Teams channels.
}>
Send alerts to Discord servers.
}>
Email notifications for alerts.
}>
Generic webhook for custom integrations.
***
## Application platforms
}>
Observe workflow automation with traces and related telemetry.
}>
Send logs, traces, Speed Insights, and Web Analytics to Parseable.
***
## Data platforms
}>
Collect Starburst Galaxy cluster metrics with Prometheus remote write.
***
## Cloud providers
}>
Getting started with AWS integrations.
}>
Ingest logs from CloudWatch.
}>
Stream logs via Kinesis Data Firehose.
}>
Direct integration with Data Firehose.
}>
Ingest Lambda function logs.
}>
Ingest logs from Azure Event Hubs.
}>
Azure API Management logs via Event Hub and Logic Apps.
}>
Ingest from Azure Service Bus.
}>
Stream logs from GCP Pub/Sub.
}>
Ingest Fly.io platform logs and metrics.
# Parseable AI Assistant (/llm/text-to-sql)
# Users and Roles (/pb-cli/admin)
These commands require admin privileges on the Parseable server.
Parseable uses role-based access control. In practice, that means you define roles first, then assign those roles to users.
## Roles
### Create a role
```bash
pb role add
```
This opens an interactive prompt where you choose the privilege level and the dataset scope.
**Privilege levels:**
| Privilege | Access |
| ---------- | ------------------------------------------------ |
| `none` | No server privileges |
| `admin` | Full system-wide access |
| `editor` | Read and write access across datasets |
| `reader` | Read-only access to a specific dataset |
| `writer` | Read and write access to a specific dataset |
| `ingestor` | Write-only (ingest) access to a specific dataset |
```bash
pb role add log-readers
```
### List roles
```bash
pb role list
```
```bash
pb role list --output json
```
### Delete a role
```bash
pb role remove
pb role rm
```
## Users
### Create a user
```bash
pb user add
```
The server generates a password and prints it once. Keep it somewhere safe because it cannot be retrieved later.
Assign roles at creation time:
```bash
pb user add bob --role log-readers,developers
```
### List users
```bash
pb user list
```
```bash
pb user list --output json
```
### Update user roles
```bash
pb user set-role
```
```bash
pb user set-role bob log-readers,admins
```
### Delete a user
```bash
pb user remove
pb user rm
```
Roles must exist before they can be assigned to a user. Create roles with `pb role add` first.
# AI Agent Access (/pb-cli/agents)
AI coding agents can use the installed `pb` binary through shell commands. The safest setup is to have a human create and verify the profile first, then let the agent use that saved connection.
```bash
pb login
pb status --output json
```
Once the profile is ready, the agent can use it directly. It does not need the API key in its prompt and should not read `~/.config/pb/config.toml`.
## Non-interactive profile setup
For automation, load the API key from a secret store or environment variable. The profile name is up to you.
```bash
pb profile add automation-readonly https://parseable.example.com \
--api-key "$PARSEABLE_API_KEY" --output json
```
```bash
pb cloud profile add --name automation-readonly \
--api-key "$PARSEABLE_CLOUD_API_KEY" --output json
```
If more than one profile exists, select the intended one before starting the agent:
```bash
pb profile default automation-readonly --output json
pb status --output json
```
## Prefer read operations and JSON
When an agent needs to read results programmatically, prefer JSON output:
```bash
pb status --output json
pb dataset list --output json
pb dataset info --output json
pb sql run "SELECT * FROM LIMIT 10" --output json
pb promql run "up" --dataset --instant --output json
```
Create a dedicated read-only API key or role with only the query and metadata permissions the agent needs. Never give an agent an administrator key or a shared human credential. Read-only access must be enforced by Parseable on the server; CLI instructions alone cannot prevent create, update, or delete attempts.
Keep API keys out of prompts, source control, logs, and agent instruction files.
# Aliases and Reference (/pb-cli/aliases)
`pb` includes Linux-style aliases for common commands. If you already use commands like `ls`, `rm`, `stat`, and `ps` in the shell, the short forms will feel familiar. The long form is still the better choice in scripts when readability matters more.
| Linux-style alias | Meaning in `pb` |
| ----------------- | -------------------------------- |
| `ls` | List items |
| `rm` | Remove an item |
| `stat` | Show dataset info and statistics |
| `ps` | Show active PromQL queries |
## Aliases
| Long command | Alias |
| ----------------------------------- | ------------------------------------ |
| `pb sql list` | `pb sql ls` |
| `pb dataset list` | `pb dataset ls` |
| `pb dataset info ` | `pb dataset stat ` |
| `pb dataset remove ` | `pb dataset rm ` |
| `pb profile list` | `pb profile ls` |
| `pb profile remove ` | `pb profile rm ` |
| `pb profile update ` | `pb profile set-url ` |
| `pb user list` | `pb user ls` |
| `pb user remove ` | `pb user rm ` |
| `pb role list` | `pb role ls` |
| `pb role remove ` | `pb role rm ` |
| `pb promql active-queries` | `pb promql ps` |
## Command groups
| Area | Commands | What you can do |
| -------------- | --------------------------------------------------------- | --------------------------------------------------------------------------- |
| Query logs | `pb sql` | Run SQL, save queries, list saved queries, open the interactive table view |
| Query metrics | `pb promql` | Run PromQL, inspect labels and series, analyze cardinality, view TSDB stats |
| Stream events | `pb tail` | Watch new events from a dataset |
| Datasets | `pb dataset` | List, inspect, create, and remove datasets |
| Profiles | `pb login`, `pb cloud profile`, `pb profile`, `pb logout` | Manage self-hosted and Parseable Cloud connections |
| Access control | `pb user`, `pb role` | Manage users and roles |
| System | `pb status`, `pb version` | Check connectivity and print CLI version details |
## Help
Use built-in help whenever you want to inspect flags and examples for a command:
```bash
pb --help
pb sql run --help
pb promql run --help
pb dataset add --help
```
# Connecting to a Server (/pb-cli/connect)
`pb` stores server connections as named **profiles**. Each profile keeps the server URL and authentication details for one Parseable instance. If you work with more than one environment, such as staging and production, profiles let you switch between them without re-entering credentials each time.
## Interactive login (recommended)
Run `pb login` to start the interactive setup. Choose self-hosted Parseable or Parseable Cloud, then pick the authentication method that fits that server.
```bash
pb login
```
For self-hosted Parseable, enter the server URL and sign in with either a username and password or an API key. For Parseable Cloud, sign in in the browser or provide a Parseable Cloud API key. The selected Cloud workspace is then saved into the profile.
When the flow finishes, that profile becomes the active default.
## Non-interactive login
For scripts and CI pipelines, use `pb profile add` when you want to create a profile without prompts:
```bash
pb profile add
```
```bash
pb profile add production https://logs.mycompany.com admin s3cr3t
```
Use a self-hosted API key instead of username and password:
```bash
pb profile add production-key https://logs.mycompany.com --api-key psk_xxx
```
Add a Parseable Cloud API-key profile:
```bash
pb cloud profile add --name production-cloud --api-key psk_xxx
```
You can also omit credentials and let `pb` prompt for them:
```bash
pb profile add production https://logs.mycompany.com
```
## Managing profiles
Once you have more than one profile, these are the commands you will use most often:
```bash
pb profile list
pb profile default staging
```
To update or remove a saved profile:
```bash
pb profile update production https://new-url.mycompany.com
pb profile set-url production https://new-url.mycompany.com
pb profile remove old-staging
pb profile rm old-staging
```
## Check connection status
```bash
pb status
```
This checks whether the active profile can reach the server and shows basic server details when they are available:
```
Profile : production
URL : https://logs.mycompany.com
User : admin
Status : Connected (server v1.4.2)
```
## Logout
```bash
pb logout
```
`pb logout` removes the active profile from the local config. If you want to log out of a different profile, switch to it first with `pb profile default `, then run `pb logout`.
If you want to remove a specific saved profile directly, use:
```bash
pb profile remove
pb profile rm
```
## Config file location
Profiles are stored in a TOML config file on disk:
| Platform | Path |
| ------------- | -------------------------- |
| macOS / Linux | `~/.config/pb/config.toml` |
| Windows | `%AppData%\pb\config.toml` |
All commands use the active default profile automatically. Use `pb profile default ` whenever you want to change which server `pb` talks to by default.
# Dataset Management (/pb-cli/datasets)
Datasets are the top-level containers for logs, metrics, and traces in Parseable. Use `pb dataset` to manage them from the terminal.
## List datasets
```bash
pb dataset list
pb dataset ls
```
```bash
pb dataset list --output json
```
## Create a dataset
```bash
pb dataset add --type
```
```bash
pb dataset add backend_logs --type logs
pb dataset add frontend_metrics --type metrics
pb dataset add checkout_traces --type traces
```
If `--type` is omitted, `pb` opens an interactive picker for `logs`, `metrics`, or `traces`.
## Inspect a dataset
```bash
pb dataset info backend_logs
pb dataset stat backend_logs
```
```bash
pb dataset info backend_logs --output json
```
Shows:
* Event count and ingestion size
* Storage size and compression ratio
* Retention period (if configured)
* Active alerts (if any)
* Dataset type (`logs`, `metrics`, or `traces`)
## Delete a dataset
```bash
pb dataset remove
pb dataset rm
```
```bash
pb dataset remove old_logs --type logs
```
Deleting a dataset removes all data it contains. This action cannot be undone.
Use `--type` with `remove` when you want the command to fail if the dataset is not the expected type.
# pb CLI (/pb-cli)
import { Activity, Bot, Command, Database, Download, Link, Radio, Search, ShieldCheck } from 'lucide-react';
`pb` is the official command line interface for self-hosted Parseable and Parseable Cloud. It gives you a terminal-first way to connect to a server, run SQL and PromQL queries, tail live events, manage datasets, and work with users and roles.
You can use it in scripts and CI, or open the interactive terminal views when you want to explore logs and metrics without switching to the browser.
## Start here
} title="Install pb">
Install on macOS, Linux, or Windows.
} title="Connect to a server">
Authenticate and save a server profile.
## Query and monitor
} title="Query logs with SQL">
Search datasets, save queries, or use the interactive table.
} title="Query metrics with PromQL">
Run expressions and explore metric series.
} title="Tail live events" className="col-span-full">
Stream new log events as they arrive.
## Manage and automate
} title="Manage datasets">
Create, inspect, and remove datasets.
} title="Manage users and roles">
Configure access to your Parseable instance.
} title="Aliases and reference">
Find short commands and the full command surface.
} title="Use pb with AI agents">
Set up read-only agent access through a saved profile.
## First commands
After installing `pb`, connect to your Parseable server and check the active profile. The [connection guide](/pb-cli/connect) covers self-hosted and Cloud authentication.
```bash
pb login
pb status
pb sql run "SELECT * FROM backend WHERE status >= 500" --from=1h
pb promql run "rate(http_requests_total[5m])" --dataset otel_metrics --from=1h
pb tail backend
```
## Interactive mode
SQL and PromQL both support an interactive full-screen TUI with `-i`. In this mode, `pb` opens a panel-based interface where you can write a query, select a dataset, adjust the time range, and browse paginated results.
```bash
pb sql run -i
```
```bash
pb promql run -i
```
Navigate between panels with `Tab` and `Shift+Tab`. Press `Ctrl+R` to run the query. Results open in a scrollable table where you can move through rows, inspect columns, and filter what you see.
## Output modes
Commands that return structured data support `--output json`, which makes them easier to use in scripts and automation. SQL queries without `-i` print directly in the terminal. Add `-i` when you want the interactive table view instead.
```bash
pb dataset list
pb dataset list --output json
pb sql run "SELECT * FROM logs LIMIT 50" --from=1h
pb sql run "SELECT * FROM logs LIMIT 50" --from=1h -i
pb sql run "SELECT level, count(*) FROM logs GROUP BY level" --from=1h --output json
pb promql run "up" --dataset otel_metrics --instant --output json
```
# Installation (/pb-cli/install)
## Quick install
The installer downloads the latest release, verifies the SHA-256 checksum, and installs `pb` into `~/.local/bin` by default.
```bash
curl -fsSL https://raw.githubusercontent.com/parseablehq/pb/main/scripts/install.sh | sh
```
Install somewhere else by setting `INSTALL_DIR`:
```bash
curl -fsSL https://raw.githubusercontent.com/parseablehq/pb/main/scripts/install.sh | INSTALL_DIR=/usr/local/bin sh
```
```powershell
powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/parseablehq/pb/main/scripts/install.ps1 | iex"
```
This installs to `%USERPROFILE%\bin` and adds that folder to your user `PATH`. Open a new PowerShell window after installation.
## Homebrew
```bash
brew install parseablehq/tap/pb
```
Use the full tap name. `brew install pb` installs an unrelated Homebrew cask.
## Release archive
If you prefer to install manually, download the latest archive for your operating system and architecture from the [GitHub releases page](https://github.com/parseablehq/pb/releases/latest), extract it, and move the binary to a directory in your `PATH`.
```bash
tar xzf pb_*.tar.gz
chmod +x pb
sudo mv pb /usr/local/bin/
```
Windows archives contain `pb.exe`; extract the `.zip` and move `pb.exe` to a folder in your `PATH`.
| Platform | Archive |
| ------------------- | ---------------------------------- |
| macOS Apple Silicon | `pb__darwin_arm64.tar.gz` |
| macOS Intel | `pb__darwin_amd64.tar.gz` |
| Linux x86 64-bit | `pb__linux_amd64.tar.gz` |
| Linux ARM 64-bit | `pb__linux_arm64.tar.gz` |
| Windows x86 64-bit | `pb__windows_amd64.zip` |
| Windows ARM 64-bit | `pb__windows_arm64.zip` |
On macOS, a manually downloaded binary may be blocked on first run. Allow it once with:
```bash
xattr -d com.apple.quarantine /usr/local/bin/pb
```
## Verify installation
After installation, confirm that `pb` is available on your `PATH`:
```bash
pb --help
```
To print version details:
```bash
pb version
pb --version
```
If you get a `command not found` error, make sure the install directory is included in your `PATH`.
# PromQL Queries (/pb-cli/promql)
Parseable can store metrics data ingested via OpenTelemetry or Prometheus remote write. The `pb promql` commands let you query, explore, and analyze those metrics from the terminal.
## Run a PromQL query
```bash
pb promql run "" --dataset --from
```
```bash
pb promql run "rate(http_requests_total[5m])" --dataset otel_metrics --from=1h --step=1m
```
```bash
pb promql run "up" --dataset otel_metrics --instant
```
### Flags
| Flag | Default | Description |
| ---------------------- | ---------------- | ------------------------------------- |
| `--dataset` / `-d` | `select-dataset` | Metrics dataset to query |
| `--from` / `-f` | `5m` | Start of query window |
| `--to` / `-t` | `now` | End of query window |
| `--step` | `auto` | Resolution step for range queries |
| `--instant` | false | Run as instant query instead of range |
| `--output` / `-o` | `text` | Output format: `text` or `json` |
| `--interactive` / `-i` | false | Open interactive TUI |
## Interactive TUI
```bash
pb promql run -i
pb promql run "http_requests_total" --dataset otel_metrics --from=1h -i
```
The TUI has panels for dataset, query expression, time range, step size, and results. Navigate with `Tab` / `Shift+Tab`, run with `Ctrl+R`, and press `Space` on the step panel to toggle between range and instant mode.
The PromQL TUI also includes a builder flow for exploring metrics, labels, and values from the selected dataset before inserting an expression.
## Explore labels and series
List all label names in a metrics dataset:
```bash
pb promql labels --dataset otel_metrics
```
List all values for a specific label:
```bash
pb promql label-values job --dataset otel_metrics
```
Find series matching a selector:
```bash
pb promql series --match 'http_requests_total' --dataset otel_metrics
pb promql series --match '{job="api-server"}' --dataset otel_metrics
```
## Cardinality analysis
High cardinality labels are a common cause of performance issues. These commands help you find them.
Labels with the most distinct values:
```bash
pb promql cardinality label-names --dataset otel_metrics --limit 20
```
Series count per value for a specific label:
```bash
pb promql cardinality label-values --label job --dataset otel_metrics
```
Currently active series:
```bash
pb promql cardinality active-series --dataset otel_metrics --selector '{job="api"}'
```
## TSDB statistics
```bash
pb promql tsdb --dataset otel_metrics --top 10
```
Shows storage statistics for the metrics dataset, broken down by the top N series by size.
## Active queries
```bash
pb promql active-queries
pb promql ps
```
Lists any PromQL queries currently running on the server. Useful for debugging slow or stuck queries.
# SQL Queries (/pb-cli/query)
`pb sql` runs SQL queries against Parseable datasets. Use it for quick one-off searches, JSON output in scripts, saved queries, or the interactive table TUI.
## Basic usage
```bash
pb sql run "SELECT * FROM backend" --from=10m --to=now
```
`--from` and `--to` define the time window to query. The dataset name is the SQL `FROM` target.
## Time range syntax
| Format | Example | Meaning |
| -------- | ---------------------- | ----------------------------- |
| Relative | `10m`, `2h`, `7d` | Last N minutes / hours / days |
| Absolute | `2024-01-15T00:00:00Z` | Exact RFC3339 timestamp |
| Keyword | `now` | Current time |
Last 30 minutes:
```bash
pb sql run "SELECT * FROM logs LIMIT 100" --from=30m
```
Specific time window:
```bash
pb sql run "SELECT count(*) FROM logs" --from=2024-01-01T00:00:00Z --to=2024-01-02T00:00:00Z
```
Default (last 1 minute if `--from` is not specified):
```bash
pb sql run "SELECT * FROM logs"
```
## Output modes
Without `-i`, `pb sql run` prints the query response directly to the terminal. Use `--output json` when you need JSON for scripts. Use `-i` when you want the interactive table UI.
```bash
pb sql run "SELECT * FROM logs LIMIT 50" --from=1h
```
```bash
pb sql run "SELECT * FROM logs LIMIT 50" --from=1h --output json
```
```bash
pb sql run "SELECT * FROM logs LIMIT 50" --from=1h -i
```
## Save and reuse queries
Save a query for later with `--save-as`:
```bash
pb sql run "SELECT level, count(*) FROM logs GROUP BY level" --from=1h --save-as=error-summary
```
Save without running the query:
```bash
pb sql save "SELECT * FROM frontend WHERE status >= 500" --name=frontend-errors --from=1h --to=now
```
List and manage saved queries:
```bash
pb sql list
pb sql ls
pb sql list --output json
```
Without `--output json`, the list command opens an interactive menu for selecting saved queries.
## Interactive TUI mode
Add `-i` to open the full-screen interactive query interface:
```bash
pb sql run -i
pb sql run "SELECT * FROM backend WHERE status = 500" --from=1h -i
```
The TUI includes panels for query editing, time range, dataset selection, columns, and results. Navigate between panels with `Tab` / `Shift+Tab`.
| Key | Action |
| --------------------- | ------------------------ |
| `Tab` / `Shift+Tab` | Switch between panels |
| `Ctrl+R` | Run query |
| `↑` / `↓` | Scroll rows |
| `Shift+↑` / `Shift+↓` | Page up / down |
| Left / Right | Scroll columns |
| `/` | Filter table rows |
| `Esc` | Clear filter |
| `Ctrl+B` | Previous page of results |
Field names containing dots (OTel convention, e.g. `service.name`) or hyphens are automatically quoted in queries. You don't need to add quotes manually.
## Command reference
| Command | Description |
| --------------------- | ----------------------------------------------- |
| `pb sql run [query]` | Run a SQL query and print plain terminal output |
| `pb sql run -i` | Open the interactive SQL table view |
| `pb sql save [query]` | Save a SQL query without running it |
| `pb sql list` | List saved queries |
| `pb sql ls` | Alias for `pb sql list` |
# Live Tail (/pb-cli/tail)
`pb tail` connects to a dataset and streams new events as they arrive. It is useful when you want to watch a service in real time, confirm that ingestion is working, or keep an eye on a pipeline while it is running.
## Usage
```bash
pb tail
```
```bash
# Watch a dataset called "backend"
pb tail backend
```
Events are printed as newline-delimited JSON, one event per line. Press `Ctrl+C` when you want to stop the stream.
```
● watching backend... (ctrl+c to stop)
{"level":"info","msg":"request received","service":"api","ts":"2024-01-15T10:23:01Z"}
{"level":"error","msg":"db timeout","service":"api","ts":"2024-01-15T10:23:02Z"}
```
## Piping to jq
Because the output is newline-delimited JSON, you can pipe it straight into `jq` for filtering and formatting:
```bash
# Pretty-print all events
pb tail backend | jq .
# Show only error-level events
pb tail backend | jq 'select(.level == "error")'
# Extract specific fields
pb tail backend | jq '{time: .ts, msg: .msg}'
```
## Piping to grep
```bash
# Show only lines containing "timeout"
pb tail backend | grep timeout
# Exclude health check noise
pb tail backend | grep -v "health_check"
```
## How it works
`pb tail` uses Apache Arrow Flight over gRPC for streaming. It first reads the gRPC port advertised by the connected Parseable server, then opens the streaming connection on that port. The default self-hosted ports are:
| Port | Protocol | Purpose |
| ------ | -------- | ------------------------------------------- |
| `8000` | HTTP | Main Parseable API (login, query, datasets) |
| `8001` | gRPC | Default streaming port used by `pb tail` |
## Troubleshooting
**Error: `rpc error: code = Unavailable ... dial tcp :8001: i/o timeout`**
This means `pb tail` reached the server over HTTP, but could not reach the gRPC port advertised by that server. `8001` is the default, but a deployment can expose a different value.
Possible causes:
* The advertised gRPC port is blocked by a firewall on the server
* The advertised gRPC port is not exposed in your Docker or Kubernetes setup
* Your network restricts outbound gRPC connections
To fix it, make sure the configured gRPC port is open and reachable from your machine. In a default Docker setup, expose it with `-p 8001:8001`. In a custom deployment, expose the port configured by the server and allow outbound TCP access to it.
# Executable Binary (/quickstart/binary)
## Start Parseable
Install the binary and start Parseable in local-store mode:
```sh
curl -fsSL https://logg.ing/install | bash
parseable local-store
```
On Windows, use the installation script in PowerShell:
```powershell
powershell -c "irm https://logg.ing/install-windows | iex"
```
The local instance listens on port `8000`. Open [the Parseable console](http://localhost:8000) and sign in with the default credentials, `admin:admin`.
## Verify and continue
When the console loads, follow the [quickstart](/get-started) to send one event to the `demo` dataset and confirm it appears in the log explorer. For production deployment options, see [self-hosted installation](/self-hosted/installation).
# Parseable Cloud (/quickstart/cloud)
Get started with Parseable Cloud in minutes — no infrastructure to manage.
## Sign Up
1. Go to [app.parseable.com](https://app.parseable.com) and create an account
2. Verify your email address
## Create a Workspace
1. Click **Create Workspace**
2. Choose your preferred **region** (select the region closest to your data sources for best performance)
3. Wait a few minutes for your Parseable workspace to be provisioned
Once ready, you'll receive your unique **ingestion endpoint**.
## Start Ingesting Data
Use the endpoint provided to start sending logs. Here's an example using curl:
```bash
curl --location --request POST \
'https://.parseable.com/api/v1/ingest' \
--header 'X-P-Stream: demo' \
--header 'Authorization: Basic ' \
--header 'Content-Type: application/json' \
--data-raw '[
{
"id": "434a5f5e-2f5f-11ed-a261-0242ac120002",
"datetime": "2023-01-05T07:20:50.52Z",
"host": "153.10.110.81",
"message": "Hello from Parseable Cloud!"
}
]'
```
Replace `` and `` with the values from your workspace dashboard.
## Query Your Data
Once data is ingested, you can query it directly from the Parseable Cloud console at [app.parseable.com](https://app.parseable.com).
## Benefits of Parseable Cloud
* **No infrastructure management** — We handle scaling, updates, and maintenance
* **Pay-as-you-go pricing** — Only pay for what you use
* **Global regions** — Deploy close to your data sources
* **Instant setup** — Start ingesting in minutes
* **Enterprise-grade security** — SOC 2 compliant, encrypted at rest and in transit
# Docker Image (/quickstart/docker)
## Start Parseable
Install [Docker](https://www.docker.com/products/docker-desktop/) if it is not already available, then run:
```sh
docker run -p 8000:8000 \
-p 8001:8001 \
-p 8002:8002 \
-v /tmp/parseable/data:/parseable/data \
-v /tmp/parseable/staging:/parseable/staging \
-e P_FS_DIR=/parseable/data \
-e P_STAGING_DIR=/parseable/staging \
quay.io/parseablehq/parseable:latest \
parseable local-store
```
The container listens on port `8000`. Open [the Parseable console](http://localhost:8000) and sign in with the default credentials, `admin:admin`.
## Verify and continue
When the console loads, follow the [quickstart](/get-started) to send one event to the `demo` dataset and confirm it appears in the log explorer. For production deployment options, see [self-hosted installation](/self-hosted/installation).
# AWS (/self-hosted/aws)
This document provides an overview of the AWS services that Parseable integrates with, and how to set up the integration.
### Authentication and access control
Parseable uses AWS IAM roles to authenticate and authorize access to AWS services. You can create an IAM role with the necessary permissions and attach it to the Parseable instance to enable access to AWS services.
### Deployment and configuration
Parseable can be deployed in AWS using the following methods:
* AWS EKS: Install Parseable in your EKS cluster using [this guide here](/self-hosted/installation/standalone/aws-eks).
* AWS ECS: Install Parseable in your ECS cluster using [this guide here](/self-hosted/installation/standalone/aws-ecs).
* AWS EC2: You can install Parseable on an EC2 (with an Ubuntu 22.04 LTS AMI) using the [standard Linux installation guide](/self-hosted/installation/standalone/linux).
### Ingest logs and events from AWS services
Parseable can ingest logs and events from various AWS services, including:
* AWS Lambda: Parseable can ingest logs from AWS Lambda functions. You can use the Lambda function log ingestion guide to set up the integration.
* AWS Kinesis Firehose: Parseable can ingest logs from Kinesis Firehose. You can use the Kinesis Firehose log ingestion guide to set up the integration.
* AWS S3: Parseable can ingest logs from S3. You can use any log agent's S3 source to ingest logs from S3. For example Vector, FileBeat, etc.
[Request an integration](https://github.com/parseablehq/parseable/issues/new)
# Configuration (/self-hosted/configuration)
This document lists all the environment variables and HTTP headers supported by Parseable server.
### Common environment variables
| Variable Name | Required | Description | Default | Example | | |
| ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | --------------------------------------------------------- | ----- | ---- |
| `P_USERNAME` | `Yes` | Username for the admin user. Will be used to access API and UI. | admin | `AKIAIOSFODNN7` | | |
| `P_PASSWORD` | `Yes` | Password for the admin user. Will be used to access API and UI. | admin | `wJalrXUtnaYrOq7phc6l` | | |
| `RUST_LOG` | `No` | Control the log level of Parseable server. By default, all logging is disabled, except for the error level. Refer the docs here for possible values. | error | `info` | | |
| `P_ADDR` | `No` | Address (IP Address and Port without the scheme) on which the Parseable server would listen for new connections. | `127.0.0.1:8000` | `127.0.0.1:7000` | | |
| `P_GRPC_PORT` | `No` | Port to be used for gRPC response. | `8001` | `5001` | | |
| `P_FLIGHT_PORT` | `No` | Port to be used for Arrow Flight response. | `8002` | `5002` | | |
| `P_TLS_CERT_PATH` | `No` | Location of the TLS Cert file on the server. Use this and P\_TLS\_KEY\_PATH variable together to enable TLS on your Parseable server. | - | `/home/user/fullchain.pem` | | |
| `P_TLS_KEY_PATH` | `No` | Location of TLS Private key file on the server. Use this and P\_TLS\_CERT\_PATH variable together to enable TLS on your Parseable server. | - | `/home/user/privkey.pem` | | |
| `P_STAGING_DIR` | `No` | Path on the local machine, where the Parseable server would stage data before pushing it to storage. | `$PWD/staging` | `/home/user/parseable/staging` | | |
| `P_CHECK_UPDATE` | `No` | Specify whether server should check for new updates from Parseable download server. | `true` | `false` | | |
| `P_SEND_ANONYMOUS_USAGE_DATA` | `No` | Specify whether the server should send anonymous usage data to Parseable analytics. | `true` | `false` | | |
| `P_PARQUET_COMPRESSION_ALGO` | `No` | Specify the compression algorithm to use for Parquet files. Support values are `UNCOMPRESSED`, `SNAPPY`, `GZIP`, `LZO`, `BROTLI`, `LZ4`, `ZSTD`. | `LZ4` | `GZIP` | | |
| `P_OPENAI_API_KEY` | `No` | Specify your OpenAI API key to generate SQL automatically from plain text. | - | `sk-open-ai-api-key` | | |
| `P_CORS` | `No` | Specify whether the server should enable/disable CORS. Supported values are `true` to disable, `false` to enable CORS | `true` | `false` | | |
| `P_ALLOW_ORIGINS` | `No` | Comma-separated list of allowed CORS origins. Will be effective only if `P_CORS=true`. | `No` | Specify the memory threshold to use for Parseable server. | `100` | `80` |
| `P_CPU_THRESHOLD` | `No` | Specify the CPU threshold to use for Parseable server. | `100` | `80` | | |
| `P_ENABLE_MULTIPART` | `No` | Enable multipart uploads to object store (S3, GCS, Azure Blob Store). When disabled, files are uploaded as a single object regardless of size. | `true` | `false` | | |
| `P_MULTIPART_MIN_SIZE` | `No` | Minimum file size (in bytes) for multipart uploads to object store. Files smaller than this are uploaded as a single object. Minimum allowed value is `26214400` (25 MB). Will be effective only if `P_ENABLE_MULTIPART` is `true`. | `26214400` | `52428800` | | |
| `P_MAX_EVENT_PAYLOAD_SIZE` | `No` | Maximum allowed event payload size in bytes for ingest endpoints. | `10485760` | `20971520` | | |
| `P_MAX_FLATTEN_LEVEL` | `No` | Maximum level of flattening allowed for events. Prevents nested list type fields from getting created. | `10` | `5` | | |
| `P_QUERY_MEMORY_LIMIT` | `No` | Set a fixed memory limit for query in GiB. | `-` | `4` | | |
| `P_SQL_TIMEOUT` | `No` | SQL query execution timeout in seconds. Queries exceeding this duration return a timeout error. | `300` | `120` | | |
| `P_ACTIX_REQUEST_TIMEOUT` | `No` | Client request timeout in seconds. | `5` | `10` | | |
| `P_ACTIX_KEEP_ALIVE` | `No` | Server keep-alive in seconds. | `5` | `10` | | |
| `P_ACTIX_NUM_WORKERS` | `No` | Number of workers for actix-web. Defaults to the number of CPU cores. | `` | `8` | | |
| `P_ACTIX_BACKLOG` | `No` | Maximum number of pending connections. | `2048` | `4096` | | |
| `P_ACTIX_MAX_CONNECTIONS` | `No` | Per worker maximum number of concurrent connections. | `25000` | `50000` | | |
| `P_MULTI_TENANCY` | `No` | Enable tenant-aware request handling and tenant-scoped storage layout. See [Multi-tenancy](/user-guide/multi-tenancy). | `false` | `true` | | |
### Applicable to distributed mode
| Variable Name | Required | Description | Default | Example |
| -------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------- |
| `P_MODE` | `Yes` | Mode for this Parseable instance. Can be INGEST or QUERY or ALL. If set to ALL, instance will behave as a standalone deployment. | `ALL` | `INGEST` |
| `P_HOT_TIER_DIR` | `No` | Path on the query node (or standalone node), where the Parseable server would store recent data. Refer [Hot Tier documentation](/user-guide/smart-cache) for details. | `-` | `/home/user/hot-tier` |
| `P_MAX_DISK_USAGE_PERCENT` | `No` | Maximum percentage of total disk that should be used for hot tier. Refer [Hot Tier documentation](/user-guide/smart-cache) for details. | `80` | `70` |
| `P_INGESTOR_ENDPOINT` | `No` | Endpoint (IP, DNS or URL and Port, without the scheme) of the ingestor. If set, the query node will use this address to access this Ingestor. If not set, the query node will use the P\_ADDR value if set. If `P_ADDR` is also not set, then defaults to `0.0.0.0:8000`. You can also set this variable to point to another environment variable. For example, if `P_INGESTOR_ENDPOINT` is set to `$HOSTNAME:$PORT`, Parseable process will look for environment variables `HOSTNAME` and `PORT` and replace their values in the `P_INGESTOR_ENDPOINT`. | `0.0.0.0:8000` | `ingestor1.parseable.svc.cluster.local` |
### Applicable to the S3 storage mode
This section lists all the environment variables applicable to the S3 storage mode. This is applicable to AWS S3 or any other S3 compatible object storage platform.
| Variable Name | Required | Description | Default | Example |
| -------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------- |
| `P_S3_URL` | `Yes` | URL for S3 or compatible object storage server. | `-` | `https://s3.us-east-1.amazonaws.com` |
| `P_S3_ACCESS_KEY` | `Yes` | Access key for S3 or compatible object storage server. | `-` | `AKIAIOSFODNN7EXAMPLE` |
| `P_S3_SECRET_KEY` | `Yes` | Secret key for S3 or compatible object storage server. | `-` | `wJalrXUtnaEXAMPLEKEY` |
| `P_S3_BUCKET` | `Yes` | Bucket to use for Parseable data storage. | `-` | `parseable` |
| `P_S3_REGION` | `Yes` | Region for the object storage platform. | `-` | `us-east-1` |
| `P_S3_PATH_STYLE` | `No` | Force Parseable to use [Path style access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html#path-style-url-ex) to S3 store. | `true` | `false` |
| `P_S3_TLS_SKIP_VERIFY` | `No` | Skip checking for S3 store's TLS certificate validity. | `false` | `true` |
| `P_S3_CHECKSUM` | `No` | Set SHA256 checksum in requests to allow S3 buckets with [WORM enabled](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html). | `false` | `true` |
| `P_S3_SSEC_ENCRYPTION_KEY` | `No` | Set server side encryption key with [customer provided key for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html). | `-` | `SSE-C:AES256:lgYvIsNHaYFh45knxlpxCdLFJaLnmXiibQcDrUYZt9Q=` |
### Applicable to the GCS storage mode
This section lists all the environment variables applicable to the Google Cloud Storage (GCS) mode. These environment variables are mandatory if you're starting Parseable server in GCS mode i.e. `parseable gcs-store`. Authentication uses [Google Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
| Variable Name | Required | Description | Default | Example |
| ----------------------- | -------- | ----------------------------------------------------------- | -------------------------------- | -------------------------------- |
| `P_GCS_BUCKET` | `Yes` | GCS bucket to use for Parseable data storage. | `-` | `parseable` |
| `P_GCS_URL` | `No` | URL for GCS or compatible object storage server. | `https://storage.googleapis.com` | `https://storage.googleapis.com` |
| `P_GCS_TLS_SKIP_VERIFY` | `No` | Skip checking for the GCS store's TLS certificate validity. | `false` | `true` |
### Applicable to the Azure storage account
This section lists all the environment variables applicable to the Azure blobstore storage mode. These environment are mandatory, if you're starting Parseable server in Azure storage mode i.e. `parseable blob-store`.
| Variable Name | Required | Description | Default | Example |
| --------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------------------------- |
| `P_AZR_URL` | `Yes` | URL to communicate with blob storage. | `-` | `https://parseable.blob.core.windows.net` |
| `P_AZR_ACCOUNT` | `Yes` | Azure storage account name. [Refer the docs here](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview). | `-` | `parseable` |
| `P_AZR_CONTAINER` | `Yes` | Container name created in the storage storage account. | `-` | `parseable` |
| `P_AZR_ACCESS_KEY` | `No` | Access key to authenticate azure storage account. | `-` | `AKIAIOSFODNN7EXAMPLE` |
| `P_AZR_CLIENT_ID` | `No` | Client ID of app registered in Azure AD to authenticate azure storage account. | `-` | `AKIAIOSFODNN7EXAMPLE` |
| `P_AZR_CLIENT_SECRET` | `No` | Client secret of app registered in Azure AD to authenticate azure storage account. | `-` | `wJalrXUtnaEXAMPLEKEY` |
| `P_AZR_TENANT_ID` | `No` | Tenant ID associated with your Azure AD | `-` | `AKIAIOSFODNN7EXAMPLE` |
### Applicable to AWS
This section lists all the environment variables applicable to the AWS blobstore storage mode. These environment are mandatory, if you're staring Parseable server in AWS storage mode i.e. `parseable server blob-store`.
| Variable Name | Required | Description | Default | Example |
| ------------------------- | -------- | ---------------------------------------------------------------- | ------------------------ | ---------------------- |
| `P_AWS_PROFILE_NAME` | `No` | Set AWS profile name which will be used for fetching credentials | `-` | `default` |
| `P_AWS_IMDSV1_FALLBACK` | `No` | Sets if object store client should fallback to imdsv1. | `false` | `true` |
| `P_AWS_METADATA_ENDPOINT` | `No` | Sets AWS instance metadata endpoint to use. | `http://169.254.169.254` | `http://fd00:ec2::254` |
### Applicable to local drive mode
This section lists all the environment variables applicable to the local drive storage mode. These environment are mandatory, if you're staring Parseable server in local drive storage mode i.e. `parseable server blob-store`.
| Variable Name | Required | Description | Default | Example |
| ------------- | -------- | ------------------------------------------------------------------ | ----------- | --------------------------- |
| `P_FS_DIR` | `No` | Path on the local machine where Parseable server would store data. | `$PWD/data` | `/home/user/parseable/data` |
### Applicable to OIDC
This section lists all the environment variables applicable to the OIDC mode. These environment are mandatory, if you're staring Parseable server in OIDC mode i.e. `parseable server oidc`.
| Variable Name | Required | Description | Default | Example |
| ---------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ----------------------------- |
| `P_OIDC_CLIENT_ID` | `Yes` | Your OIDC client identifier, provided by your identity provider. | `-` | `client-id` |
| `P_OIDC_CLIENT_SECRET` | `Yes` | Your OIDC client secret, provided by your identity provider. | `-` | `client-secret` |
| `P_OIDC_ISSUER` | `Yes` | The OIDC issuer URL, typically provided by your identity provider. It points to the OIDC authorization server. Should support discovery protocol. | `-` | `https://accounts.google.com` |
| `P_ORIGIN_URI` | `Yes` | The URI where Parseable is hosted or accessible. This should be the base URL of your Parseable instance. | `-` | `https://demo.parseable.com/` |
| `P_OIDC_SCOPE` | `No` | OIDC scope to request from the identity provider. | `openid profile email offline_access` | `openid profile email` |
Refer to [OIDC](/user-guide/openid) section for more details.
### Applicable to Parseable Enterprise
This section lists environment variables that apply only to Parseable Enterprise deployments.
| Variable Name | Required | Description | Default | Example |
| ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------------------------------------- |
| `P_CLUSTER_SECRET` | `Yes` | Shared secret used to authenticate inter-node communication within a Parseable Enterprise cluster. The server fails to start if this is not set. | `-` | `` |
| `P_LICENSE_DATA_FILE_PATH` | `Yes` | Local path to the Parseable Enterprise license data file (JSON). | `-` | `/etc/parseable/license.json` |
| `P_LICENSE_SIGNATURE_FILE_PATH` | `Yes`\* | Local path to the raw (binary) license signature file. *Either this or `P_LICENSE_SIGNATURE_BASE64` must be set.* | `-` | `/etc/parseable/license.sig` |
| `P_LICENSE_SIGNATURE_BASE64` | `Yes`\* | Base64-encoded license signature provided inline as an environment variable. Takes priority over `P_LICENSE_SIGNATURE_FILE_PATH` when both are set. | `-` | `MEUCIQDx...==` |
| `P_DB_CONNECTION_STRING` | `No` | PostgreSQL connection string used for storing Parseable Enterprise metadata. Set `DB_SSL_ROOT_CERT` to a CA cert path to enable TLS. | `-` | `postgresql://user:pass@host:5432/parseable` |
| `P_INGESTOR_URL_INFO` | `No` | Public ingestor URL exposed by the `/about` endpoint and used by the OTel generator. Set this to the externally reachable URL of your ingestor(s). | `-` | `https://ingest.example.com` |
| `P_SERVER_LOGS_FILE_PATH` | `No` | File path where Parseable server logs are written. If unset, server logs are not persisted to file. | `-` | `/var/log/parseable/server.log` |
| `P_AUDIT_LOGS_FILE_PATH` | `No` | File path where Parseable audit logs (JSON) are written. If unset, audit logs are not persisted to file. | `-` | `/var/log/parseable/audit.log` |
| `P_LOGS_MAX_FILE_SIZE` | `No` | Maximum size (in bytes) of a single rotated log file. | `10485760` (10 MB) | `52428800` |
| `P_LOGS_MAX_TOTAL_SIZE` | `No` | Maximum total size (in bytes) of all retained log files. Must be greater than or equal to `P_LOGS_MAX_FILE_SIZE`. | `1073741824` (1 GB) | `5368709120` |
# Parseable Metrics (/self-hosted/metrics)
## Export Parseable to Prometheus
Prometheus offers a multi-dimensional data model with time series data identified by metric name and key/value pairs. The data collection happens via a pull model over HTTP/HTTPS.
Parseable server exposes Prometheus metrics at the `/api/v1/metrics` endpoint in Prometheus exposition format. If authentication is enabled for your Parseable instance, scrape the endpoint with valid Parseable credentials.
## Prerequisites
Parseable server installed and running. See installation guide for more details.
Prometheus installed and running. See installation guide for more details.
## Configuration
Prometheus configuration is done via a YAML file. Locate your Prometheus configuration file `prometheus.yml` and add the following section to scrape Parseable server metrics.
```yaml
scrape_configs:
- job_name: parseable-job
metrics_path: /api/v1/metrics
scheme: http
static_configs:
- targets: ['localhost:8000']
basic_auth:
username: "admin"
password: "admin"
```
Make sure to replace localhost:8000 with the actual host and port where Parseable server is running.
### List of metrics exposed by Parseable
Parseable server exposes the following metrics on the `/api/v1/metrics` endpoint. All of these can be accessed via Prometheus dashboard. A sample list of exposed metrics along with their definition is available in the demo server at `https://demo.parseable.com/api/v1/metrics`.
For a local self-hosted instance, authenticate with your configured Parseable credentials:
```bash
curl -u admin:admin http://localhost:8000/api/v1/metrics
```
# Telemetry (/self-hosted/telemetry)
### Overview
Parseable collects anonymized usage statistics from users in order to improve the server. You can deactivate at any time by setting `P_SEND_ANONYMOUS_USAGE_DATA=false`.
Any data that has already been collected can be deleted on request.
### Why is data collected?
Parseable is a young project and we want to make it fast, reliable and resource efficient. While we perform several benchmarks internally, it is impossible to capture all the scenarios, hardware and software combinations that Parseable is used in.
In order to identify performance characteristics and improve Parseable, we need to collect information about how it is used.
### What data is collected?
Broadly, there are two types of data collected:
* System information - general information about the system, such as CPU, RAM, and disk type. As well as the configuration of the Parseable instance.
* Usage - information about data stored and compressed.
We do not collect any data that can be used to identify the user or the user’s organization. This includes:
* Any data that can be used to identify the user or the user’s organization
* Any data stored in the dataset
* Any names of the dataset
* Any URLs
### Sample JSON data
This is a sample data received by our servers, when a Parseable installation reports anonymous telemetry.
```json
{
"commit_hash": "f053d2f",
"cpu_count": 8,
"deployment_id": "01GTB8PGHM5AK9XQXPMTTCBQ0S",
"memory_total_bytes": 16287862784,
"metrics": {
"cpu_cpu0_usage_percent": 1.5296164751052856,
"cpu_cpu1_usage_percent": 1.6184406280517578,
"cpu_cpu2_usage_percent": 2.738511800765991,
"cpu_cpu3_usage_percent": 2.094792366027832,
"cpu_cpu4_usage_percent": 2.183173179626465,
"cpu_cpu5_usage_percent": 1.5540571212768557,
"cpu_cpu6_usage_percent": 1.4056124687194824,
"cpu_cpu7_usage_percent": 1.449198842048645,
"memory_free_bytes": 349634560,
"memory_in_use_bytes": 6138183680,
"stream_count": 7,
"total_events_count": 864611,
"total_json_bytes": 396726688,
"total_parquet_bytes": 311480071
},
"mode": "S3 bucket",
"os_name": "Debian GNU/Linux",
"os_version": "11",
"platform": "Docker",
"report_created_at": "2023-04-25T10:00:10.000638350Z",
"uptime_secs": 12469729,
"version": "0.4.0"
}
```
### Request data deletion
To request data deletion, send an email to [hi@parseable.com](mailto:hi@parseable.com) containing the unique `Deployment UID` generated for your Parseable installation.
You can find this identifier in the server startup banner under the field `Deployment UID`.
# AWS S3 (/storage/awss3)
This document is a deep dive into Parseable and AWS S3 setup
## Prerequisites
* New Parseable cluster requires a fully empty S3 bucket.
* For optimum performance, please ensure the S3 bucket is in the same region as the compute (EC2, EKS, ECS, Lightsail or others) instances.
## Authentication
Parseable supports authentication to S3 via
* Access Key and Secret Key: The AWS access key and the secret key can be used to authenticate to AWS S3 bucket. To use this method, set the environment variables `P_S3_ACCESS_KEY` and `P_S3_SECRET_KEY` before starting the Parseable server(s).
* Instance Metadata Service (IMDS): For Parseable instances running on EC2, AWS credentials can be sourced from the Instance Metadata Service (IMDS), avoiding the need for explicit `P_S3_ACCESS_KEY` and `P_S3_SECRET_KEY`. To use this method,
* Ensure that `Instance Metadata Service (IMDS)` is enabled when creating the EC2 instance (under Advanced details section). Select the Metadata version to `V1` and `V2` (token optional). Please refer to the [metadata service docs](https://docs.aws.amazon.com/sdkref/latest/guide/feature-imds-credentials.html) for more. You set `P_AWS_IMDSV1_FALLBACK` to true if you want to use the V1 method.
* By default, Parseable uses the standard AWS metadata endpoint. If you’re using a custom metadata endpoint, add the optional environment variable `P_AWS_METADATA_ENDPOINT` to specify a [custom endpoint URL](https://docs.aws.amazon.com/sdkref/latest/guide/feature-imds-credentials.html) for retrieving instance metadata.
* IAM Roles for Service Accounts (IRSA): This is useful for EKS deployments. With IRSA you can associate an IAM role with a Kubernetes service account and configure Parseable to use the service account. To use this method, refer to [Parseable EKS documentation](https://www.parseable.com/docs/self-hosted/installation/standalone/aws-eks).
## SSE-C Support
Parseable supports server side encryption for [AWS S3 with customer provided encryption keys (SSE-C)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html). With SSE-C, you can store your data encrypted with your own encryption keys. Amazon S3 or a compatible service like MinIO manages data encryption and decryption transparently.
Note that SSE-C requires HTTPS and Amazon S3 or compatible service might reject any requests made over HTTP when using SSE-C.
## Setting up
The encryption key must be a 256-bit key for AES-256 encryption. To add SSE-C encryption key, add the environment variable `P_S3_SSEC_ENCRYPTION_KEY` before starting the server. The value should be in the format `- SSE-C:AES256:`. Here is example of adding the SSE-C encryption key to Parseable
Generate a 256-bit AES key and `Base64` encode it
```bash
ENCRYPTION_KEY=$(openssl rand -base64 32)
echo "Encryption Key: $ENCRYPTION_KEY"
```
Add the `Base64` encoded encryption key to the environment variable
```bash
P_S3_SSEC_ENCRYPTION_KEY=SSE-C:AES256:$ENCRYPTION_KEY
```
For distributed deployments, the environment variable `P_S3_SSEC_ENCRYPTION_KEY` needs to be set for Query as well as Ingestor nodes.
If you lose the encryption key, you’ll lose access to the log data. Hence, we recommend secure storage for the encryption key such as [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) or similar.
# Agent Observability (/user-guide/agent-observability)
Agents are already becoming part of everyday software. Sometimes users work with them directly. Sometimes they sit behind a workflow and quietly make decisions on their own. In both cases, a single request can quickly turn into multiple model calls, tool calls, retries, and intermediate steps before the final answer shows up.
That is usually where debugging gets hard. When the final answer is wrong, slow, or incomplete, the answer itself is not enough. You need to see the full run: what prompt started it, which model was used, what tools were called, what each tool received, what came back, how many tokens were spent, and where the run started to drift. That is the job of Agent Observability.
## Agent observability with Parseable
Parseable Agent Observability gives you that view inside Prism. You can begin with a high-level view of agent activity, look at model and tool behavior separately, and then move into a single agent run when you need the full execution path.
It is built on the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), so the telemetry follows a standard shape for model calls, token usage, inputs, outputs, tool calls, and agent spans. For message content, Parseable follows the native GenAI model: use `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`, and the `gen_ai.client.inference.operation.details` event when you capture full request and response details.
Parseable is not tied to one agent framework. Any supported framework or instrumentation path that sends compatible OpenTelemetry trace data can feed Agent Observability. Use the framework-specific guides when you want setup instructions for a particular stack.
Once your agent is instrumented, the Agents page becomes the place where you understand how that agent behaves in production. You can see how often it runs, which models it uses, how many tokens it consumes, where it spends time, which tools it calls, and where a run needs review.
## Explore agent activity
The Agents page is organized around four tabs: Overview, Models, Tools, and Agent Runs. The dataset selector at the top lets you choose the traces dataset you want to inspect, while the time range picker controls the window used by every chart and table on the page.
### Overview
Start with the Overview tab when you want to understand the shape of agent activity before going deeper. It gives you the main health and usage signals in one place.
The summary cards show total cost, agent runs, errors, total tokens, LLM calls, and tool calls for the selected period. These are useful first checks. You can quickly see whether usage increased, whether errors appeared, and whether the growth is coming from more agent runs, more model calls, more tool calls, or heavier token usage.
Below the summary, the charts break the run down by model, tool, user, and time. Total cost by model shows which model is driving spend. Tool usage shows which tools are being called most often. Token usage by user and model helps you see who or what is consuming tokens. Cost over time, latency by tool, and latency by model help you spot spikes that are easier to miss in a table.
Use the time range picker in the top-right corner when you want to move from a broad view to a smaller investigation window. The whole page follows that selected range.
### Models
The Models tab is for understanding model behavior. Use it when you want to answer questions like: which model is used most, which model is the most expensive, where are tokens being spent, and whether a model is getting slower over time.
The top cards show total cost, most used model, highest spend model, input tokens, output tokens, and cached tokens. This gives you a quick model-level summary before you inspect the charts.
The charts let you compare total cost by model, cost over time by model, tokens by user and model, cost per agent, cost per team, cost by agent run, token usage over time, and model usage over time. This helps you see whether a cost spike came from a model switch, a specific user, a team, or a small number of expensive agent runs.
### Tools
The Tools tab helps you understand how the agent interacts with the outside world. This matters because many agent failures are not model failures. They happen when a tool is slow, returns unexpected output, fails repeatedly, or gets called too often.
The top cards show the most used tool, slowest tool, total tool calls, total agent runs, and errors. The charts below show tool calls over time, tool latency over time, tool usage, and tool failures.
The tool failures table gives you a path back to the run that failed. It includes the start time, tool name, duration, and trace ID, so you can move from a failing tool trend to the exact trace that explains it.
### Agent Runs
The Agent Runs tab is where you move from charts to individual runs. Use it when you need to inspect a specific prompt, a failed run, an expensive run, or a run that called too many tools.
The timeline at the top shows when runs happened. The table below lists each run with date, prompt, model, tokens, cost, duration, and operations. Operations show how many model interactions and tool calls happened inside the run, including failed tool calls when they exist.
Use Add filter to narrow the table when you already know what kind of run you are looking for. You can filter by fields such as model, tool, token counts, duration, errors, and other captured attributes. Use Find in data when you want to search within the currently visible results.
When you open a run, Parseable shows the trace behind it. User-facing input appears as user messages in the UI. Model or agent output appears as agent messages. Under the hood, keep this telemetry native to OpenTelemetry by recording message content with `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.client.inference.operation.details` rather than relying on custom message event names.
## Instrument agents with supported frameworks
Parseable supports Agent Observability for multiple agent frameworks and instrumentation libraries. The exact setup depends on the framework, but the data path is the same: capture agent, model, and tool telemetry, export it through OpenTelemetry, and send the trace dataset to Parseable with the `agent-observability` dataset tag.
At a high level, the flow looks like this:
```text
Supported agent framework
|
| OpenTelemetry traces
v
OpenTelemetry Collector
|
v
Parseable traces dataset
|
v
Agents page in Prism
```
Start with the guide for your framework:
| Framework guide | What it covers | Framework guide | What it covers |
| ----------------------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------- |
| [Pydantic AI](/ingest-data/ai-agents/pydantic-ai) | Agent, model, and tool-call traces | [CrewAI](/ingest-data/ai-agents/crewai) | Crews, agents, tasks, LLM calls, and errors |
| [Restate](/ingest-data/ai-agents/restate-openai-agents) | Durable agent runs, model calls, tools, retries, and recovery | [n8n](/ingest-data/ai-agents/n8n) | Workflow automation traces and related telemetry |
| [Mastra](/ingest-data/ai-agents/mastra) | Agent logs, traces, and metrics | [Hermes Agent](/ingest-data/ai-agents/hermes) | Agent invocations, token usage, model calls, tools, logs, and metrics |
| [OpenLIT](https://www.parseable.com/docs/ingest-data/ai-agents/openlit) | Agent, workflow, tool, and model telemetry | [LangChain](/ingest-data/ai-agents/langchain) | LangChain application traces |
| [LlamaIndex](/ingest-data/ai-agents/llamaindex) | Retrieval and generation workflow traces | [AutoGen](/ingest-data/ai-agents/autogen) | Multi-agent conversations and failures |
| [DSPy](/ingest-data/ai-agents/dspy) | DSPy programs and prompt optimization workflows | | |
After choosing a framework guide, use the collector or direct OTLP export options below to send the resulting traces to Parseable.
## Framework support matrix
Agent Observability is framework-aware when the incoming traces include agent, model, and tool attributes. Some frameworks emit those attributes directly. Others use instrumentation libraries such as [OpenLIT](https://www.parseable.com/docs/ingest-data/ai-agents/openlit) or [OpenInference](https://github.com/Arize-ai/openinference) to translate framework activity into OpenTelemetry spans.
| Path | Status |
| ------------------------------- | -------------------------------- |
| Supported agent framework guide | Recommended |
| OpenTelemetry Collector export | Recommended for production |
| Direct OTLP export to Parseable | Useful for small or local setups |
***
## Collector configuration
After instrumentation is in place, the next step is deciding how you want to send the trace data to Parseable. For most teams, an OpenTelemetry Collector is the better default because it gives you batching and a more reliable path. If your setup is small, you can also export directly from the application.
### OpenTelemetry Collector (Recommended)
Put an OpenTelemetry Collector between your application and Parseable when you want a cleaner and more production-friendly path for export. Save the following as `parseable-genai-collector.yaml`:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 256
exporters:
otlphttp/parseable:
endpoint: ${PARSEABLE_URL}
encoding: json
headers:
Authorization: "Bearer ${PARSEABLE_API_KEY}"
X-P-Stream: "${STREAM_NAME}"
X-P-Dataset-Tag: "agent-observability"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable]
```
Run the collector:
```bash
export PARSEABLE_URL=${PARSEABLE_URL} # e.g. https://ingest.parseable.com
export PARSEABLE_API_KEY=${PARSEABLE_API_KEY}
export STREAM_NAME=${PARSEABLE_DATASET_NAME}
otelcol-contrib --config parseable-genai-collector.yaml
```
### Direct to Parseable
For smaller setups, you can send traces directly from the application to Parseable with OTLP environment variables. This avoids running a separate collector process.
```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=${PARSEABLE_URL}
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${PARSEABLE_API_KEY},X-P-Stream=${PARSEABLE_DATASET_NAME},X-P-Dataset-Tag=agent-observability"
export OTEL_SERVICE_NAME=my-agent
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
```
## SQL query templates
The Agents page should be the first place you inspect agent behavior. When you need to go deeper, you can query the same traces dataset directly with SQL. The examples below assume a dataset named `genai-traces`. Replace it with your dataset name if you use something different.
Span rows have `event_name IS NULL`. Most analytics queries should include this filter so span data is not mixed with event rows that carry prompt, completion, or other event-level content.
### LLM call summary
Use this when you want a quick per-model view of call volume, token usage, latency, and errors over the last 24 hours.
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS call_count,
SUM("gen_ai.usage.input_tokens") AS total_input_tokens,
SUM("gen_ai.usage.output_tokens") AS total_output_tokens,
SUM(p_genai_tokens_total) AS total_tokens,
ROUND(AVG(p_genai_duration_ms), 2) AS avg_latency_ms,
ROUND(100.0 * SUM(CASE WHEN span_status_code = 2 THEN 1 ELSE 0 END) / COUNT(*), 2) AS error_rate_pct
FROM "genai-traces"
WHERE p_timestamp > NOW() - INTERVAL '24 hours'
AND event_name IS NULL
AND "gen_ai.operation.name" = 'chat'
GROUP BY "gen_ai.request.model"
ORDER BY total_tokens DESC;
```
### Reconstruct an agent run
Use this after you find a trace ID in the Agent Runs table. It lists the spans in that run in the order they happened.
```sql
SELECT
span_name,
"gen_ai.operation.name" AS operation,
ROUND(p_genai_duration_ms, 2) AS duration_ms,
"gen_ai.usage.input_tokens" AS input_tokens,
"gen_ai.usage.output_tokens" AS output_tokens,
span_status_description AS status,
span_parent_span_id
FROM "genai-traces"
WHERE span_trace_id = ''
AND event_name IS NULL
ORDER BY span_start_time_unix_nano;
```
### Token usage over time
Use this to see hourly token consumption by model over the last 7 days.
```sql
SELECT
DATE_TRUNC('hour', p_timestamp) AS hour,
"gen_ai.request.model" AS model,
SUM("gen_ai.usage.input_tokens") AS input_tokens,
SUM("gen_ai.usage.output_tokens") AS output_tokens,
SUM(p_genai_tokens_total) AS total_tokens
FROM "genai-traces"
WHERE p_timestamp > NOW() - INTERVAL '7 days'
AND event_name IS NULL
AND "gen_ai.usage.input_tokens" IS NOT NULL
GROUP BY hour, "gen_ai.request.model"
ORDER BY hour DESC, total_tokens DESC;
```
### Tool usage
Use this to understand which tools are called most often, how long they take, and how often they fail.
```sql
SELECT
"gen_ai.tool.name" AS tool,
COUNT(*) AS invocations,
ROUND(AVG(p_genai_duration_ms), 2) AS avg_duration_ms,
ROUND(100.0 * SUM(CASE WHEN span_status_code = 2 THEN 1 ELSE 0 END) / COUNT(*), 2) AS failure_rate_pct
FROM "genai-traces"
WHERE "gen_ai.operation.name" = 'execute_tool'
AND p_timestamp > NOW() - INTERVAL '24 hours'
AND event_name IS NULL
GROUP BY "gen_ai.tool.name"
ORDER BY invocations DESC;
```
### Recent errors
Use this when you want the latest failing model calls, tool calls, or agent spans.
```sql
SELECT
p_timestamp,
"gen_ai.request.model" AS model,
"gen_ai.operation.name" AS operation,
span_status_message AS error_message,
span_trace_id,
span_span_id,
"gen_ai.agent.name" AS agent
FROM "genai-traces"
WHERE span_status_code = 2
AND event_name IS NULL
AND p_timestamp > NOW() - INTERVAL '24 hours'
ORDER BY p_timestamp DESC
LIMIT 50;
```
### Model comparison
Use this to compare models by latency, throughput, and token usage over the last 7 days.
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS calls,
ROUND(AVG(p_genai_duration_ms), 2) AS avg_latency_ms,
ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY p_genai_duration_ms), 2) AS p95_latency_ms,
ROUND(AVG(p_genai_tokens_per_sec), 2) AS avg_tokens_per_sec,
SUM(p_genai_tokens_total) AS total_tokens
FROM "genai-traces"
WHERE event_name IS NULL
AND "gen_ai.operation.name" = 'chat'
AND p_timestamp > NOW() - INTERVAL '7 days'
GROUP BY "gen_ai.request.model"
ORDER BY total_tokens DESC;
```
# API Keys (/user-guide/api-keys)
You can use API Keys to authenticate API calls to Parseable. They are best suited for log forwarders, CLI tools, automation scripts, AI agents, and any other client where Basic Auth or OIDC is impractical.
A Parseable instance has three types of identities — **Native** (Basic Auth), **OAuth** (OIDC), and **API Key**. API keys behave like users, i.e. they are assigned roles, and each role is a set of [RBAC](/user-guide/rbac) privileges. The key inherits exactly the access its roles grant.
You can create and manage API keys via the Parseable Prism UI or the REST API. This section covers the API operations. For UI, click on your profile on top right corner and then select **Access management > API Keys**.
### When to use API keys
* **Ingestion agents** — Create an API key with the `ingestor` role and use this key to ship telemetry without storing a username and password in the forwarder config.
* **CLI / scripts** — `pb` CLI or any custom automation that needs scoped, revocable credentials.
* **Agents** — AI agents and other autonomous workloads that need read access to specific datasets.
Treat an API key like a password. The full key value is shown **only once** at creation time. Store it in a secrets manager and never commit it to source control.
### How API keys work
An API key has:
* a **key id** (`keyId`) — a stable identifier used for management operations
* a **key name** (`keyName`) — a human-readable label, must be unique within the tenant/organization
* a **secret value** (`apiKey`) — a UUID, sent on each request via the `X-API-Key` header
* a set of **roles** — pre-existing roles that grant privileges (same roles used for Native and OAuth users)
When a request arrives with an `X-API-Key` header, Parseable looks up the key, resolves its roles into permissions, and authorizes the request the same way it authorizes any other user.
### Create an API key
Roles must already exist in the tenant/organization. See [RBAC](/user-guide/rbac) for how to create roles.
```sh
curl -X POST https:///api/prism/v1/apikeys \
-u admin:admin \
-H "Content-Type: application/json" \
-d '{
"keyName": "frontend-ingest",
"roles": ["ingestor-frontend"]
}'
```
Response — the **full key value is returned only on this call**:
```json
{
"keyId": "01ARZ3NDEKTSV4RRFFQ69G5FAV",
"keyName": "frontend-ingest",
"apiKey": "9b2f8c4e-1d6a-4f2b-8e7c-3a5d9c1b7e4f",
"roles": ["ingestor-frontend"],
"createdBy": "admin",
"createdAt": "2026-04-30T10:30:00Z",
"modifiedAt": "2026-04-30T10:30:00Z"
}
```
Copy the `apiKey` value somewhere safe before you leave the response. Subsequent list calls return a masked version, and there is no way to recover the original after it leaves your screen.
### List API keys
```sh
curl https:///api/prism/v1/apikeys -u admin:admin
```
Each entry returns the metadata, but `apiKey` is masked to its last four characters:
```json
[
{
"keyId": "01ARZ3NDEKTSV4RRFFQ69G5FAV",
"keyName": "frontend-ingest",
"apiKey": "****7e4f",
"roles": ["ingestor-frontend"],
"createdBy": "admin",
"createdAt": "2026-04-30T10:30:00Z",
"modifiedAt": "2026-04-30T10:30:00Z"
}
]
```
API-key identities are not surfaced in the regular `GET /api/v1/users` listing — they are managed exclusively under `/api/v1/apikeys`.
### Get an API key by id
Fetching a single key by its `keyId` returns the **unmasked** value, so an admin can recover a key without re-creating it.
```sh
curl https:///api/prism/v1/apikeys/01ARZ3NDEKTSV4RRFFQ69G5FAV \
-u admin:admin
```
```json
{
"keyId": "01ARZ3NDEKTSV4RRFFQ69G5FAV",
"keyName": "frontend-ingest",
"apiKey": "9b2f8c4e-1d6a-4f2b-8e7c-3a5d9c1b7e4f",
"roles": ["ingestor-frontend"],
"createdBy": "admin",
"createdAt": "2026-04-30T10:30:00Z",
"modifiedAt": "2026-04-30T10:30:00Z"
}
```
### Delete an API key
Deletion is the only revocation mechanism. Once deleted, every request using that key value returns `401 Unauthorized`.
```sh
curl -X DELETE \
https:///api/prism/v1/apikeys/01ARZ3NDEKTSV4RRFFQ69G5FAV \
-u admin:admin
```
### Use an API key
Send the secret value in the `X-API-Key` header. This replaces the `Authorization: Basic ...` header — do not send both.
### Ingest logs
```sh
curl -X POST https:///api/v1/ingest \
-H "X-API-Key: 9b2f8c4e-1d6a-4f2b-8e7c-3a5d9c1b7e4f" \
-H "X-P-Stream: frontend" \
-H "Content-Type: application/json" \
-d '[{"level":"info","message":"hello"}]'
```
### Query
```sh
curl -X POST https:///api/v1/query \
-H "X-API-Key: 9b2f8c4e-1d6a-4f2b-8e7c-3a5d9c1b7e4f" \
-H "Content-Type: application/json" \
-d '{
"query": "SELECT * FROM frontend",
"startTime": "1h",
"endTime": "now"
}'
```
In multi-tenant deployments, also send the tenant header:
```sh
-H "X-P-Tenant: "
```
### Scoping access via roles
Because API keys reuse the role system, you can pick any role supported by Parseable RBAC. The role JSON is identical to what you would assign to a Native user — see [RBAC](/user-guide/rbac) for the full schema.
#### Global ingestion key
Use a single key for forwarders that send logs to every dataset.
```json
[
{ "privilege": "ingestor" }
]
```
#### Dataset-scoped ingestion key
Restrict the key to one or more named datasets. Useful when each environment or team has its own forwarder.
```json
[
{ "privilege": "ingestor", "resource": { "dataset": "frontend" } },
{ "privilege": "ingestor", "resource": { "dataset": "checkout" } }
]
```
#### Global reader for query agents
Read-only access across every dataset — appropriate for analytics scripts or AI agents that need broad query coverage.
```json
[
{ "privilege": "reader" }
]
```
#### Dataset-scoped reader
Limit a query agent to a single dataset (and optionally to events matching a tag).
```json
[
{
"privilege": "reader",
"resource": { "dataset": "frontend" }
}
]
```
#### Admin key
Full control over the tenant — create datasets, dashboards, filters, alerts, retention policies, manage users and roles. Use sparingly and only for trusted automation.
```json
[
{ "privilege": "admin" }
]
```
#### Editor / Writer key
Authoring access to dashboards, filters, and alerts on selected datasets — useful for CI pipelines that provision dashboards as code.
```json
[
{ "privilege": "writer", "resource": { "dataset": "frontend" } }
]
```
### Best practices
* **One key per workload.** Issue separate keys for each forwarder, agent, or pipeline so you can rotate or revoke without disrupting others.
* **Scope tightly.** Prefer dataset-scoped roles over global ones. An ingestion key should not be able to query, and a reader key should not be able to write.
* **Rotate periodically.** Create a new key, switch the workload over, then delete the old one.
* **Store keys in a secrets manager.** Inject them into the workload at runtime — do not bake them into images or commit them to repositories.
* **Audit regularly.** List keys and remove any whose owner or purpose is unclear.
# APM (/user-guide/apm)
APM in Parseable gives you a service-level view of your traced applications. It is where you start when you know something is happening in the application, but you still need to understand which service is involved and which trace explains the behavior.
The page is built on trace data. You choose a trace dataset, inspect the services found in that dataset, open a service, and then move between `Insights`, `Traces`, and `Errors`. The idea is simple: start from the service, understand the shape of traffic and failures, then open the trace that gives you the exact request path.
## APM with Parseable
The APM page is most helpful when you want to compare services by request volume, latency, and errors before opening individual traces. For example, you may want to know which service is receiving the most traffic, which one has the highest p95 duration, or whether errors are isolated to one service or visible across a larger request path.
APM is different from the raw Traces page because it starts from services. The Traces page is better when you already want to inspect trace records directly. APM is better when you want a higher-level service view first and then drill down.
## Typical workflow
### Dataset and time range
Start from the `dataset selector` at the top of the page. Pick the trace dataset you want to inspect. The services list, filters, and service-level details all update around that dataset.
The `time range` picker controls how much data is included. A shorter range works well during an active incident or rollout. A wider range is better when you want to compare service behavior across days.
You can also choose the timezone from the picker, including UTC or your local timezone, so the charts and trace timestamps match how your team is investigating the issue.
The top toolbar also includes `refresh` and `auto-refresh`. `Auto-refresh` is useful when you are watching a live system and want the service view to keep updating.
APM reads from trace datasets. Make sure your application is sending OpenTelemetry traces into Parseable before using this page.
### Services list
The default APM view shows the services found in the selected trace dataset. Each row represents a service and gives you the basic health signals for that service.
The table shows the service name, request count, total duration, p95 duration, error count, and error rate. This makes it easy to scan the application from a service point of view instead of starting with thousands of individual spans.
The `search` bar helps when you already know the service name. Column sorting helps when you want to bring the busiest, slowest, or highest-error services to the top.
If your traces contain extra service metadata, `Add columns` lets you bring those fields into the table. This helps when service name alone is not enough and you want to include fields such as namespace, version, environment, or deployment metadata.
### Service list
The service list is usually the fastest way to compare services by request count, p95 duration, or errors. It uses the selected dataset and `time range`, so changing either one updates the service table around the same investigation context.
### Open a service
Click a service row to open the service-level APM view. This scopes the page to that service and gives you three tabs. `Insights` shows request, duration, error, and resource charts. `Traces` shows traces related to the selected service. `Errors` focuses on traces and spans where failures were reported.
The `service selector` at the top lets you move between services without returning to the main list. This is useful when you are comparing two services during the same investigation.
## Insights tab
The `Insights` tab is the first place to look after opening a service. It summarizes the selected service over the current `time range`.
At the top, you get service-level numbers such as request count, duration, and error rate. Below that, the charts show how the service behaved over time.
This is where you can quickly tell whether traffic changed, whether errors appeared as one-off spikes or repeated patterns, and whether latency moved at the same time as request volume. The resource charts and table help you see which routes, operations, or resources are contributing the most requests, duration, or errors.
The resources table helps you move from service-level behavior into endpoint or operation-level behavior. When you already know the route or operation name, search within resources. When the default columns are not enough, add the fields you need to the table.
For example, if a service has an error spike, use the resource charts and table to check whether the errors are coming from one route, one operation, or the service more broadly.
## Traces tab
The `Traces` tab shows traces related to the selected service. It becomes useful once `Insights` tells you that something changed and you want to open the actual traces behind that change.
This tab is useful when you want to find slow traces for a service, traces with many spans, traces around a specific time window, or traces related to a resource you noticed in `Insights`.
Open a trace row to move into trace detail. The trace detail view shows the trace start time, total duration, span count, and waterfall.
The waterfall shows how the request moved across services and where time was spent. Expand span groups when you need more detail. Click a span when you want to inspect fields, events, service metadata, HTTP attributes, database attributes, or other span-level data.
This is usually where APM becomes a debugging workflow. You start from the service, narrow down to the trace, and then inspect the spans that explain the behavior.
## Errors tab
The `Errors` tab focuses the service view on failed traces and error spans. It is the tab to open when the service list or `Insights` tab shows an error count or error-rate spike.
This is the place to check which traces failed for the service, which operation or resource is involved, and whether the errors are spread across many traces or concentrated in a few repeated paths. From there, open the trace and inspect the span that reported the error.
If the same error needs to be watched going forward, you can also create an alert directly from the error detail view.
From here, open a trace and inspect the waterfall the same way you would from the `Traces` tab. The difference is that the `Errors` tab starts from failures, so you spend less time filtering the broader trace list.
## Working through an issue
A usual investigation starts by selecting the trace dataset and `time range`, then looking at the service list to see which service needs attention. From there, open the service, check `Insights` for request volume, latency, and error patterns, and then move into `Traces` or `Errors` when you need the request-level detail.
Once you open a trace, the waterfall gives you the full path of the request. Click into spans when you need the exact fields, events, and service metadata behind the behavior. APM works best when your traces include consistent service names, span names, status fields, and resource attributes, because those fields make the service list easier to scan and the drilldown easier to trust.
# Dashboards (/user-guide/dashboards)
Parseable Dashboards are customizable and support querying multiple data streams for comprehensive insights. You can also leverage text to SQL conversion for quick query creation.
## Creating a Dashboard
1. **Start from the Side Nav:** Head over to Dashboards in the sidebar and hit Create New
2. **Name & Tag:** Give your dashboard a name and a tag (for easy organizing), then click Create Dashboard
3. **Build Your Canvas:** Select your new dashboard to land on the dashboard canvas. Click Add Tiles to start building charts
4. **Save and Favorite:** Add as many tiles as you need, then save your dashboard. You can also mark dashboards as favorites for quick access later
## Adding Charts to Dashboards
Charts can be added to dashboards from multiple sources:
* **SQL Editor**: Run queries in the SQL editor, visualize results, and add charts directly to dashboards
* **Keystone**: AI-generated insights can be added as dashboard tiles
* **Chart Builder**: Build visualizations using the built-in chart builder
## Chart Builder Types
There are two ways to build charts:
### SQL Editor
Write custom SQL queries, visualize the results, and add them to your dashboard. The SQL editor supports up to 3 datasets simultaneously for complex joins and comparative analysis.
After running a query, you can use the chart view customization options and add these charts to dashboards directly.
### Visual Chart Builder
The chart builder follows a 3-step process to create powerful visualizations:
### Step 1: Initialize
**Chart Type Selection:**
* **Timeseries**: For time-based data visualization
* **Line**: For trend analysis and continuous data
* **Bar**: For categorical comparisons
* **Area**: For cumulative values over time
* **Pie**: For showing proportions of a whole
* **Donut**: Similar to pie with a hollow center
* **Query Value**: For displaying single metric values
* **Gauge**: For showing progress or thresholds
**Dataset Selection:**
Choose your data source from the dropdown menu (e.g., vLLMmetrics or your custom log streams).
### Step 2: Plot Configuration
The configuration options vary based on your selected chart type:
#### For Timeseries, Line, Bar, and Area Charts:
* **Y-Axis**: Select the plot field (e.g., `aggregation_temporality_desc`)
* **Aggregate**: Choose aggregation method (COUNT)
* **Filters**: Add conditions to filter your data
* **Group by**: Group data by specific fields for segmentation
* **Sort by**: Order your results (ascending or descending)
* **X-Axis**: Select the dimension field (e.g., `aggregation_temporality`)
#### For Pie and Donut Charts:
* **Plot**: Select the field to visualize
* **Aggregate**: Choose how to aggregate values (COUNT)
* **Filters**: Apply data filters as needed
* **Note**: Grouping and sorting options are not available for these chart types
#### For Query Value and Gauge:
* **Plot**: Select the metric field
* **Aggregate**: Define the aggregation method
* **Filters**: Filter data to focus on specific values
* **Note**: These visualizations display single values, no grouping or sorting needed
### Step 3: Style Your Visualization
**Customization Options:**
* **X-axis Label**: Custom label for the X-axis
* **Y-axis Label**: Custom label for the Y-axis
* **Colour Scheme**: Choose from predefined color palettes (Forest, Ocean, Sunset, etc.)
* **Legend Position**: Place legend where it best fits your layout
* **Grid Lines**: Toggle grid visibility for better readability
* **Data Labels**: Show/hide values on the chart
**Chart-Specific Styling:**
* **Line/Area**: Smooth curves, fill opacity, stroke width
* **Bar**: Orientation (vertical/horizontal), bar spacing
* **Pie/Donut**: Inner radius (for donut), label display
**Gauge and Query Value Specific Styling:**
* **Units**: Add unit labels (e.g., %, ms, GB)
* **Auto-formatting**: Enable automatic number formatting
* **Precision**: Set decimal places for displayed values
* **Range**: Define min and max values (from 0 to 100 for percentages)
* **Segments**: Configure color-coded thresholds
* Set breakpoints (e.g., 10, 90 for three segments)
* Assign colors to each segment (green, yellow, red)
* Use visual indicators for performance levels
**Quick Table View:**
Want to see raw data? Click the **Table** button in the top right corner to skip chart setup and view your query results in a tabular format.
## How it works
Dashboards in Parseable are a collection of tiles, each representing a visualization of a query result. You can create multiple tiles on a dashboard, each with its own query and chart type. Further, a tile's chart can be configured for different colors, units and formatting. Each tile can be based on a query targeting different dataset. The tiles can be resized, repositioned, and exported in various formats for easy sharing and collaboration.
Dashboard tiles can be dragged and repositioned anywhere on the dashboard. You can also adjust the tile size to fit your layout preferences, selecting from small, medium, large, or full screen widths (1/4, 2/4, 3/4, and full screen).
## Visualization Types
Parseable offers eight distinct visualization types, each optimized for different data patterns:
### Time-Series Charts
* **Timeseries**: Specialized for temporal data with automatic time formatting
* **Line**: Connect data points for trend visualization
* **Area**: Show cumulative values with filled areas
These charts support full configuration: Y-axis plotting, filtering, grouping, and sorting.
### Comparison Charts
* **Bar**: Compare categories with vertical or horizontal bars
* Supports: Y-axis plot, filters, grouping, sorting
* Best for: Categorical comparisons, distributions
### Proportion Charts
* **Pie**: Display parts of a whole in circular format
* **Donut**: Similar to pie with hollow center for modern aesthetics
* Supports: Plot selection, aggregation, filtering
* Best for: Showing percentages and proportions
### Metric Displays
* **Query Value**: Single metric display for KPIs
* **Gauge**: Visual progress indicator with thresholds
* Supports: Plot selection, aggregation, filtering
* Best for: Performance indicators, threshold monitoring
### Data Table
* **Simple Table**: Raw data view with sorting and filtering
* Access via the Table button in any chart builder
* Export capabilities for further analysis
Each visualization includes customizable styling options for colors, labels, and formatting to match your dashboard aesthetic.
## Time Range
Set a fixed time range for your dashboard to ensure all tiles load data consistently. You can save this time range with the dashboard, allowing for synchronized data views across all visualizations.
# Errors (/user-guide/errors)
Parseable Errors page is purpose built to help you debug errors in OpenTelemetry traces. It surfaces the most important errors in your stack, gives you a high-level understanding of what is breaking, and helps you jump into the right trace to see the full context of a failure.
The error page collapses thousands of exception spans into a short list of distinct failures. This saves time and effort in scrolling through the Traces explorer with a filter on error status.
## Data pre-requisites
* Error view works with **OTel Trace** datasets only. Error groups are built from the `event_exception.type`, `event_exception.message`, and `event_exception.stacktrace` fields that OpenTelemetry records on span events.
* Error view requires events with `event_exception.type`, `event_exception.message`, and `service.name` fields.
If you are chasing log-level errors instead, start from the [Logs explorer](/user-guide/logs) with a filter on your log level field. Log based error grouping is not available here yet.
## When to use this page
* You were paged or noticed a dashboard spike and need to find what is actually failing
* You want to know which service is the source of a wave of errors rippling across your stack
* You need to tell a new error apart from a long-running one: "has this been failing for hours, or did it start five minutes ago?"
* You want an LLM-generated first guess at the root cause and a remediation checklist before diving into stack traces
If you already have a trace ID, skip this page and jump straight to the [Traces explorer](/user-guide/traces). If you know the log line you are after, use the [Logs explorer](/user-guide/logs).
## The debugging workflow
The page is designed around a single flow: **scope, triage, open, correlate, escalate**. Each section below is one step of that flow. Read them in order the first time and skip around later.
### Scope the investigation
Pick the trace dataset of the service you suspect, then narrow the time range to the incident window. Keep the window tight: a one-hour range over a fresh deploy almost always carries more signal than "last 24 hours" because the spark charts and the AI summary both work better when the list is not dominated by background noise. Widen it only after you have ruled out a recent regression.
### Find the error that matters
Read the error list the way you would read a ranked incident list rather than a UI inventory:
* **Scan the Occurrences column for the loudest error.** High-count rows are the ones most likely to match whatever your alert fired on.
* **Use the spark chart to read the shape of the failure.** A steady bar pattern is a chronic issue that has been running for a while. A single tall block is a burst tied to a specific moment. A "started, stopped, started again" pattern usually points at a flapping dependency.
* **Narrow with the sidebar facets when you have a hunch.** Click a service under **Services** to see only that service's errors. Click an exception class under **Error types** (for example `14 UNAVAILABLE` or `Grpc.Core.RpcException`) when you want to look at one failure mode across services.
* **Use Search errors for a known message.** If a teammate told you "the cart service is throwing `Wasn't able to connect to redis`", paste that phrase and jump directly to the group.
### Understand what broke
Click the error row to open its detail view.
Start with the header. The **first seen / last seen** pair and the **error frequency** chart answer two triage questions you need before doing anything else:
* **Is this new?** A first-seen timestamp within the last few minutes almost always means a recent deploy, config change, or traffic shift. Check the deploy log before anything else.
* **Is this getting worse?** A frequency chart that is climbing rather than flat means the situation is escalating. Do not wait on a long investigation before paging upstream teams.
Then expand the **AI Summary**.
The summary takes the exception type, message, and stack trace as context and returns three things: a plain-language **root cause**, an **explanation** of how the error manifests and which components are involved, and a numbered **suggested fixes** checklist. Treat it as a knowledgeable first responder. It is often enough to unblock a known-class error (dependency down, misconfiguration, misrouted traffic) without reading a single stack frame.
If the summary nails it, follow the suggested fixes and stop here. If it feels generic or wrong, move on.
### Open a real failing request
Below the header, the **impacted spans** table lists every span that carried this exception. Pick a representative one, usually the most recent, but prefer a long-duration row when you are chasing a latency-plus-error problem.
Clicking a span opens the error trace view.
This view exists to answer the "what was the request doing when it failed" question using two pieces of evidence:
* **The stack trace** shows exactly where in your code the exception was raised. Copy it into your issue tracker or a chat thread with one click.
* **The events timeline** lists every span event in the trace in chronological order, with errors highlighted in red. Read it to see what else was happening in the same request: retries, upstream calls, and the normal span events that surrounded the failure.
Click **View trace** in the header to jump to the full waterfall in the [Traces explorer](/user-guide/traces) with the failing span highlighted. Do that when you need the end-to-end request tree, for example to verify that a timeout in an upstream service is the real cause rather than the local service itself.
### Correlate with logs
Errors on this page are linked to logs via OpenTelemetry trace and span IDs. Two pivots matter:
* **Trace to logs**: from an error trace, jump to every log line that shares the same `trace.id` to see what the services in the request were logging around the failure
* **Span to logs**: from a specific error span, jump to log lines that share the same `span.id` for the narrow slice of logs tied to that one operation
Reach for the trace-level pivot when you want the full story of a request and the span-level pivot when you already know which operation broke and want the logs that service emitted right around it. Both only work when your logs and traces are ingested with matching IDs. See the [instrumentation guide](/cookbook/instrumentation) for the required attributes if you are instrumenting by hand.
### Escalate to Keystone when the surface info is not enough
Click **Analyze** on the error detail or trace view to hand the full error context to [Keystone](/user-guide/ai-native/keystone), Parseable's agentic debugging assistant. Keystone receives the exception type, message, stack trace, the full span tree for the trace, and service and operation metadata, and gives you a natural-language debugging session on top of that data.
Use Keystone when:
* The failure touches multiple services and you want to know which one is the actual source
* The AI Summary is correct but not specific enough, and you want to ask follow-ups like "are other services seeing the same upstream error right now?"
* You want a probable root cause backed by SQL queries you can run directly against your data
* You need mitigation steps that take the rest of your telemetry into account, not just the stack trace of one span
If all you need is a quick explanation of the exception in front of you, the AI Summary is faster. Reach for Keystone when you need the broader picture.
### Monitor if it happens again
Before closing the detail view, use **Create alert** in the header to create an [alert](/user-guide/alerting) scoped to this error group. This is especially valuable for silent failures (ones you only found out about because a customer complained), so the next occurrence pages an on-call engineer instead of quietly piling up.
## Debugging tips
* **Compare first seen and last seen before writing an RCA.** The same `14 UNAVAILABLE` that has been quietly failing all week is a very different story from one that started ten minutes ago.
* **Trust the spark chart to judge urgency.** A tall, sharp block means "something happened now." A steady bar pattern means "this has been broken and nobody noticed."
* **Do not stop at the AI Summary for novel errors.** It is strong on known-class failures and weaker on one-off bugs. When it feels generic, drop into a real trace.
* **Filter before you paginate.** On a noisy dataset, narrowing by service or exception type is much faster than scrolling through the full list.
* **Close the loop with an alert.** Every error you triage from a customer report is an alert you should have had. The **Create alert** button in the header is one click away.
# Log IQ (/user-guide/log-iq)
Log IQ allows identifying the format of unstructured log data, transforming it into structured columns within ingested events in JSON format. This helps in easy and optimized query, search, debug and visualize the data.
## How Log IQ works
Log IQ requires specific HTTP headers when ingesting data to properly identify and parse log formats:
### Required Headers
* `X-P-Log-Source` - **Mandatory** - Identifies the log format name (e.g., `syslog`, `nginx_access`, `zookeeper`)
* `X-P-Extract-Log` - **Required for unstructured data** - Specifies which field in the incoming JSON contains the raw log text (typically `log`)
### Processing Logic
**For structured data:**
* Only `X-P-Log-Source` is required
* Parseable assumes the data is already in a structured format
* The specified format is used for validation and additional processing
**For unstructured data:**
* Both `X-P-Log-Source` and `X-P-Extract-Log` are required
* Parseable extracts the raw log text from the field specified in `X-P-Extract-Log`
* The system applies regex patterns based on the format specified in `X-P-Log-Source`
* If the content matches the format, it's parsed into structured fields
* If the content doesn't match the format, the original value is retained in the specified field
### Outcome
* After successful format detection, a `p_format` field is added to the log event containing the log source name
* The dataset info is updated with an array of detected log sources
* Parseable UI (Prism) automatically displays filters on the `p_format` field
* If the log format is not detected, `p_format_verified=false` is added to the event
* Data is always ingested, regardless of format detection success
> **Note:** Even if your unstructured data doesn't match any of the supported formats listed below, you must still specify both headers. Choose the format that most closely aligns with your log structure.
### Example: Processing a Syslog Entry
Let's walk through a practical example of how Log IQ processes a syslog entry:
**1. Original log sent by an agent (e.g., FluentBit):**
```json
{
"log": "2025-07-11T14:57:33.000111+05:30 node01 exporter[9012]: [2025/07/11 14:57:33] [error] [output:http:http.8] Failed to push metrics to endpoint /metrics"
}
```
**2. HTTP headers used when sending to Parseable:**
```
X-P-Log-Source: syslog_log
X-P-Extract-Log: log
```
**3. Parseable's processed output:**
```json
{
"body": "[2025/07/11 14:57:33] [error] [output:http:http.8] Failed to push metrics to endpoint /metrics",
"log": "2025-07-11T14:57:33.000111+05:30 node01 exporter[9012]: [2025/07/11 14:57:33] [error] [output:http:http.8] Failed to push metrics to endpoint /metrics",
"log_hostname": "node01",
"log_pid": "9012",
"log_procname": "exporter",
"log_syslog_tag": "exporter[9012]",
"p_format": "syslog_log",
"p_format_verified": "true",
"p_src_ip": "127.0.0.1",
"p_timestamp": "2025-07-11T09:20:23.019",
"p_user_agent": "PostmanRuntime/7.44.1",
"timestamp": "2025-07-11T09:27:33"
}
```
In this example:
1. The agent (like FluentBit) collects the log and places it in the `log` field
2. Parseable receives this with the appropriate headers
3. The system identifies it as a syslog format and extracts structured fields:
* `log_hostname`: The host that generated the log ("node01")
* `log_pid`: The process ID ("9012")
* `log_procname`: The process name ("exporter")
* `log_syslog_tag`: The syslog tag ("exporter\[9012]")
* `body`: The actual message content
4. Parseable adds its metadata fields:
* `p_format`: The detected format ("syslog\_log")
* `p_format_verified`: Confirmation that the format was successfully detected
* Other `p_` prefixed fields with request metadata
This structured data is now ready for efficient querying and analysis.
## Supported Formats
Parseable Log IQ supports a wide range of log formats. You can specify these formats using the `X-P-Log-Source` header when ingesting logs. The currently supported formats include:
| Format | Description |
| ----------------- | --------------------------------------------------- |
| `access_log` | Common web server access logs (Apache, Nginx, etc.) |
| `alb_log` | AWS Application Load Balancer logs |
| `block_log` | Generic block-style logs |
| `candlepin_log` | Candlepin service logs |
| `choose_repo_log` | Repository selection logs |
| `cloudvm_ram_log` | Cloud VM RAM usage logs |
| `cups_log` | Common UNIX Printing System logs |
| `dpkg_log` | Debian package manager logs |
| `elb_log` | AWS Elastic Load Balancer logs |
| `engine_log` | Generic engine logs |
| `env_logger_log` | Environment logger format |
| `error_log` | Common error log format |
| `esx_syslog_log` | VMware ESX syslog format |
| `haproxy_log` | HAProxy load balancer logs |
| `katello_log` | Katello service logs |
| `lnav_debug_log` | LNAV debug logs |
| `nextflow_log` | Nextflow workflow logs |
| `openam_log` | OpenAM authentication logs |
| `openamdb_log` | OpenAM database logs |
| `openstack_log` | OpenStack service logs |
| `page_log` | Printer page logs |
| `procstate_log` | Process state logs |
| `proxifier_log` | Proxifier logs |
| `rails_log` | Ruby on Rails application logs |
| `redis_log` | Redis database logs |
| `s3_log` | AWS S3 access logs |
| `simple_rs_log` | Simple Rust logs |
| `snaplogic_log` | SnapLogic integration logs |
| `sssd_log` | System Security Services Daemon logs |
| `strace_log` | System call trace logs |
| `sudo_log` | Sudo command logs |
| `syslog_log` | Standard system logs |
| `tcf_log` | Target Communication Framework logs |
| `tcsh_history` | TCSH shell history |
| `uwsgi_log` | uWSGI server logs |
| `vmk_log` | VMware kernel logs |
| `vmw_log` | VMware general logs |
| `vmw_py_log` | VMware Python logs |
| `vmw_vc_svc_log` | VMware vCenter service logs |
| `vpostgres_log` | VMware Postgres database logs |
| `web_robot_log` | Web crawler/robot logs |
| `xmlrpc_log` | XML-RPC logs |
Each format has specific patterns and fields that are extracted. When a log matches one of these formats, Parseable automatically extracts the structured fields and makes them available for querying and analysis.
### Extracted Fields by Format
Below are the fields extracted for each supported log format:
access_log
\- Web server access logs
* `timestamp` - Time when the request was received
* `c_ip` - Client IP address
* `cs_username` - Username if authentication was used
* `cs_method` - HTTP method (GET, POST, etc.)
* `cs_uri_stem` - Requested URI path
* `cs_uri_query` - Query string parameters
* `cs_version` - HTTP protocol version
* `sc_status` - HTTP status code
* `sc_bytes` - Response size in bytes
* `cs_referer` - Referer URL
* `cs_user_agent` - User agent string
* `cs_host` - Host header value
* `body` - Any additional content
alb_log
\- AWS Application Load Balancer logs
* `type` - Connection type (HTTP, HTTPS, etc.)
* `timestamp` - Request timestamp
* `elb` - Load balancer name
* `client_ip` - Client IP address
* `client_port` - Client port
* `target_ip` - Target IP address
* `target_port` - Target port
* `request_processing_time` - Time from connection to routing decision
* `target_processing_time` - Time from request to response from target
* `response_processing_time` - Time from response from target to client
* `elb_status_code` - Response code from load balancer
* `target_status_code` - Response code from target
* `received_bytes` - Bytes received from client
* `sent_bytes` - Bytes sent to client
* `cs_method` - HTTP method
* `cs_uri_whole` - Request URL
* `cs_version` - HTTP version
* `user_agent` - User agent string
* `ssl_cipher` - SSL cipher
* `ssl_protocol` - SSL/TLS protocol
syslog_log
\- Standard system logs
* `timestamp` - Log timestamp
* `log_hostname` - Host name
* `log_syslog_tag` - Syslog tag
* `log_procname` - Process name
* `log_pid` - Process ID
* `body` - Log message content
* `log_pri` - Priority value
* `syslog_version` - Syslog version
* `log_msgid` - Message ID
* `log_struct` - Structured data
redis_log
\- Redis database logs
* `pid` - Process ID
* `timestamp` - Log timestamp
* `level` - Log level
* `role` - Redis role (master, slave, etc.)
* `body` - Log message content
This is not an exhaustive list of all fields for all formats. Each format has specific patterns and may extract additional fields based on the log content. When using Log IQ, you can explore the extracted fields in the Parseable UI or through SQL queries.
In case of p\_format\_verified = false, for a known format listed above, raise a Git issue to add the format.
# Logs (/user-guide/logs)
Logs are often the first place you go when something in production needs context. A chart can tell you that traffic changed. An alert can tell you something failed. Logs help you see what the service actually wrote at that exact moment.
The Logs page in Parseable is built for that flow. You choose a dataset, pick the time range, look at log volume, narrow the data with fields and filters, and then inspect the records that explain what happened.
## Logs with Parseable
Parseable stores logs as structured records. That means the Logs page is not only a place to search text. You can use fields like service name, host name, container metadata, HTTP details, trace ids, span ids, and log category to move from a broad view to a focused set of rows.
Use this page when you want to answer questions like:
* Which service produced these logs
* When did the log volume change
* Are the records mostly `INFO`, `WARN`, `ERROR`, or `UNSPECIFIED`
* Which host, container, endpoint, or trace is involved
* What fields should be added to the table so the data is easier to scan
## Typical workflow
### Dataset and time range
Start with the dataset selector and the time range picker.
Select the log dataset from the dataset selector at the top of the page. The rest of the page updates for that dataset, including the available fields, filters, chart, and table columns.
Use the time range picker to decide how much data you want to inspect. A smaller time range works well when you already know when something happened. A wider range is useful when you want to see how log volume changed over time.
You can also choose the timezone from the picker, including your local timezone, so timestamps are easier to read for your team.
The top toolbar also gives you quick access to refresh, auto-refresh, saved views, and summarization. Use auto-refresh when you want to keep watching a live window of logs.
Every logs query is scoped by time. This keeps the query focused on the period you want to inspect and avoids reading more data than needed.
### Logs over time
After choosing the dataset and time range, look at the Logs over time chart. It shows how many log records were found in the selected window.
The chart is split by log category. Categories such as `UNSPECIFIED`, `INFO`, `WARN`, and `ERROR` appear in the legend. Click a category when you want to focus on that type of log. The chart and table update around that selection.
Turn on **Forecast** when you want Parseable to project the current log volume forward. This is useful when you are watching traffic during a rollout, load test, or spike.
### Field groups
The fields panel on the left shows the fields Parseable found in the selected dataset. Use it before adding filters or columns. It helps you understand what data is available.
Fields are grouped into sections such as:
* **Core**, for common log fields and log category
* **Error**, for error-related fields
* **Service**, for service name, service instance, namespace, and version
* **Span**, for trace and span context
* **Host**, **Container**, **HTTP**, **Process**, **Telemetry**, and other runtime fields
* **All fields**, when you want to search across everything
Use **Search field names** when you already know the field you need. Expand a field group to see values and counts. The counts help you decide which values are worth filtering on.
### Filters
When the table has too many records, use **Add filter** to narrow the page.
The filter panel lets you search across fields and values. Values are shown with counts, so you can choose the value that matches what you want to inspect.
For example:
* Filter by `service.name` to inspect one service
* Filter by `host.name` when a problem looks node-specific
* Filter by `p_log_category` to look at `INFO`, `WARN`, or `ERROR` logs
* Filter by HTTP fields when you are following a request path
* Filter by trace or span fields when you want to connect logs back to traces
Active filters appear as pills above the chart. You can remove one filter at a time, or use **Clear all** to return to the broader view.
After filters are applied, use **Find in data** to search within the visible rows. This is useful when you want to find a word, id, product name, endpoint, or message fragment inside the current page.
### Group by
Use **Group by** when you want to compare records by a field instead of reading individual rows.
Good fields to group by include:
* `p_log_category`, to compare `INFO`, `WARN`, and `ERROR` volume
* `service.name`, to see which service is producing the most logs
* `host.name` or container fields, to find noisy infrastructure
* `http.status_code`, to separate successful and failing requests
* Error fields, to see repeated failure patterns
Grouping is useful when the table is too large to read directly. It helps you see which services, hosts, categories, or status codes are contributing the most records.
### Log table
The table shows the actual log records. By default, you see ingestion time and log data. If the dataset has many fields, add the columns that are useful for the current task.
Use **Add columns** to choose fields that should appear as table columns. For example, you might add `service.name`, `host.name`, `http.status_code`, `trace_id`, or a field from the log body.
The table toolbar also gives you a few useful controls:
* **Expand rows**, to show more of each record without opening records one by one
* **Share**, to copy a link to the current view
* **Find in data**, to search within the current rows
* **Pagination and page size**, to move through large result sets without changing the query
If a record contains a long message or many fields, use the row actions to inspect more detail. Structured logs make this easier because each field can be read, filtered, or added as a column.
### Log context
Sometimes one log row is not enough. An error record usually makes more sense when you can see what the service wrote just before it and what happened immediately after it.
Open the row actions menu at the end of a log row and choose **Show context**.
Parseable opens the **Log context** view around that row. The row you started from becomes the referenced line. By default, the view starts with one minute before and one minute after the referenced line, so you can inspect the short window around the event without building a new query from scratch.
Use the timeline to widen or narrow the context window when you need more history around the event. For example, you can move from a small window around the error to a wider window when the cause may have started earlier.
If you scroll away from the original row, use **Referenced line** to jump back to the log line that opened the context. Use search when you want to find a value inside the surrounding records, and use wrap when long messages need to be easier to read.
You can also apply filters inside Log context. This helps when the surrounding window has many records and you only want to keep a service, log category, source IP, user agent, or another field that matters for the investigation.
### Saved views, sharing, and summaries
When you have a useful view, save it. Saved views help you come back to the same dataset, filters, fields, and time range without setting everything again.
Use the share control when you want another teammate to open the same view. Use export when you need the current result set outside Parseable.
The **Summarize** button can generate a report for the selected data. Use it when you want a quick read of the current logs before looking deeper.
The summary can also suggest drilldown queries. Use those as starting points. Open a query in a new tab when you want to keep the current Logs page open and continue in SQL.
## A simple way to use the page
Most log investigations follow the same loop:
1. Pick the dataset and time range.
2. Look at the chart to understand volume and severity.
3. Use fields or **Add filter** to narrow the data.
4. Add columns that make the table easier to scan.
5. Open or expand the records that look relevant.
6. Save, share, export, summarize, or move into SQL when needed.
With Parseable logs, the goal is to narrow the data until the records you need are easy to inspect.
# Metrics (/user-guide/metrics)
The Metrics explorer is the main interface where you browse, analyze, and take action on your metric data. It provides a clean, visual grid overview of all metrics in a dataset, along with powerful tools to filter data, change time windows, get automated summaries, and more.
## Page layout
The Metrics Explorer page is organized into three tabs.
The "All Metrics" tab is the default view, which displays all metrics in the selected dataset as a grid of individual metric cards. This layout makes it easy to monitor items like CPU, memory, and disk utilization right away as you land here. This tab has a toggle on the right side with options to view the metrics as "Chart" or "List". List view shows the same metrics in a tabular form, allowing drill down into a specific metric and its associated labels.
The "Explore" tab is the second tab. It is an ad-hoc query workspace for isolating specific metrics and comparing performance curves. It is designed for isolating specific metrics, running, and comparing performance curves against static baselines.
The "Table" tab switches to a raw data metrics spreadsheet view. It displays incoming data points in horizontal rows showing exact timestamps, ingestion values, and structured metadata attributes.
Additionally, the page has a top control bar with search and filtering inputs, layout scaling toggles, and a time range picker. You can also
* **Save view:** Saves your active configuration settings, filters, and zoom levels into a persistent workspace profile.
* **Summarize button:** A sparkle icon that prints a plain text summary explaining telemetry anomalies and potential root causes.
## Typical workflow
### Start by selecting a dataset
Datasets are the [primary data organization unit](https://www.parseable.com/docs/key-concepts/data-model) in Parseable. Select the dataset you want to explore from the dropdown menu at the very top-left of the page. Changing this option immediately updates the entire page to reflect the telemetry of the new dataset.
### Search and filter
Below the tab selection area, the main search input and `Add filter` button enable you to narrow down data fields by specific label-value attributes. For example, you could filter to a specific `host` or `region` to isolate a single system component across the entire visual dashboard. The search input supports free text search across all label values, while the `Add filter` button provides a structured interface for building complex queries with multiple label conditions.
### Working with metric cards
Each individual card in the main grid acts as a standalone monitoring panel for a specific metric key. The card includes a few important indicators:
* **Metric key name:** Located at the top-left corner of the card (for example, `process.cpu.time`, `process.memory.usage`, or `process.disk.io`).
* **Card context menu:** This three-dot icon sits at the top-right corner of each card, providing a shortcut to launch a detailed breakdown for that specific metric exploration.
Selecting a metric card directly will trigger a comprehensive breakdown view, which also includes quick-access options to analyze additional related telemetry signals.
### Exploring data with PromQL
Additionally, Parseable enterprise includes a [built-in PromQL engine](https://www.parseable.com/docs/user-guide/promql) that lets you query metrics stored in Parseable using the standard Prometheus Query Language.
* **No separate instance needed:** You can ingest, store, and query metrics all within Parseable.
* **Standard conventions:** All endpoints live under the `/prometheus/api/v1` base path and follow the Prometheus HTTP API conventions.
* **Compatibility:** Existing Grafana dashboards and Prometheus-compatible tooling work without changes pointing to the datasource at your Parseable instance. Refer to the [Parseable Grafana plugin](https://www.parseable.com/docs/integrations/visualization/grafana) for details on how to configure Grafana to use Parseable as a Prometheus-compatible datasource.
# Multi tenancy (/user-guide/multi-tenancy)
Multi tenancy lets one Parseable deployment serve multiple teams, customers, or internal workspaces while keeping their data, users, roles, and queries scoped to the right tenant. This matters when you want a shared observability platform without asking every team to run and operate its own Parseable cluster.
It is also useful when a cluster has more capacity than one team needs right now. Instead of running separate low-utilization clusters for every workspace, teams can share the same Parseable deployment and still keep tenant data and access separate.
In Parseable, a tenant is the boundary used to decide where telemetry is written, which datasets a user can see, which roles apply, and which tenant metadata should be loaded. The same Parseable process can receive data for many tenants, but every request has to resolve to one tenant before it reaches storage or query execution.
## When to use it
Use multi tenancy when you want one Parseable deployment for multiple isolated workspaces. Typical examples are internal platform teams serving several engineering groups, managed environments where every customer needs a separate workspace, or shared clusters where teams should not see each other's datasets.
A common reason is capacity utilization. If a Parseable cluster is sized for production growth but currently runs below that capacity, multi tenancy lets more teams use the same cluster while tenant isolation keeps their telemetry, users, and permissions separate.
If you only run Parseable for one team, keep the default single-tenant mode. In single-tenant mode, requests do not need a tenant header and Parseable uses the default tenant internally.
## Enable multi tenancy
Multi tenancy is controlled by `P_MULTI_TENANCY`.
```bash
export P_MULTI_TENANCY=true
```
When this flag is enabled, Parseable expects tenant-aware request handling. Tenant metadata is loaded during startup, tenant directories are recognized from storage, and runtime operations use the tenant ID to scope users, roles, datasets, staging files, object storage paths, query schemas, and metrics.
When this flag is disabled, Parseable runs in single-tenant mode. In that mode, tenant-aware public headers are not expected from clients. If a client sends `x-p-tenant` to a single-tenant deployment, Parseable treats that as an unexpected header.
## Request flow
At a high level, a request moves through these steps:
1. Parseable receives the request.
2. Authentication middleware resolves the user and tenant.
3. The tenant is attached to the request context.
4. The handler uses that tenant to read or write datasets, metadata, roles, alerts, dashboards, and other resources.
5. In distributed deployments, internal requests forward the tenant to the next node using internal cluster headers.
```text
Client request
|
| x-p-tenant: acme
| x-api-key: ...
v
Auth middleware
|
| resolves user + tenant
v
Tenant scoped request
|
+--> ingest path writes under tenant storage prefix
+--> query path builds tenant specific query schema
+--> RBAC path checks tenant specific users and roles
+--> cluster path forwards tenant internally when required
```
The important part is that tenant resolution happens before the request reaches the feature handler. After that, most of the code receives a `tenant_id` and uses it as part of its normal read or write path.
## How Parseable identifies a tenant
Parseable treats the tenant as the workspace boundary. In practice, a tenant usually maps to a team, customer, or internal workspace. That tenant then owns its users, API keys, roles, datasets, dashboards, alerts, and metadata.
For ingestion and API-key based requests, the tenant is identified from `x-p-tenant`. Parseable reads this value early in the request, uses it to find the right tenant metadata, and then checks the API key or user inside that tenant. For logged-in Prism users and Basic Auth users, Parseable can derive the tenant from the authenticated user and attach it to the request before the handler runs.
This is what keeps the shared cluster model safe to use. The infrastructure can be shared, but the request is handled inside one tenant boundary before it can read or write data.
## What is shared and what is isolated
Multi tenancy does not mean every tenant gets a separate Parseable process. The cluster is shared, and the tenant boundary decides which resources are scoped.
| Shared at the cluster level | Scoped to each tenant |
| -------------------------------- | ------------------------------------------------ |
| Parseable nodes | Datasets and stream metadata |
| Ingest and query services | Users, API keys, roles, and user groups |
| Object storage backend | Dashboards, alerts, filters, and saved resources |
| Cluster capacity | Storage prefixes and query schemas |
| Internal node-to-node forwarding | Tenant metadata and service suspension state |
## Ingestion behavior
In multi tenant mode, ingestion requests must include `x-p-tenant`.
```bash
curl -X POST "https://parseable.example.com/api/v1/ingest" \
-H "x-p-tenant: acme" \
-H "x-p-stream: app-logs" \
-H "x-api-key: ${PARSEABLE_API_KEY}" \
-H "content-type: application/json" \
-d '[{"level":"info","message":"checkout started"}]'
```
The tenant header tells Parseable which workspace should receive the payload. Parseable does not allow the default tenant to be used as the explicit ingestion tenant in multi tenant mode. This avoids accidentally writing multi tenant data into the single-tenant compatibility path.
If a request uses Basic Auth, Parseable also checks that the authenticated user belongs to the tenant in the header. This prevents a user from one tenant from writing data into another tenant by changing only the request header.
## API keys
API keys are tenant-scoped in multi tenant mode. When a request carries `x-api-key`, Parseable uses the incoming `x-p-tenant` value as the tenant hint and looks for the API key user inside that tenant.
That means a multi tenant API key request needs both headers:
```bash
x-p-tenant: acme
x-api-key:
```
If `P_MULTI_TENANCY=true` and the tenant hint is missing, the API key cannot be resolved. In single-tenant mode, API keys are looked up under the default tenant.
The lookup is intentionally tenant-aware. Parseable does not search every tenant for the API key. It searches the users for the tenant named by `x-p-tenant`, finds the API-key user, builds permissions from that user's roles, and then lets the normal authorization flow decide whether the action is allowed.
## Query and Prism behavior
For most non-ingestion requests, users do not manually choose a tenant through a raw header. Parseable derives the tenant from the authenticated user session or Basic Auth user and attaches it to the request before the handler runs.
This is why the same query endpoint can stay simple from the user's point of view. The query handler receives the tenant-aware request, builds the query against that tenant's datasets, and uses the tenant as the default query schema.
For example, if a user belongs to tenant `acme`, a query against `app-logs` is resolved inside the `acme` workspace. Another tenant can have a dataset with the same name without sharing data with `acme`.
## Storage layout
Tenant isolation is also reflected in storage. When a tenant ID is present, Parseable writes tenant data under a tenant prefix. This applies to staging paths, stream metadata, schema files, alerts, dashboards, filters, and data files.
In simple terms:
```text
single-tenant mode
staging/
parseable.json
app-logs/
multi tenant mode
staging/
acme/
parseable.json
app-logs/
globex/
parseable.json
app-logs/
```
The same pattern is used for object storage paths. A tenant-aware write uses the tenant ID as part of the relative path, so datasets with the same name in two tenants are still stored separately.
## Tenant metadata
Every tenant has its own metadata. Parseable uses this metadata to understand the tenant's users, roles, datasets, default role, suspended services, owner, plan, and other workspace-level details.
Parseable does not infer team ownership from dataset names. The tenant is the explicit boundary. If a team, customer, or workspace should be isolated, it should have its own tenant metadata, users, API keys, and role assignments.
At startup, Parseable loads recognized tenant metadata from storage and builds the in-memory tenant map. From that point on, request handling can quickly look up the tenant and apply the right permissions and service state.
## RBAC boundary
RBAC is evaluated inside the tenant boundary. Users, roles, user groups, filters, dashboards, alerts, and saved resources are read through the resolved tenant.
This gives you two layers of protection:
1. The request first resolves to a tenant.
2. The user's roles are checked inside that tenant before the action is allowed.
For example, a user with query access in tenant `acme` does not automatically get query access in tenant `globex`. The same username can exist in different tenants with different roles because the tenant is part of the lookup.
## Service suspension
Parseable can suspend parts of a tenant workspace. The supported service states are:
| Service | Effect |
| ----------- | -------------------------------------------------- |
| `ingest` | New ingestion requests for that tenant are blocked |
| `query` | Query requests for that tenant are blocked |
| `workspace` | The tenant workspace is suspended |
Suspension is checked during request authorization. If a tenant's ingest service is suspended, ingestion fails for that tenant while other tenants continue to work.
## Operational notes
* Enable `P_MULTI_TENANCY=true` only when the deployment is meant to run with tenant-aware metadata and storage.
* Use `x-p-tenant` for ingestion and API-key based requests in multi tenant mode.
* Do not send `x-p-tenant` to a single-tenant deployment.
* Avoid using `default` as an explicit tenant ID for multi tenant ingestion.
## Troubleshooting
* If ingestion fails with a missing tenant error, check that the request includes `x-p-tenant`.
* If an API key request fails, check that both `x-api-key` and `x-p-tenant` are present. API keys are resolved inside the tenant named by `x-p-tenant`.
* If `x-p-tenant` is rejected, check whether the deployment is running in single-tenant mode. Single-tenant deployments do not expect tenant headers from clients.
* If a user can log in but cannot see expected datasets, check the user's tenant and role assignment.
# OpenID Connect (/user-guide/openid)
[OpenID Connect](https://openid.net/) (OIDC) lets Parseable delegate user
sign-in to an identity provider while keeping Parseable roles and permissions in
control of what each user can access after login. Teams usually use it when they
already manage users in systems such as Auth0, Okta, Azure AD, Google Workspace,
Keycloak, or Authentik.
Parseable connects to the identity provider through OIDC discovery. The provider
must expose its metadata at a discovery URL such as
`https://identity-provider.example.com/.well-known/openid-configuration`.
Parseable uses that metadata to find the authorization, token, userinfo, logout,
and JWKS endpoints required for the login flow.
## Prerequisites
Before you start, keep these values ready:
* A Parseable instance with admin access.
* A public HTTPS URL for Parseable, for example `https://demo.parseable.com`.
* An OIDC client or application created in your identity provider.
* The OIDC client ID, client secret, and issuer URL.
* A redirect URI registered in the identity provider as
`/api/v1/o/code`. For example, when Parseable is
hosted at `https://demo.parseable.com`, register
`https://demo.parseable.com/api/v1/o/code`.
* A default OIDC role in Parseable, or OIDC group claims that match role names
already created in Parseable.
If you use group-to-role mapping, make sure your identity provider includes the
`groups` claim in the ID token. Parseable reads those group values and matches
them with roles created in Parseable.
## Environment Variables
Set the following environment variables on the Parseable node that serves the
Console and login endpoints:
| Variable Name | Required | Description | Default | Example |
| ---------------------- | -------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------- | ----------------------------- |
| `P_OIDC_CLIENT_ID` | Yes | Client ID from the OIDC client or application created in your identity provider. | `""` | `parseable` |
| `P_OIDC_CLIENT_SECRET` | Yes | Client secret from the same OIDC client or application. | `""` | `client-secret` |
| `P_OIDC_ISSUER` | Yes | Issuer URL for the identity provider. The issuer must support OIDC discovery. | `""` | `https://accounts.google.com` |
| `P_ORIGIN_URI` | Yes | Public base URL where users access Parseable. This is used to build the callback URL. | `""` | `https://demo.parseable.com/` |
| `P_OIDC_SCOPE` | No | OIDC scopes requested during login. Keep the default unless your provider requires a different scope set. | `openid profile email offline_access` | `openid profile email groups` |
Parseable builds the OIDC callback URL from `P_ORIGIN_URI`. Do not set
`P_OIDC_REDIRECT_URI`. In your identity provider, register the redirect URI as
the Parseable public URL followed by `/api/v1/o/code`. For example, when
`P_ORIGIN_URI=https://demo.parseable.com`, register
`https://demo.parseable.com/api/v1/o/code`.
## Kubernetes with Helm chart 3.0.0 and later
Helm chart `3.0.0` and later configures environment variables per Parseable
node. Add the OIDC variables to the node that serves the Console and login
endpoints. The global `parseable.env` value used by chart `2.x` is no longer
supported.
Keep the values file containing `P_OIDC_CLIENT_SECRET` secure and do not commit
it to source control.
### Standalone OSS
```yaml
parseable:
standalone:
unified:
env:
P_OIDC_CLIENT_ID: ""
P_OIDC_CLIENT_SECRET: ""
P_OIDC_ISSUER: "https://identity-provider.example.com"
P_ORIGIN_URI: "https://logs.example.com/"
```
### Distributed OSS
The querier serves the Console and login endpoints:
```yaml
parseable:
distributed:
querier:
env:
P_OIDC_CLIENT_ID: ""
P_OIDC_CLIENT_SECRET: ""
P_OIDC_ISSUER: "https://identity-provider.example.com"
P_ORIGIN_URI: "https://logs.example.com/"
```
### Distributed Enterprise
Prism is the public entry point in an Enterprise deployment:
```yaml
parseable:
distributed:
prism:
env:
P_OIDC_CLIENT_ID: ""
P_OIDC_CLIENT_SECRET: ""
P_OIDC_ISSUER: "https://identity-provider.example.com"
P_ORIGIN_URI: "https://logs.example.com/"
```
When upgrading from chart `2.x`, move the variables from `parseable.env` to
`parseable.standalone.unified.env`, `parseable.distributed.querier.env`, or
`parseable.distributed.prism.env`, depending on the deployment mode. Helm
does not migrate the old values automatically.
If a reverse proxy terminates TLS, configure it to preserve the original host
and protocol headers. `P_ORIGIN_URI` must be the same public HTTPS URL that is
used to register the redirect URI in the identity provider.
## Privilege Management with OIDC
After a user signs in through OIDC, Parseable still needs to decide what that
user can do. You can handle this in two ways:
* Set a default OIDC role for new OIDC users.
* Map OIDC groups from the identity provider to Parseable roles.
The group-to-role path is useful when your identity provider already manages
teams such as `platform`, `security`, or `support`. The default role path is
useful when every OIDC user should start with the same baseline access.
### Assign default role to any new OIDC user
Users receive the default OIDC role when they are not part of any mapped group,
or when their OIDC groups do not have matching roles in Parseable.
To set a default OIDC role:
1. Log in to Parseable as an admin.
2. Open **Users** from the left navigation.
3. Create a role with the privileges you want new OIDC users to receive.
4. Click **Set Default OIDC Role**.
5. Select the role you created and save the setting.
Once this is set, new OIDC users who do not match a group-specific role will be
assigned this default role at login.
### Multi-Role Support
Parseable can assign multiple roles to the same OIDC user. This is useful when a
user belongs to more than one team or needs access to different datasets with
different permissions.
For example:
* A platform user can have `writer` access on one dataset and `reader` access on another.
* A support user can receive read access across selected datasets without manual user-by-user setup.
* The same role assignments are respected by both the UI and API.
### Map user groups to roles on Parseable
To map OIDC groups to Parseable roles, create roles in Parseable with the same
names as the group values sent by your identity provider. For example, if the ID
token contains a group named `platform-admins`, create a Parseable role named
`platform-admins`.
You can create roles from the Parseable UI or through the create role API. Once
the roles exist, users can sign in with SSO and Parseable will apply the matching
role permissions during login.
Per-user customization is not available for OIDC users. If a user needs a
different permission set, create a new role and manage that assignment from the
identity provider.
# PromQL (/user-guide/promql)
Parseable Enterprise includes a built-in PromQL engine that lets you query metrics stored in Parseable using the standard Prometheus Query Language. There is no need to run a separate Prometheus instance — you can ingest, store, and query metrics all within Parseable.
All endpoints live under the `/prometheus/api/v1` base path and follow the Prometheus HTTP API conventions, so existing Grafana dashboards and Prometheus-compatible tooling work without changes — point the datasource at your Parseable instance.
## Capabilities
* **Instant and range queries** with full PromQL expression support
* **50+ PromQL functions** — `rate`, `histogram_quantile`, `holt_winters`, label operations, and more
* **12 aggregation operators** — `sum`, `avg`, `topk`, `quantile`, `count_values`, etc.
* **All standard binary operators** — arithmetic, comparison, and set operations
* **Prometheus Remote Write** ingestion — send metrics from any Prometheus-compatible agent
* **Cardinality analysis** endpoints for understanding your metrics landscape
* **Distributed execution** — queries automatically fan out across queriers in cluster mode
## Querying from Grafana
You can query Parseable PromQL from Grafana in two ways: the built-in Prometheus data source, or the Parseable Grafana plugin.
### Prometheus data source
Add a Prometheus data source in Grafana with the following settings:
* **Prometheus server URL:** the URL of your Parseable Prism node (e.g. `https://prism.example.com`)
* **Authentication:** enable basic auth and provide your Parseable username and password
* **Custom query parameters:** `stream=` (for example, `stream=otel_metrics`)
* **HTTP method:** `GET`
Because Parseable uses the dataset name as the logical table, the Prometheus data source works with **one dataset at a time** — the `stream` query parameter pins it. If you have multiple metrics datasets, you'll need to add a separate Prometheus data source per dataset.
### Parseable Grafana plugin
If you have multiple metrics datasets (or want to mix telemetry signals), use the Parseable Grafana plugin instead. The plugin lets you switch between datasets and write PromQL queries directly in the editor.
The plugin works in **Dashboards** and **Explore**, but is **not** supported in Grafana **Drilldown**.
A useful side benefit: the same plugin lets you build dashboards from logs, traces, and metrics together — all telemetry signals in one place.
## API Endpoints
All endpoints require the `Authorization` header with a valid Parseable credential (basic auth). All query and metadata endpoints require the `Query` action permission. The remote write endpoint requires the `Ingest` action permission.
### Instant Query
Evaluate a PromQL expression at a single point in time.
**Endpoint:** `GET /prometheus/api/v1/query` or `POST /prometheus/api/v1/query`
| Parameter | Required | Type | Description |
| ------------------ | -------- | ------ | ------------------------------------------------------------------------------------------------ |
| `query` | Yes | string | PromQL expression to evaluate |
| `stream` | Yes | string | Name of the metrics stream to query (e.g. `otel_metrics`) |
| `time` | No | string | Evaluation timestamp. RFC3339 (`2024-01-15T10:30:00Z`) or unix seconds. Defaults to current time |
| `timeout` | No | float | Query timeout in seconds. Default: `120` |
| `limit` | No | int | Maximum number of data points to return |
| `offset` | No | int | Number of data points to skip |
| `trace` | No | string | Set to `1` or `true` to include execution trace in response |
| `timestamp_format` | No | string | `rfc3339` for ISO 8601 timestamps in results, otherwise unix epoch seconds |
```bash
# Simple metric lookup
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/query?\
query=http_requests_total&\
stream=otel_metrics"
# Rate calculation at a specific time
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/query?\
query=rate(http_requests_total[5m])&\
stream=otel_metrics&\
time=2024-01-15T10:30:00Z"
```
**Success response:**
```json
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"__name__": "http_requests_total",
"job": "api-server",
"instance": "10.0.0.1:8080"
},
"value": [1705312200, "1542.5"]
}
]
}
}
```
### Range Query
Evaluate a PromQL expression over a time range, returning a time series matrix.
**Endpoint:** `GET /prometheus/api/v1/query_range` or `POST /prometheus/api/v1/query_range`
| Parameter | Required | Type | Description |
| ------------------ | -------- | ------ | ------------------------------------------------------------------------------------------------------------- |
| `query` | Yes | string | PromQL expression to evaluate |
| `start` | Yes | string | Start timestamp. RFC3339 or unix seconds |
| `end` | Yes | string | End timestamp. RFC3339 or unix seconds |
| `stream` | Yes | string | Name of the metrics stream to query |
| `step` | No | string | Query resolution step. Duration string (`15s`, `1m`, `5m`, `1h`) or float seconds. Auto-calculated if omitted |
| `timeout` | No | float | Query timeout in seconds. Default: `120` |
| `limit` | No | int | Maximum number of data points to return |
| `offset` | No | int | Number of data points to skip |
| `trace` | No | string | `1` or `true` to include execution trace |
| `timestamp_format` | No | string | `rfc3339` for ISO 8601 timestamps in results |
```bash
# Range query over the last hour with 1-minute resolution
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/query_range?\
query=rate(http_requests_total[5m])&\
stream=otel_metrics&\
start=2024-01-15T09:00:00Z&\
end=2024-01-15T10:00:00Z&\
step=60s"
# 99th percentile latency, aggregated by job
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/query_range?\
query=histogram_quantile(0.99,%20sum(rate(http_request_duration_seconds_bucket[5m]))%20by%20(le,%20job))&\
stream=otel_metrics&\
start=1705276800&\
end=1705312200&\
step=60"
```
**Success response:**
```json
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {"job": "api-server", "instance": "10.0.0.1:8080"},
"values": [
[1705276800, "12.5"],
[1705276860, "14.2"],
[1705276920, "13.8"]
]
}
]
}
}
```
### Series Metadata
Find time series that match certain label matchers.
**Endpoint:** `GET /prometheus/api/v1/series` or `POST /prometheus/api/v1/series`
| Parameter | Required | Type | Description |
| --------- | -------- | --------- | ------------------------------------------------------------------------------- |
| `match[]` | Yes | string\[] | Repeated parameter. Series selector (e.g. `{job="api"}`, `http_requests_total`) |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `start` | No | string | Start timestamp (RFC3339 or unix seconds) |
| `end` | No | string | End timestamp (RFC3339 or unix seconds) |
### Labels
Get a list of all label names present in the data.
**Endpoint:** `GET /prometheus/api/v1/labels`
| Parameter | Required | Type | Description |
| --------- | -------- | ------ | ------------------------------------------ |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `start` | No | string | Start timestamp |
| `end` | No | string | End timestamp |
### Label Values
Get a list of distinct values for a specific label.
**Endpoint:** `GET /prometheus/api/v1/label/{label_name}/values`
| Parameter | Required | Type | Description |
| --------- | -------- | ------ | ------------------------------------------ |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `start` | No | string | Start timestamp |
| `end` | No | string | End timestamp |
```bash
# All values of the "job" label
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/label/job/values?\
stream=otel_metrics"
# All metric names
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/label/__name__/values?\
stream=otel_metrics"
```
### TSDB Status
Get statistics about the time series database. Compatible with the VictoriaMetrics TSDB status API.
**Endpoint:** `GET /prometheus/api/v1/status/tsdb`
| Parameter | Required | Type | Description |
| ------------ | -------- | --------- | --------------------------------------------------------- |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `topN` | No | int | Max entries per category. Default: `10` |
| `date` | No | string | Date to analyze in `YYYY-MM-DD` format. Defaults to today |
| `match[]` | No | string\[] | Series selector to filter by |
| `focusLabel` | No | string | Label to break down series counts by value |
### Cardinality
#### Label Names
Find which labels have the highest cardinality (most distinct values).
**Endpoint:** `GET /prometheus/api/v1/cardinality/label_names`
| Parameter | Required | Type | Description |
| ---------- | -------- | ------ | ------------------------------------------------- |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `lookback` | No | int | Seconds to look back from now. Default: `3600` |
| `limit` | No | int | Maximum number of labels to return. Default: `20` |
| `selector` | No | string | Label selector to filter series |
#### Label Values
For a given label, find how many series exist for each of its values.
**Endpoint:** `GET /prometheus/api/v1/cardinality/label_values`
| Parameter | Required | Type | Description |
| ------------ | -------- | ------ | ------------------------------------------------- |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `label_name` | No | string | The label to analyze |
| `lookback` | No | int | Seconds to look back from now. Default: `3600` |
| `limit` | No | int | Maximum number of values to return. Default: `20` |
#### Active Series
List all currently active series matching a selector.
**Endpoint:** `GET /prometheus/api/v1/cardinality/active_series`
| Parameter | Required | Type | Description |
| ---------- | -------- | ------ | ------------------------------------------------- |
| `stream` | No | string | Metrics stream. Defaults to `otel_metrics` |
| `lookback` | No | int | Seconds to look back from now. Default: `3600` |
| `limit` | No | int | Maximum number of series to return. Default: `20` |
| `selector` | No | string | Label selector |
### Active Queries
List all currently executing PromQL queries.
**Endpoint:** `GET /prometheus/api/v1/status/active_queries`
```bash
curl -u admin:admin "http://localhost:8000/prometheus/api/v1/status/active_queries"
```
## Supported PromQL Functions
### Rate and Counter Functions
These functions operate on counter and gauge metrics over a time range (matrix selector).
* `rate(v range-vector)` — per-second average rate of increase. Handles counter resets.
* `irate(v range-vector)` — per-second instant rate using only the last two data points.
* `increase(v range-vector)` — total increase in the counter over the time range.
* `delta(v range-vector)` — difference between the first and last value (use with gauges).
* `idelta(v range-vector)` — like `delta`, but uses only the last two data points.
```
rate(http_requests_total[5m])
increase(http_requests_total[1h])
```
### Aggregation Over Time Functions
Aggregate samples within a time range for each series independently.
* `avg_over_time(v range-vector)`
* `sum_over_time(v range-vector)`
* `min_over_time(v range-vector)`
* `max_over_time(v range-vector)`
* `count_over_time(v range-vector)`
* `last_over_time(v range-vector)`
* `stddev_over_time(v range-vector)`
* `stdvar_over_time(v range-vector)`
* `quantile_over_time(scalar, v range-vector)` — φ-quantile (0 ≤ φ ≤ 1) of values over the time range
* `present_over_time(v range-vector)` — value `1` for any series that has at least one sample
```
avg_over_time(cpu_usage_percent[10m])
quantile_over_time(0.95, http_request_duration_seconds[1h])
```
### Counter Tracking Functions
* `changes(v range-vector)` — number of times a value changed in the range.
* `resets(v range-vector)` — number of counter resets (value decreases) in the range.
### Trend and Prediction Functions
* `deriv(v range-vector)` — per-second derivative via simple linear regression.
* `predict_linear(v range-vector, t scalar)` — predicts the value `t` seconds from now.
```
# Will we run out of disk in 24 hours?
predict_linear(node_filesystem_free_bytes[6h], 24 * 3600) < 0
```
### Sorting Functions
* `sort(v instant-vector)` — ascending by sample value.
* `sort_desc(v instant-vector)` — descending by sample value.
* `sort_by_label(v instant-vector, label string, ...)` — alphabetical by label values.
* `sort_by_label_desc(v instant-vector, label string, ...)` — reverse alphabetical.
### Math Functions
All math functions operate element-wise on each sample value.
| Function | Description |
| ---------- | -------------------------- |
| `abs(v)` | Absolute value |
| `ceil(v)` | Round up |
| `floor(v)` | Round down |
| `round(v)` | Round to nearest |
| `sqrt(v)` | Square root |
| `exp(v)` | Exponential (e^x) |
| `ln(v)` | Natural logarithm |
| `log2(v)` | Base-2 logarithm |
| `log10(v)` | Base-10 logarithm |
| `sgn(v)` | Sign function: -1, 0, or 1 |
### Clamping Functions
* `clamp(v instant-vector, min scalar, max scalar)`
* `clamp_min(v instant-vector, min scalar)`
* `clamp_max(v instant-vector, max scalar)`
### Type Conversion Functions
* `scalar(v instant-vector)` — converts a single-element vector to a scalar (returns `NaN` otherwise).
* `vector(s scalar)` — converts a scalar to a single-element vector with no labels.
### Time Functions
* `time()` — current evaluation timestamp (seconds since epoch).
* `timestamp(v instant-vector)` — timestamp of each sample as its value.
| Function | Returns |
| ------------------- | -------------------------- |
| `hour(v?)` | Hour (0–23) |
| `minute(v?)` | Minute (0–59) |
| `month(v?)` | Month (1–12) |
| `year(v?)` | Year |
| `day_of_month(v?)` | Day of month (1–31) |
| `day_of_week(v?)` | Day of week (0=Sun, 6=Sat) |
| `day_of_year(v?)` | Day of year (1–365/366) |
| `days_in_month(v?)` | Days in the month (28–31) |
### Label Manipulation Functions
* `label_replace(v, dst_label, replacement, src_label, regex)` — replace or create a label using regex match and substitution.
* `label_join(v, dst_label, separator, src_label_1, src_label_2, ...)` — join multiple label values into a new label.
```
label_replace(up, "port", "$1", "instance", ".*:(.*)")
label_join(up, "job_instance", "/", "job", "instance")
```
### Histogram Functions
* `histogram_quantile(φ scalar, v instant-vector)` — φ-quantile from a conventional Prometheus histogram. Input must contain bucket series with the `le` label.
* `histogram_count(v range-vector)` — count from histogram data.
* `histogram_sum(v range-vector)` — sum from histogram data.
* `histogram_fraction(lower scalar, upper scalar, v range-vector)` — estimated fraction of observations between bounds.
```
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
```
### Absence Detection Functions
* `absent(v instant-vector)` — vector with value `1` if input is empty.
* `absent_over_time(v range-vector)` — vector with value `1` if range vector has no samples.
### Smoothing Functions
* `holt_winters(v range-vector, sf scalar, tf scalar)` — double exponential smoothing. `sf` is the smoothing factor and `tf` is the trend factor; both must be in `(0, 1)`.
```
holt_winters(rate(http_requests_total[5m])[30m:1m], 0.5, 0.7)
```
## Aggregation Operators
Aggregation operators take a vector and aggregate across series. All support `by` and `without` clause modifiers for grouping.
```
([parameter,] ) [by|without ()]
```
| Operator | Description |
| -------------------------- | ------------------------------------- |
| `sum` | Sum of values across series |
| `avg` | Arithmetic mean across series |
| `count` | Number of series |
| `min` | Minimum value across series |
| `max` | Maximum value across series |
| `stddev` | Population standard deviation |
| `stdvar` | Population variance |
| `quantile(φ, v)` | φ-quantile across series (0 ≤ φ ≤ 1) |
| `topk(k, v)` | Top K series by value |
| `bottomk(k, v)` | Bottom K series by value |
| `count_values("label", v)` | Count series with each distinct value |
| `group` | Returns 1 for each distinct label set |
```
sum(rate(http_requests_total[5m])) by (job)
topk(5, rate(http_requests_total[5m]))
quantile(0.9, http_request_duration_seconds) by (service)
```
## Binary Operators
Binary operators work between two scalars, two vectors, or a scalar and a vector.
### Arithmetic Operators
| Operator | Description |
| -------- | -------------- |
| `+` | Addition |
| `-` | Subtraction |
| `*` | Multiplication |
| `/` | Division |
| `%` | Modulo |
| `^` | Power |
| `atan2` | Arc tangent |
### Comparison Operators
Return the left-hand value if the comparison is true; otherwise the series is dropped. Add `bool` to return 0/1 instead.
| Operator | Description |
| -------- | ---------------- |
| `==` | Equal |
| `!=` | Not equal |
| `>` | Greater than |
| `<` | Less than |
| `>=` | Greater or equal |
| `<=` | Less or equal |
```
# Filter: only series where error rate exceeds 5%
rate(errors[5m]) / rate(requests[5m]) > 0.05
# Boolean mode: returns 0 or 1
rate(errors[5m]) > bool 0.05
```
### Set/Logical Operators
| Operator | Description |
| -------- | ----------------------------------------------------- |
| `and` | Intersection — left series that have a matching right |
| `or` | Union — all series from both sides |
| `unless` | Complement — left series that have no matching right |
## Vector Matching
When applying binary operators between two vectors, Parseable supports standard Prometheus vector matching.
```
# Match on all labels (default)
rate(errors[5m]) / rate(requests[5m])
# Match on specific labels
rate(errors[5m]) / on(job, instance) rate(requests[5m])
# Match on all labels except specific ones
rate(errors[5m]) / ignoring(status) rate(requests[5m])
# group_left: many-to-one (left side has more series)
rate(http_requests_total[5m]) / on(job) group_left rate(capacity[5m])
# group_right: one-to-many
rate(capacity[5m]) / on(job) group_right rate(http_requests_total[5m])
```
## Subqueries
Evaluate an instant query over a range of time with a given resolution.
```
[:]
```
```
# Max of 5-minute rate, evaluated every minute over the last hour
max_over_time(rate(http_requests_total[5m])[1h:1m])
```
## Response Headers
Query endpoints include performance statistics in response headers on every successful response. Some headers only appear when the corresponding execution path was taken.
| Header | When set | Description |
| ------------------------------- | ----------------------------------------------- | ------------------------------------------------------ |
| `X-Parseable-Execution-Time-Ms` | Always (range queries) | Query execution time in milliseconds |
| `X-Parseable-Cache-Hit` | Always | Whether the result was served from cache |
| `X-Parseable-Cache-Status` | Range queries | `full`, `partial`, or `miss` |
| `X-Parseable-Query-Split` | Range queries | Whether the query was split into sub-ranges |
| `X-Parseable-Series-Fetched` | Local and split execution paths | Number of time series loaded from storage |
| `X-Parseable-Samples-Scanned` | Local and split execution paths | Total number of individual samples processed |
| `X-Parseable-Partial-Results` | Only when a querier returned partial results | Indicates incomplete distributed result |
| `X-Parseable-Query-Pushdown` | Only when the query was pushed down to queriers | Indicates distributed query pushdown strategy was used |
## Error Responses
All errors follow a consistent JSON format.
```json
{
"status": "error",
"errorType": "",
"error": ""
}
```
| Error Type | HTTP Status | Description |
| ----------- | ----------- | --------------------------------------------- |
| `bad_data` | 400 / 422 | Invalid PromQL syntax or malformed parameters |
| `execution` | 500 | Internal error during query evaluation |
| `not_found` | 404 | Referenced stream or dataset does not exist |
| `timeout` | 503 | Query exceeded the timeout limit |
## Environment Variables
These variables tune Parseable's PromQL query engine — query splitting, per-query limits, distributed execution, and result caching.
### Query splitting and execution
| Variable Name | Required | Description | Default | Example |
| ---------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- |
| `P_PROMQL_SPLIT_INTERVAL_HOURS` | `No` | Time interval (in hours) used to split long-range PromQL queries. | `24` | `12` |
| `P_PROMQL_MAX_CONCURRENT_SPLITS` | `No` | Maximum number of split sub-queries executed concurrently for a single PromQL request. | `8` | `16` |
| `P_PROMQL_ALIGN_QUERIES_WITH_STEP` | `No` | When `true`, range query `start`/`end` are floored to step boundaries to improve cache hit rates on jittered Grafana refreshes. | `true` | `false` |
| `P_PROMQL_PARTITION_OVERLAP_MINS` | `No` | Overlap window (in minutes) applied when fetching data across partition boundaries. | `5` | `10` |
### Per-query limits
| Variable Name | Required | Description | Default | Example |
| -------------------------------- | -------- | -------------------------------------------------------------------------------------------------------- | ------------------- | ------------ |
| `P_PROMQL_MAX_MEMORY_PER_QUERY` | `No` | Maximum estimated memory (in bytes) allowed per PromQL query. Queries exceeding this limit are rejected. | `2147483648` (2 GB) | `4294967296` |
| `P_PROMQL_MAX_SERIES_PER_QUERY` | `No` | Maximum number of time series a single PromQL query may evaluate. | `1000000` | `500000` |
| `P_PROMQL_MAX_SAMPLES_PER_QUERY` | `No` | Maximum number of samples a single PromQL query may evaluate. | `100000000` | `50000000` |
### Distributed execution
| Variable Name | Required | Description | Default | Example |
| --------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------- | ------- | ------- |
| `P_PROMQL_MAX_CONCURRENT_DISTRIBUTED_QUERIES` | `No` | Maximum number of distributed PromQL queries executed concurrently across the cluster. | `32` | `64` |
| `P_PROMQL_MAX_CONCURRENT_QUERIER_REQUESTS` | `No` | Maximum number of concurrent in-flight requests to querier nodes per PromQL query. | `16` | `32` |
| `P_PROMQL_ADMISSION_TIMEOUT_SECS` | `No` | Timeout (in seconds) for admitting a query into the distributed execution pool. | `10` | `30` |
| `P_PROMQL_DATA_FETCH_TIMEOUT_SECS` | `No` | Timeout (in seconds) for fetching data from a querier node during distributed PromQL execution. | `120` | `300` |
| `P_PROMQL_HOTTIER_INVENTORY_TIMEOUT_SECS` | `No` | Timeout (in seconds) for fetching hot-tier inventory from querier nodes. | `5` | `10` |
| `P_PROMQL_HOTTIER_CACHE_TTL_SECS` | `No` | TTL (in seconds) for the hot-tier inventory cache used during query planning. | `30` | `60` |
### Result cache
| Variable Name | Required | Description | Default | Example |
| ------------------------------ | -------- | ------------------------------------------------------------------------- | --------------------- | ----------------------------- |
| `P_PROMQL_CACHE_MAX_ENTRIES` | `No` | Maximum number of entries kept in the PromQL query result cache. | `10000` | `50000` |
| `P_PROMQL_CACHE_MAX_MEMORY_MB` | `No` | Maximum memory (in MB) used by the PromQL query result cache. | `8192` | `16384` |
| `P_PROMQL_CACHE_TTL_SECS` | `No` | TTL (in seconds) for entries in the PromQL query result cache. | `3600` | `7200` |
| `P_PROMQL_RESULTS_CACHE_DIR` | `No` | Local directory where PromQL query results are persisted across restarts. | System temp directory | `/var/cache/parseable/promql` |
# RBAC (/user-guide/rbac)
There are five entities in Parseable Access Control model - **Action**, **Privilege**, **Resource**, **Role** and **User**. Below section explains each of these entities in detail.
* **Actions**: Each API corresponds to an Action on the Parseable server.
* **Privilege**: It is a group of allowed actions. Actions and Privileges are predefined within a Parseable server instance. Current Privileges are Admin, Editor, Writer, Reader and Ingester.
* **Resources**: Log datasets are Resources. Each Resource has a unique name. For example, a log dataset with name my\_dataset is a Resource.
* **Roles**: Roles are dynamic, named entities on a Parseable server instance. Each role has a set of privileges and resources associated with it. A role can be assigned to several users. A user can have multiple roles assigned to it.
* **Users**: Users refer to human or machine entities that can perform actions on a Parseable server instance. Each user has a unique username and password. A user can be assigned one or more roles.
User passwords are hashed and stored in Parseable metadata file. Parseable does not store the password in plain text.
## Overview of Roles & Access
Each role—Admin, Editor, Writer, Reader, and Ingestor—has varying access to different endpoints, categorized into six sections: General, Access Management, Resource Based, Stream Related, and Query & Ingest Logs Related. Access permissions are denoted with either `✓` (allowed) or `x` (denied).
### General
This section covers general system and informational endpoints, which are accessible to most roles for actions such as viewing the system's status or metrics.
| Action | Endpoint | Admin | Editor | Writer | Reader | Ingester |
| ------------------ | --------------------------- | ----- | ------ | ------ | ------ | -------- |
| GetAbout | GET /about | ✓ | ✓ | ✓ | ✓ | x |
| GetAnalytics | GET /analytics | ✓ | x | x | x | x |
| GetLiveness | HEAD /liveness | ✓ | ✓ | ✓ | ✓ | x |
| GetReadiness | HEAD /readiness | ✓ | ✓ | ✓ | ✓ | x |
| ListCluster | GET /cluster/info | ✓ | x | x | x | x |
| ListClusterMetrics | GET /cluster/metrics | ✓ | x | x | x | x |
| DeleteIngestor | DELETE /cluster/\{ingestor} | ✓ | x | x | x | x |
| Metrics | GET /metrics | ✓ | ✓ | x | x | x |
### Access Management
This section deals with endpoints for managing roles and users. Only Admins have access to critical actions like creating, updating, and deleting roles or users, ensuring proper control over access management in the system.
| Action | Endpoint | Admin | Editor | Writer | Reader | Ingester |
| ------------ | -------------------------------------------- | ----- | ------ | ------ | ------ | -------- |
| PutRole | PUT /role/default | ✓ | x | x | x | x |
| PutRole | PUT /role/\{name} | ✓ | x | x | x | x |
| GetRole | GET /role/default | ✓ | x | x | x | x |
| GetRole | GET /role/\{name} | ✓ | x | x | x | x |
| DeleteRole | DELETE /role/\{name} | ✓ | x | x | x | x |
| ListRole | GET /role | ✓ | x | x | x | x |
| PutUser | POST /user/\{username} | ✓ | x | x | x | x |
| PutUser | POST /user/\{username}/generate-new-password | ✓ | x | x | x | x |
| ListUser | GET /user | ✓ | x | x | x | x |
| DeleteUser | DELETE /user/\{username} | ✓ | x | x | x | x |
| PutUserRoles | PUT /user/\{username}/role | ✓ | x | x | x | x |
| GetUserRoles | GET /user/\{username}/role | ✓ | ✓ | ✓ | ✓ | x |
### Resource Management
This section defines access to resources such as dashboards and filters. While most roles can view and create resources, only Admins and Editors have permission to modify or delete them.
| Action | Endpoint | Admin | Editor | Writer | Reader | Ingester |
| --------------- | ----------------------------------- | ----- | ------ | ------ | ------ | -------- |
| ListDashboard | GET /dashboards | ✓ | ✓ | ✓ | ✓ | x |
| GetDashboard | GET /dashboards/\{dashboard\_id} | ✓ | ✓ | ✓ | ✓ | x |
| CreateDashboard | POST /dashboards | ✓ | ✓ | ✓ | ✓ | x |
| CreateDashboard | PUT /dashboards/\{dashboard\_id} | ✓ | ✓ | ✓ | ✓ | x |
| DeleteDashboard | DELETE /dashboards/\{dashboard\_id} | ✓ | ✓ | ✓ | ✓ | x |
| ListFilter | GET /filters | ✓ | ✓ | ✓ | ✓ | x |
| GetFilter | GET /filters/\{filter\_id} | ✓ | ✓ | ✓ | ✓ | x |
| CreateFilter | POST /filters | ✓ | ✓ | ✓ | ✓ | x |
| CreateFilter | PUT /filters/\{filter\_id} | ✓ | ✓ | ✓ | ✓ | x |
| DeleteFilter | DELETE /filters/\{filter\_id} | ✓ | ✓ | ✓ | ✓ | x |
### Stream Related
This section deals with endpoints for managing datasets. Admins and Editors have full access to these endpoints, while other roles have limited or no access to dataset management functionalities.
| Action | Endpoint | Admin | Editor | Writer | Reader | Ingester |
| -------------------- | -------------------------------------- | ----- | ------ | ------ | ------ | -------- |
| CreateStream | PUT /logstream/\{logstream} | ✓ | ✓ | x | x | x |
| DeleteStream | DELETE /logstream/\{logstream} | ✓ | ✓ | x | x | x |
| GetSchema | GET /logstream/\{logstream}/schema | ✓ | ✓ | ✓ | ✓ | x |
| GetStats | GET /logstream/\{logstream}/stats | ✓ | ✓ | ✓ | ✓ | x |
| GetStreamInfo | GET /logstream/\{logstream}/info | ✓ | ✓ | ✓ | ✓ | x |
| ListStream | GET /logstream | ✓ | ✓ | ✓ | ✓ | x |
| PutAlert | PUT /logstream/\{logstream}/alert | ✓ | ✓ | ✓ | x | x |
| GetAlert | GET /logstream/\{logstream}/alert | ✓ | ✓ | ✓ | x | x |
| PutHotTierEnabled | PUT /logstream/\{logstream}/hottier | ✓ | ✓ | ✓ | x | x |
| GetHotTierEnabled | GET /logstream/\{logstream}/hottier | ✓ | ✓ | ✓ | x | x |
| DeleteHotTierEnabled | DELETE /logstream/\{logstream}/hottier | ✓ | ✓ | ✓ | x | x |
| GetRetention | GET /logstream/\{logstream}/retention | ✓ | ✓ | ✓ | x | x |
| PutRetention | PUT /logstream/\{logstream}/retention | ✓ | ✓ | ✓ | x | x |
### Query & Ingest Logs Related
This section highlights endpoints related to querying and ingesting logs. Admins and Editors have full access to these functionalities, while other roles, like Readers and Ingestors, may have restricted access depending on their responsibilities.
| Action | Endpoint | Admin | Editor | Writer | Reader | Ingester |
| -------- | ---------------------------- | ----- | ------ | ------ | ------ | -------- |
| Ingest | POST /logstream/\{logstream} | ✓ | ✓ | ✓ | x | ✓ |
| Ingest | POST /ingest | ✓ | ✓ | ✓ | x | ✓ |
| Query | POST /query | ✓ | ✓ | ✓ | ✓ | x |
| QueryLLM | POST /llm | ✓ | ✓ | ✓ | ✓ | x |
### Get started
#### Creating a Role
This is the first step in setting up Role Based Access Control (RBAC) for Parseable. Use the Create Role API to create a role. The Create Role API request body requires the role definition in JSON format. Below examples demonstrate sample JSON for different types of role and privileges.
Role JSON with Admin Privilege
```json
[
{
"privilege": "admin"
}
]
```
Role JSON with Editor Privilege
```json
[
{
"privilege": "editor"
}
]
```
Role JSON with Writer Privilege: The Writer privilege is resource specific. A user with above role json, will be able to call the Writer specific API only on the specified resource. In the above example, the user will be able to call Writer specific API on backend and frontend datasets only.
```json
[
{
"privilege": "writer",
"resource": {
"dataset": "backend"
}
},
{
"privilege": "writer",
"resource": {
"dataset": "frontend"
}
}
]
```
Role JSON with Ingester Privilege: The Ingester privilege is resource specific. A user with above role json, will be able to call the Ingester specific API only on the specified resource. In the above example, the user will be able to call Ingester specific API on backend and frontend datasets only. This privilege is useful to be set in log agents, forwarders, and other log ingestion tools.
```json
[
{
"privilege": "ingester",
"resource": {
"dataset": "backend"
}
},
{
"privilege": "ingester",
"resource": {
"dataset": "frontend"
}
}
]
```
Role JSON with Reader Privilege: The Reader privilege is resource specific. A user with above role json, will be able to call the Reader specific API only on the specified resources. In the above example, the user will be able to call Reader specific API on frontend dataset, and only on events with tag source=web.
```json
[
{
"privilege": "reader",
"resource": {
"dataset": "frontend",
"tag": "source=web" // optional field
}
}
]
```
### Creating User
To create a User, use the Create User API. Here you can optionally pass a request body that has appropriate role name (as explained in the role section) to assign a role to the user.
After successful Create User API call, you'll get the user's password in the response. Keep it in a safe place as this is the only time server will return the password in plain text.
### Assign a role
To assign a role to a user after creating a user, use the Assign Role API. This API takes the username and role name as input. After a successful API call, the user will be able to perform actions allowed by the assigned role.
### Reset password
In any case if you need to reset password for a user. This can be done through Reset Password API.
### Delete user
To delete a user, use the Delete User API. This API will delete the user and all the roles assigned to it.
### OpenID Connect
For managing roles for your OAuth2 users, refer to OIDC section. Roles are automatically assigned by matching the role name with group name that is obtained to groups claim in the id token.
# Retention (/user-guide/retention)
Parseable allows setting the retention, or the amount of time that log data is kept in the system, for each dataset. The time can be set to a multiple of 1 day. Note that retention works at a dataset level, and each dataset can have a different retention period. Also, you can only set a single dataset per retention period.
## Setting up
You can set Retention via the Dataset Management page (Dataset >> Manage >> Retention). If you're using external applications to interact with Parseable, you can also use the retention API calls. Refer to the [API documentation](/api) for details.
## Configuration
Here is sample retention configuration, with all the available options.
```json
[
{
"duration": "20d",
"action": "delete",
"description": "delete logs after 20 days"
}
]
```
This table explains the configuration options.
| Variable Name | Required | Description |
| ------------- | -------- | ---------------------------------------------------------------------------------------------- |
| duration | Yes | Total duration for which logs should be retained. Can be multiple of 1 day, e.g. 20d. |
| action | Yes | Action to be taken when log data passes retention duration. Currently only delete is supported |
| description | No | Human friendly description of the log retention rule |
# Smart Cache (/user-guide/smart-cache)
Smart cache is only available to be set-up through API and not natively supported in the UI from Release `v2.0.0` onwards. We are working on adding this feature to Prism and it will be available soon.
## How it works
Tiering in Parseable allows keeping a copy of log data on the query node (in addition to the object store). You can create storage tiers on query node disks, allowing hot/recent data on SSD and older data backed by S3/object storage. This architecture allows for much faster query response, while keeping costs very low because data is always backed on object store.
The tiered storage capacity works at the log dataset level. You can specify the size on disk available to a specific dataset for its hot tier data. This is useful for situations where different datasets have different query patterns, i.e. some dataset need to be queried for predominantly recent data, while others not so much.
## Setup hot tier
To enable hot tier for a query node, add the environment variable `P_HOT_TIER_DIR` to the query node (or the standalone node) before starting the server. The value of environment variable should be set to the path of directory that you want to use for the data store. For example, `P_HOT_TIER_DIR=/path/to/hot/tier/directory`.
Setting the environment variable enables the global hot tier mechanism. You'll now need to set hot tier size for specific datasets based on your requirements. The setting is available in the Manage page of each dataset, under the Hot Tier Storage Size section.
## Under the hood
When the global hot tier mechanism is enabled, the server identifies the drive (where the hot tier directory is created) and calculates the total size and the free size of the drive. The upper threshold for hot tier size is set to 80% of the total drive capacity. So if the drive is of 10 TiB, then 8 TiB is automatically considered the maximum size of hot tier (subject to disk availability).
## Size allocation
Now when a specific dataset requests a hot tier capacity of let's say `2 TiB`, server checks if it is possible to allocate. The maths is : max size possible (8 TiB) - total used size of the disk (assume `1.3 TiB`) = `6.7 TiB`. The server then allocates the `2 TiB` to this dataset.
Once the hot tier is set up, a scheduler is configured to run every minute. This scheduler verifies if new files are available in remote object store and downloads them to the hot tier, ensuring that your most recent and frequently accessed data is always readily available.
## Populating hot tier
Based on the size allocated for the hot tier (for a dataset), the server starts downloading Parquet files from object store, beginning with the most recent data and moving backward in time. This approach ensures that the latest data, which is more likely to be queried, is prioritized.
As each file is downloaded, it’s recorded in a `hottier.manifest.json` file. This manifest file is crucial for tracking which Parquet files are stored locally in the hot tier. Along with this, the system also updates the available and used sizes in the hot tier's JSON file, providing a clear view of the hot tier’s current state.
The server deletes the oldest files when necessary. This happens under two conditions:
* Size exhaustion: When the total size of the files in the hot tier reaches the allocated limit.
* Disk usage threshold: When the combined disk usage, including the hot tier, exceeds the configured disk usage threshold (e.g., 80%).
The `hottier.manifest.json` is updated to reflect the removal of old files, ensuring that the hot tier remains within its defined constraints while continuing to serve the most relevant data efficiently.
## Query flow for hot tier
On receiving a query, the server fetches the dataset.json and related manifest.json files based on the query time range. It then identifies the list of Parquet file paths from the manifest. The server checks if these files are available in the hot tier. If any of the Parquet files are present in the hot tier path, server utilizes those file, avoiding S3 GET calls. For files not in the hot tier, the system fetches the necessary data from S3.
## Adjusting the hot tier Size
If you need to adjust the size of the hot tier for an existing dataset, you can do so with via the dataset Management page. Here’s how it works:
* Increasing the hot tier Size: When you increase the size of an existing hot tier, the system updates the meta file to reflect the new size. This allows for additional data to be stored locally without any interruption in service.
* Decreasing the hot tier Size: Reducing the size of the hot tier is not allowed. If you attempt to do so, the server will respond with an error, maintaining the integrity of your current data storage setup.
# SQL Editor (/user-guide/sql-editor)
Parseable provides a powerful SQL editor that allows you to query your telemetry data (logs, metrics, and traces) using PostgreSQL compatible SQL syntax.
## Getting Started
1. Navigate to the SQL Editor in the Parseable UI
2. Select your dataset from the dropdown
3. Write your SQL query
4. Click "Run" to execute
## Features
* **Full SQL Support** - Use standard SQL queries to analyze your data
* **Auto-completion** - Intelligent suggestions as you type
* **Query History** - Access your previous queries
* **Export Results** - Download query results in various formats
## Example Queries
```sql
-- Get recent data
SELECT * FROM mystream LIMIT 100
-- Count by level
SELECT level, COUNT(*) as count
FROM mystream
GROUP BY level
-- Filter by time range
SELECT * FROM mystream
WHERE p_timestamp > NOW() - INTERVAL '1 hour'
```
## Query with Regular Expressions
Examples of how to use regular expressions in Parseable queries:
### Match regular expression (Case Sensitive)
```sql
SELECT * FROM frontend where message ~ 'failing' LIMIT 9000;
```
### Match regular expression (Case Insensitive)
```sql
SELECT * FROM frontend where message ~* 'application' LIMIT 9000;
```
### Does not match regular expression
```sql
SELECT * FROM frontend where message !~ 'started' LIMIT 9000;
```
### Matches the beginning/end of the string
```sql
SELECT * FROM frontend where message ~* '^a' LIMIT 9000;
SELECT * FROM frontend where message ~ 'failing$' LIMIT 9000;
```
### Replace with REGEXP\_REPLACE
```sql
SELECT REGEXP_REPLACE(uuid,'[0-9]','*','g') FROM frontend LIMIT 9000;
```
### Split a string with SPLIT\_PART
```sql
SELECT SPLIT_PART(email,'@',1) FROM frontend where email is not null LIMIT 10;
```
### Pattern matching with LIKE
```sql
SELECT * FROM frontend where email LIKE '%test%' LIMIT 10;
```
## Case Sensitivity
Dataset column names are case sensitive. To query column names with uppercase letters, use double quotes:
```sql
SELECT * FROM dataset WHERE "Foo" = 'bar'
```
## Query Analysis
To view the detailed query execution plan, use the `EXPLAIN ANALYZE` keyword:
```sql
EXPLAIN ANALYZE SELECT * FROM frontend LIMIT 100
```
## Response Fields Information
To get the query result fields as a part of query API response, add the query parameter `fields=true`:
```
http://localhost:8000/api/v1/query?fields=true
```
## Add Charts to Dashboard
After running a query, you can visualize the results and add them directly to a dashboard.
## Filters and Group By
Use the filter panel to slice and dice your data, and group results by any field.
## Query Library
Save queries you want to reuse in the Library for quick access.
## Tips
* Use `p_timestamp` for time-based filtering
* Use `LIMIT` to prevent large result sets
* Use `GROUP BY` for aggregations
* For AI-powered query generation, see [Text to SQL](/user-guide/ai-native/text-to-sql)
# Traces (/user-guide/traces)
Traces help you understand how a request moved through your system. A log can show what one service wrote. A metric can show that latency changed. A trace shows the path of the request across services and spans, along with where time was spent.
The Traces page in Parseable is built for that flow. You choose a trace dataset, pick the time range, look at trace duration over time, narrow the data with filters, and then open a trace to inspect the waterfall.
## Traces with Parseable
Parseable stores trace data as structured records. That means you can start with a high-level trace view and still move into the fields behind each span when you need detail.
Use this page when you want to answer questions like:
* Which service produced a slow trace
* Which trace has the most spans
* Which traces contain errors
* Which span took the most time inside a trace
* Which service, host, container, or HTTP field is involved
* Which trace id from logs or another tool should be opened directly
## Typical workflow
### Dataset and time range
Start with the dataset selector and the time range picker.
Select the trace dataset from the dataset selector at the top of the page. The page updates around that dataset, including the available fields, chart, trace list, and table columns.
Use the time range picker to decide how much trace data you want to inspect. A smaller time range works well for a known issue. A wider range is useful when you want to compare behavior over time.
You can also choose the timezone from the picker, including your local timezone, so timestamps are easier to read for your team.
The top toolbar also gives you quick access to refresh, auto-refresh, saved views, and summarization. Use auto-refresh when you want to keep watching a live window of traces.
Every trace query is scoped by time. This keeps the query focused on the period you want to inspect and avoids reading more data than needed.
### Traces view
The **Traces** tab is the main place to start. It shows trace duration over time as bubble markers and lists the matching traces below the chart.
Each marker represents a trace. Its position shows when the trace happened and how long it took. Larger markers usually point to traces with more spans. Hover over a marker to see the service, span name, duration, and span count.
The trace list below the chart shows the start time, service, span name, duration, error count, and span count. Click a row when you want to open the full trace detail.
Use the controls above the chart to change what you are looking at:
* **Traces**, for regular trace exploration
* **Error spans**, when you want to focus on traces with errors
* **Spans**, when span count is the main signal
* **Sort**, when you want to move between most recent, longest first, shortest first, most spans, and least spans
* **Lookup by Trace ID**, when you already have a trace id from logs or another system
### Error spans
When you need to debug failures, switch to **Error spans**. The chart and trace list update to show traces that include errors.
Use this view when you want to find traces where something failed and then open the trace detail to see which span reported the error.
### Field groups and filters
The fields panel on the left shows the fields Parseable found in the selected trace dataset. Use it before adding filters or columns. It helps you understand what data is available.
Fields are grouped into sections such as:
* **Core**, for common trace fields
* **Error**, for error-related fields
* **Service**, for service name, namespace, version, and instance id
* **Span**, for span name, span id, parent span id, trace id, duration, and status
* **Metric** and **Event**, when the trace data includes related event fields
* **Database**, **HTTP**, **Container**, **Telemetry**, and other runtime fields when available
* **All fields**, when you want to search across everything
Click **Add filter** when you want to narrow traces to a specific service, span name, status code, trace id, host, or any other field in the dataset.
Active filters appear above the chart. You can remove one filter at a time, or clear all filters when you want to return to the broader view.
After filters are applied, use **Find in data** to search within the visible trace list. This is useful when you want to find a service, span name, id, endpoint, or message fragment inside the current page.
### Table view
Switch to the **Table** tab when you want to inspect spans as raw records instead of trace-level rows.
The Table view is useful when you want to add columns, group by a field, or scan span records directly. For example, you might add `service.name`, `span_name`, `span_duration_ns`, `span_status_code`, `http.response.status_code`, or `span_trace_id`.
Use **Expand rows** from the table toolbar when you want to see more of each span record inline without opening a separate view.
Use **Group by** when you want to compare span records by a field. Use **Find in data** when you want to search within the current table rows.
### Trace detail
Open a trace from the Traces view when you want to understand the request path.
The detail view shows the trace start time, total duration, and span count at the top. Below that, the waterfall shows how spans relate to each other and where time was spent.
Use the waterfall to answer questions like:
* Which span started the trace
* Which child spans ran under it
* Which span took the longest
* Where the trace has many repeated operations
* Whether the time is spent in one service or spread across many spans
You can expand and collapse span groups, search spans by name, and use the minimap at the top to understand the full trace timeline.
### Span detail
Click a span in the waterfall to open the span detail panel.
The span detail panel shows the selected span's start time, duration, span id, fields, and events. Use it when you need the exact attributes behind a span.
The **Fields** tab is useful for service metadata, span ids, trace ids, timing fields, status fields, code fields, HTTP attributes, and resource attributes. These fields help you connect the trace back to logs, services, hosts, containers, and code paths.
The **Events** tab is useful when the span includes event records. This can help explain what happened inside the span, not only how long the span took.
### Saved views, sharing, and summaries
When you have a useful view, save it. Saved views help you come back to the same dataset, filters, fields, and time range without setting everything again.
Use the share control when you want another teammate to open the same view. Use export or SQL when you need to continue the investigation outside the current Traces page.
The **Summarize** button can generate a report for the selected trace data. Use it when you want a quick read of the current traces before looking deeper.
## A simple way to use the page
Most trace investigations follow the same loop:
1. Pick the dataset and time range.
2. Look at trace duration over time.
3. Use filters, error spans, sorting, or Trace ID lookup to narrow the data.
4. Open a trace from the list.
5. Use the waterfall to understand where time was spent.
6. Click spans to inspect fields and events.
7. Save, share, summarize, or move into SQL when needed.
# Create a dataset (/api/v1/createLogStream)
{/* Define variables for path parameters */}
export const stream_name = "stream_name";
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Create a dataset in Parseable using the dataset name in the path.
This endpoint is used for stream creation before writing JSON events directly to `/api/v1/logstream/{stream_name}`.
# Delete a dataset (/api/v1/deleteLogStream)
{/* Define variables for path parameters */}
export const stream_name = "stream_name";
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Delete an existing dataset.
When a dataset is deleted, all associated data is permanently removed.
# Ingest JSON events with headers (/api/v1/ingestLogsWithHeaders)
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Ingest JSON events and route them with request headers.
Use `X-P-Stream` to specify the destination dataset.
# Ingest OTLP logs (/api/v1/ingestOtlpLogs)
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Ingest OpenTelemetry logs through Parseable's native OTLP logs endpoint.
Use `X-P-Stream` to choose the destination dataset. Parseable identifies the OTLP signal type from the `/v1/logs` endpoint.
# Ingest OTLP metrics (/api/v1/ingestOtlpMetrics)
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Ingest OpenTelemetry metrics through Parseable's native OTLP metrics endpoint.
Use `X-P-Stream` to choose the destination dataset. Parseable identifies the OTLP signal type from the `/v1/metrics` endpoint.
# Ingest OTLP traces (/api/v1/ingestOtlpTraces)
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Ingest OpenTelemetry traces through Parseable's native OTLP traces endpoint.
Use `X-P-Stream` to choose the destination dataset. Parseable identifies the OTLP signal type from the `/v1/traces` endpoint.
# Ingest Prometheus remote write metrics (/api/v1/prometheusRemoteWrite)
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Ingest metrics using the Prometheus Remote Write protocol.
This endpoint accepts snappy-compressed protobuf payloads and routes the converted metrics into the target dataset.
# Query a dataset with SQL (/api/v1/queryLogStream)
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Run a SQL query in Parseable.
The dataset is selected inside the SQL statement itself, typically in the `FROM` clause.
# Send JSON events to a dataset (/api/v1/sendLogsToStream)
{/* Define variables for path parameters */}
export const stream_name = "stream_name";
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
Send JSON events directly to an existing dataset.
The destination dataset is taken from the `{stream_name}` path parameter, so no `X-P-Stream` header is needed.
# ArgoCD (/ingest-data/cicd/argocd)
Collect ArgoCD GitOps events and application logs in Parseable.
## Overview
Integrate ArgoCD with Parseable to:
* **Deployment Events** - Track application sync events
* **GitOps Audit** - Monitor configuration changes
* **Health Status** - Track application health over time
* **Troubleshooting** - Debug deployment issues
## Prerequisites
* ArgoCD installed in Kubernetes
* Parseable instance accessible from cluster
* ArgoCD notifications configured
## Method 1: ArgoCD Notifications
Use ArgoCD Notifications to send events to Parseable.
### Install Notifications Controller
```bash
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/release-1.0/manifests/install.yaml
```
### Configure Webhook Service
Create notification configuration:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
data:
service.webhook.parseable: |
url: http://parseable:8000/api/v1/ingest
headers:
- name: Authorization
value: Basic YWRtaW46YWRtaW4=
- name: X-P-Stream
value: argocd-events
- name: Content-Type
value: application/json
template.app-sync-status: |
webhook:
parseable:
method: POST
body: |
[{
"timestamp": "{{.app.status.operationState.finishedAt}}",
"event": "sync",
"app_name": "{{.app.metadata.name}}",
"project": "{{.app.spec.project}}",
"repo": "{{.app.spec.source.repoURL}}",
"revision": "{{.app.status.sync.revision}}",
"sync_status": "{{.app.status.sync.status}}",
"health_status": "{{.app.status.health.status}}",
"message": "{{.app.status.operationState.message}}"
}]
template.app-health-degraded: |
webhook:
parseable:
method: POST
body: |
[{
"timestamp": "{{now | date \"2006-01-02T15:04:05Z\"}}",
"event": "health_degraded",
"app_name": "{{.app.metadata.name}}",
"project": "{{.app.spec.project}}",
"health_status": "{{.app.status.health.status}}",
"sync_status": "{{.app.status.sync.status}}"
}]
trigger.on-sync-succeeded: |
- when: app.status.operationState.phase in ['Succeeded']
send: [app-sync-status]
trigger.on-sync-failed: |
- when: app.status.operationState.phase in ['Failed', 'Error']
send: [app-sync-status]
trigger.on-health-degraded: |
- when: app.status.health.status == 'Degraded'
send: [app-health-degraded]
```
### Subscribe Applications
Add annotation to applications:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
annotations:
notifications.argoproj.io/subscribe.on-sync-succeeded.parseable: ""
notifications.argoproj.io/subscribe.on-sync-failed.parseable: ""
notifications.argoproj.io/subscribe.on-health-degraded.parseable: ""
```
## Method 2: Event Collector
Deploy a collector to watch ArgoCD events.
### Collector Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-event-collector
namespace: argocd
spec:
replicas: 1
selector:
matchLabels:
app: argocd-event-collector
template:
spec:
serviceAccountName: argocd-event-collector
containers:
- name: collector
image: bitnami/kubectl:latest
command:
- /bin/bash
- -c
- |
while true; do
kubectl get events -n argocd --watch -o json | while read event; do
curl -s -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: argocd-events" \
-H "Content-Type: application/json" \
-d "[${event}]"
done
done
env:
- name: PARSEABLE_URL
value: "http://parseable:8000"
- name: PARSEABLE_AUTH
valueFrom:
secretKeyRef:
name: parseable-credentials
key: auth
```
## Method 3: ArgoCD Logs
Collect ArgoCD component logs using Fluent Bit.
### Fluent Bit Configuration
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: argocd
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level info
[INPUT]
Name tail
Path /var/log/containers/argocd-*.log
Parser cri
Tag argocd.*
Refresh_Interval 5
[FILTER]
Name kubernetes
Match argocd.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
[OUTPUT]
Name http
Match *
Host parseable
Port 8000
URI /api/v1/ingest
Format json
Header Authorization Basic YWRtaW46YWRtaW4=
Header X-P-Stream argocd-logs
```
## Querying ArgoCD Events
```sql
-- Recent sync events
SELECT timestamp, app_name, sync_status, health_status, revision
FROM "argocd-events"
WHERE event = 'sync'
ORDER BY timestamp DESC
LIMIT 100
-- Failed syncs
SELECT timestamp, app_name, project, message
FROM "argocd-events"
WHERE sync_status = 'Failed' OR sync_status = 'Error'
ORDER BY timestamp DESC
-- Application health history
SELECT
app_name,
health_status,
COUNT(*) as count
FROM "argocd-events"
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY app_name, health_status
ORDER BY app_name, count DESC
-- Deployment frequency
SELECT
DATE_TRUNC('day', timestamp) as day,
app_name,
COUNT(*) as deployments
FROM "argocd-events"
WHERE event = 'sync' AND sync_status = 'Succeeded'
GROUP BY day, app_name
ORDER BY day DESC
```
## Best Practices
1. **Track All Events** - Sync, health, and errors
2. **Include Revision** - Track which commits deployed
3. **Monitor Health** - Alert on degraded applications
4. **Audit Changes** - Log who triggered syncs
## Troubleshooting
### Notifications Not Sending
1. Check notification controller logs
2. Verify webhook URL is accessible
3. Check ConfigMap syntax
4. Verify application annotations
### Missing Events
1. Check trigger conditions
2. Verify template syntax
3. Check ArgoCD logs
## Next Steps
* Set up [alerts](/user-guide/alerting) for failed deployments
* Create [dashboards](/user-guide/dashboards) for GitOps metrics
* Configure [GitHub Actions](/ingest-data/cicd/github-actions) for CI
# CircleCI (/ingest-data/cicd/circleci)
Collect and analyze CircleCI pipeline logs in Parseable for CI/CD observability.
## Overview
Integrate CircleCI with Parseable to:
* **Pipeline Logs** - Collect all job and workflow logs
* **Debug Failures** - Quickly find and analyze failed builds
* **Track Performance** - Monitor build times and trends
* **Audit Pipelines** - Maintain compliance with log retention
## Prerequisites
* CircleCI project
* Parseable instance accessible from CircleCI
* CircleCI environment variables configured
## Direct Log Shipping
Send logs directly from your CircleCI config.
### Config.yml
```yaml
version: 2.1
executors:
default:
docker:
- image: cimg/base:stable
commands:
send_to_parseable:
parameters:
status:
type: string
message:
type: string
default: ""
steps:
- run:
name: Send to Parseable
command: |
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: circleci-pipelines" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CIRCLE_PROJECT_REPONAME}\",
\"workflow\": \"${CIRCLE_WORKFLOW_ID}\",
\"job\": \"${CIRCLE_JOB}\",
\"build_num\": ${CIRCLE_BUILD_NUM},
\"branch\": \"${CIRCLE_BRANCH}\",
\"sha\": \"${CIRCLE_SHA1}\",
\"username\": \"${CIRCLE_USERNAME}\",
\"status\": \"<< parameters.status >>\",
\"message\": \"<< parameters.message >>\"
}]"
when: always
jobs:
build:
executor: default
steps:
- checkout
- send_to_parseable:
status: "started"
message: "Build started"
- run:
name: Build
command: |
echo "Building..."
npm ci
npm run build
- send_to_parseable:
status: "completed"
message: "Build completed"
test:
executor: default
steps:
- checkout
- run:
name: Test
command: npm test
- send_to_parseable:
status: "completed"
message: "Tests passed"
workflows:
build-and-test:
jobs:
- build
- test:
requires:
- build
```
### With Status Detection
```yaml
commands:
send_to_parseable_with_status:
steps:
- run:
name: Send to Parseable
command: |
if [ "${CIRCLE_JOB_STATUS}" = "success" ]; then
STATUS="success"
else
STATUS="failure"
fi
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: circleci-pipelines" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CIRCLE_PROJECT_REPONAME}\",
\"job\": \"${CIRCLE_JOB}\",
\"build_num\": ${CIRCLE_BUILD_NUM},
\"status\": \"${STATUS}\"
}]"
when: always
```
## Orb Integration
Create a reusable orb for Parseable logging.
### Orb Definition
```yaml
# .circleci/orbs/parseable.yml
version: 2.1
description: Send logs to Parseable
commands:
log:
parameters:
status:
type: string
message:
type: string
default: ""
dataset:
type: string
default: "circleci-pipelines"
steps:
- run:
name: Log to Parseable
command: |
curl -s -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: << parameters.dataset >>" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CIRCLE_PROJECT_REPONAME}\",
\"workflow_id\": \"${CIRCLE_WORKFLOW_ID}\",
\"job\": \"${CIRCLE_JOB}\",
\"build_num\": ${CIRCLE_BUILD_NUM},
\"branch\": \"${CIRCLE_BRANCH}\",
\"sha\": \"${CIRCLE_SHA1}\",
\"status\": \"<< parameters.status >>\",
\"message\": \"<< parameters.message >>\"
}]" || true
when: always
```
### Using the Orb
```yaml
version: 2.1
orbs:
parseable: your-org/parseable@1.0.0
jobs:
build:
docker:
- image: cimg/node:18.0
steps:
- checkout
- parseable/log:
status: "started"
- run: npm ci && npm run build
- parseable/log:
status: "completed"
```
## Webhook Integration
Use CircleCI webhooks for workflow events.
### Webhook Receiver
```javascript
const express = require('express');
const axios = require('axios');
const crypto = require('crypto');
const app = express();
app.use(express.json());
const CIRCLECI_SECRET = process.env.CIRCLECI_WEBHOOK_SECRET;
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;
app.post('/webhook', async (req, res) => {
// Verify signature
const signature = req.headers['circleci-signature'];
// Verification logic here
const event = req.body;
const logEntry = {
timestamp: new Date().toISOString(),
event_type: event.type,
project: event.project?.name,
workflow_id: event.workflow?.id,
workflow_name: event.workflow?.name,
job_name: event.job?.name,
status: event.job?.status || event.workflow?.status,
branch: event.pipeline?.vcs?.branch,
commit: event.pipeline?.vcs?.revision
};
try {
await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, [logEntry], {
headers: {
'Authorization': `Basic ${PARSEABLE_AUTH}`,
'X-P-Stream': 'circleci-webhooks',
'Content-Type': 'application/json'
}
});
} catch (error) {
console.error('Error:', error);
}
res.status(200).json({ status: 'received' });
});
app.listen(3000);
```
## Environment Variables
Configure in CircleCI project settings:
| Variable | Description |
| ---------------- | ---------------------------------- |
| `PARSEABLE_URL` | Parseable instance URL |
| `PARSEABLE_AUTH` | Base64 encoded `username:password` |
## Querying CircleCI Logs
```sql
-- Recent builds
SELECT timestamp, project, job, status, branch
FROM "circleci-pipelines"
ORDER BY timestamp DESC
LIMIT 100
-- Failed builds
SELECT timestamp, project, job, branch, message
FROM "circleci-pipelines"
WHERE status = 'failure'
ORDER BY timestamp DESC
-- Build success rate
SELECT
project,
COUNT(*) as total,
SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successful,
ROUND(SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)::float / COUNT(*) * 100, 2) as success_rate
FROM "circleci-pipelines"
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY project
```
## Best Practices
1. **Use Orbs** - Centralize logging logic
2. **Log All Jobs** - Track every job in workflow
3. **Include Context** - Add branch, commit, and user info
4. **Handle Failures** - Use `when: always` for logging steps
## Next Steps
* Set up [alerts](/user-guide/alerting) for failed builds
* Create [dashboards](/user-guide/dashboards) for CI/CD metrics
* Configure [GitHub Actions](/ingest-data/cicd/github-actions) for comparison
# GitHub Actions (/ingest-data/cicd/github-actions)
Collect and analyze GitHub Actions workflow logs in Parseable for CI/CD observability.
## Overview
Integrate GitHub Actions with Parseable to:
* **Centralize CI/CD Logs** - Collect all workflow logs in one place
* **Debug Failures** - Quickly find and analyze failed jobs
* **Track Performance** - Monitor build times and trends
* **Audit Pipelines** - Maintain compliance with log retention
## Prerequisites
* GitHub repository with Actions enabled
* Parseable instance accessible from GitHub Actions runners
* GitHub Personal Access Token (for API access)
## Method 1: Direct Log Shipping
Send logs directly from your workflow to Parseable.
### Workflow Configuration
Add a step to send logs to Parseable:
```yaml
name: CI Pipeline
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build
id: build
run: |
echo "Starting build..."
# Your build commands here
npm ci
npm run build
echo "Build completed successfully"
- name: Send logs to Parseable
if: always()
run: |
curl -X POST "${{ secrets.PARSEABLE_URL }}/api/v1/ingest" \
-H "Authorization: Basic ${{ secrets.PARSEABLE_AUTH }}" \
-H "X-P-Stream: github-actions" \
-H "Content-Type: application/json" \
-d '[{
"timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'",
"repository": "${{ github.repository }}",
"workflow": "${{ github.workflow }}",
"job": "${{ github.job }}",
"run_id": "${{ github.run_id }}",
"run_number": "${{ github.run_number }}",
"actor": "${{ github.actor }}",
"event": "${{ github.event_name }}",
"ref": "${{ github.ref }}",
"sha": "${{ github.sha }}",
"status": "${{ job.status }}",
"conclusion": "${{ steps.build.conclusion }}"
}]'
```
### Reusable Action
Create a reusable action for consistent logging:
```yaml
# .github/actions/log-to-parseable/action.yml
name: Log to Parseable
description: Send workflow logs to Parseable
inputs:
parseable_url:
description: Parseable instance URL
required: true
parseable_auth:
description: Base64 encoded credentials
required: true
dataset:
description: Parseable dataset name
default: github-actions
status:
description: Job status
required: true
message:
description: Log message
default: ''
runs:
using: composite
steps:
- name: Send to Parseable
shell: bash
run: |
curl -X POST "${{ inputs.parseable_url }}/api/v1/ingest" \
-H "Authorization: Basic ${{ inputs.parseable_auth }}" \
-H "X-P-Stream: ${{ inputs.dataset }}" \
-H "Content-Type: application/json" \
-d '[{
"timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'",
"repository": "${{ github.repository }}",
"workflow": "${{ github.workflow }}",
"job": "${{ github.job }}",
"run_id": "${{ github.run_id }}",
"run_number": "${{ github.run_number }}",
"actor": "${{ github.actor }}",
"event": "${{ github.event_name }}",
"ref": "${{ github.ref }}",
"sha": "${{ github.sha }}",
"status": "${{ inputs.status }}",
"message": "${{ inputs.message }}"
}]'
```
Use the action in your workflow:
```yaml
- name: Log success
if: success()
uses: ./.github/actions/log-to-parseable
with:
parseable_url: ${{ secrets.PARSEABLE_URL }}
parseable_auth: ${{ secrets.PARSEABLE_AUTH }}
status: success
message: Build completed successfully
- name: Log failure
if: failure()
uses: ./.github/actions/log-to-parseable
with:
parseable_url: ${{ secrets.PARSEABLE_URL }}
parseable_auth: ${{ secrets.PARSEABLE_AUTH }}
status: failure
message: Build failed
```
## Method 2: OpenTelemetry Tracing
Use OpenTelemetry to trace your CI/CD pipelines.
### Workflow with OTEL
```yaml
name: CI Pipeline with Tracing
on:
push:
branches: [main]
env:
OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.PARSEABLE_OTEL_ENDPOINT }}
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${{ secrets.PARSEABLE_API_KEY }},X-P-Stream=github-actions-traces"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup OTEL CLI
run: |
curl -L https://github.com/equinix-labs/otel-cli/releases/latest/download/otel-cli-linux-amd64 -o otel-cli
chmod +x otel-cli
sudo mv otel-cli /usr/local/bin/
- name: Build with tracing
run: |
otel-cli exec \
--name "build" \
--service "github-actions" \
--attrs "repository=${{ github.repository }},workflow=${{ github.workflow }}" \
-- npm run build
```
## Method 3: Webhook Integration
Use GitHub webhooks to capture workflow events.
### Webhook Receiver
Create a webhook receiver that forwards events to Parseable:
```javascript
// github-webhook-to-parseable.js
const express = require('express');
const crypto = require('crypto');
const axios = require('axios');
const app = express();
app.use(express.json());
const WEBHOOK_SECRET = process.env.GITHUB_WEBHOOK_SECRET;
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;
function verifySignature(payload, signature) {
const hmac = crypto.createHmac('sha256', WEBHOOK_SECRET);
const digest = 'sha256=' + hmac.update(JSON.stringify(payload)).digest('hex');
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(digest));
}
app.post('/webhook', async (req, res) => {
const signature = req.headers['x-hub-signature-256'];
if (!verifySignature(req.body, signature)) {
return res.status(401).json({ error: 'Invalid signature' });
}
const event = req.headers['x-github-event'];
const payload = req.body;
// Only process workflow events
if (event === 'workflow_run' || event === 'workflow_job') {
const logEntry = {
timestamp: new Date().toISOString(),
event_type: event,
action: payload.action,
repository: payload.repository?.full_name,
workflow: payload.workflow?.name || payload.workflow_run?.name,
run_id: payload.workflow_run?.id || payload.workflow_job?.run_id,
conclusion: payload.workflow_run?.conclusion || payload.workflow_job?.conclusion,
actor: payload.sender?.login,
url: payload.workflow_run?.html_url || payload.workflow_job?.html_url
};
try {
await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, [logEntry], {
headers: {
'Authorization': `Basic ${PARSEABLE_AUTH}`,
'X-P-Stream': 'github-webhooks',
'Content-Type': 'application/json'
}
});
} catch (error) {
console.error('Error sending to Parseable:', error);
}
}
res.status(200).json({ status: 'received' });
});
app.listen(3000, () => {
console.log('GitHub webhook receiver listening on port 3000');
});
```
### Configure GitHub Webhook
1. Go to your repository **Settings** → **Webhooks**
2. Click **Add webhook**
3. Set Payload URL to your webhook receiver
4. Set Content type to `application/json`
5. Set Secret to your webhook secret
6. Select events: **Workflow runs** and **Workflow jobs**
## Querying GitHub Actions Logs
Query your CI/CD logs in Parseable:
```sql
-- Get recent workflow runs
SELECT timestamp, repository, workflow, status, actor
FROM "github-actions"
ORDER BY timestamp DESC
LIMIT 100
-- Find failed builds
SELECT timestamp, repository, workflow, run_id, message
FROM "github-actions"
WHERE status = 'failure'
ORDER BY timestamp DESC
-- Build success rate by repository
SELECT
repository,
COUNT(*) as total_runs,
SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successful,
ROUND(SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)::float / COUNT(*) * 100, 2) as success_rate
FROM "github-actions"
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY repository
ORDER BY success_rate ASC
```
## GitHub Secrets Configuration
Add these secrets to your repository:
| Secret | Description |
| ------------------------- | ---------------------------------- |
| `PARSEABLE_URL` | Your Parseable instance URL |
| `PARSEABLE_AUTH` | Base64 encoded `username:password` |
| `PARSEABLE_API_KEY` | Parseable API key for OTLP tracing |
| `PARSEABLE_OTEL_ENDPOINT` | OTEL endpoint (for tracing) |
## Best Practices
1. **Use Secrets** - Never hardcode credentials in workflows
2. **Log on Failure** - Always capture failure information
3. **Include Context** - Add repository, workflow, and run details
4. **Use Consistent Streams** - Organize logs by dataset
5. **Set Retention** - Configure appropriate log retention
## Troubleshooting
### Logs Not Appearing
1. Verify Parseable URL is accessible from GitHub runners
2. Check authentication credentials are correct
3. Verify the dataset exists or auto-creation is enabled
4. Check workflow logs for curl errors
### Authentication Failures
1. Verify Base64 encoding of credentials
2. Check the Parseable user has ingest permissions
3. Verify secrets are properly configured
## Next Steps
* Set up [alerts](/user-guide/alerting) for failed builds
* Create [dashboards](/user-guide/dashboards) for CI/CD metrics
* Configure [Jenkins](/ingest-data/cicd/jenkins) for additional pipelines
# GitLab CI (/ingest-data/cicd/gitlab-ci)
Collect and analyze GitLab CI/CD pipeline logs in Parseable for comprehensive observability.
## Overview
Integrate GitLab CI with Parseable to:
* **Centralize Pipeline Logs** - Collect all CI/CD logs in one place
* **Debug Failures** - Quickly find and analyze failed jobs
* **Track Performance** - Monitor pipeline times and trends
* **Audit Pipelines** - Maintain compliance with log retention
## Prerequisites
* GitLab repository with CI/CD enabled
* Parseable instance accessible from GitLab runners
* GitLab CI/CD variables configured
## Method 1: Direct Log Shipping
Send logs directly from your `.gitlab-ci.yml`.
### Basic Configuration
```yaml
variables:
PARSEABLE_URL: ${PARSEABLE_URL}
PARSEABLE_AUTH: ${PARSEABLE_AUTH}
stages:
- build
- test
- deploy
.send_to_parseable: &send_to_parseable
- |
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: gitlab-pipelines" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CI_PROJECT_PATH}\",
\"pipeline_id\": ${CI_PIPELINE_ID},
\"job_id\": ${CI_JOB_ID},
\"job_name\": \"${CI_JOB_NAME}\",
\"stage\": \"${CI_JOB_STAGE}\",
\"ref\": \"${CI_COMMIT_REF_NAME}\",
\"sha\": \"${CI_COMMIT_SHA}\",
\"author\": \"${GITLAB_USER_LOGIN}\",
\"status\": \"${1}\",
\"message\": \"${2}\"
}]"
build:
stage: build
script:
- echo "Building..."
- npm ci
- npm run build
after_script:
- |
if [ "$CI_JOB_STATUS" == "success" ]; then
STATUS="success"
MESSAGE="Build completed successfully"
else
STATUS="failure"
MESSAGE="Build failed"
fi
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: gitlab-pipelines" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CI_PROJECT_PATH}\",
\"pipeline_id\": ${CI_PIPELINE_ID},
\"job_id\": ${CI_JOB_ID},
\"job_name\": \"${CI_JOB_NAME}\",
\"stage\": \"${CI_JOB_STAGE}\",
\"ref\": \"${CI_COMMIT_REF_NAME}\",
\"sha\": \"${CI_COMMIT_SHA}\",
\"author\": \"${GITLAB_USER_LOGIN}\",
\"status\": \"${STATUS}\",
\"message\": \"${MESSAGE}\"
}]"
test:
stage: test
script:
- npm test
after_script:
- |
STATUS="${CI_JOB_STATUS}"
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: gitlab-pipelines" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CI_PROJECT_PATH}\",
\"pipeline_id\": ${CI_PIPELINE_ID},
\"job_id\": ${CI_JOB_ID},
\"job_name\": \"${CI_JOB_NAME}\",
\"stage\": \"${CI_JOB_STAGE}\",
\"status\": \"${STATUS}\"
}]"
```
### Reusable Template
Create a reusable template for consistent logging:
```yaml
# templates/parseable.yml
.parseable_logging:
after_script:
- |
STATUS="${CI_JOB_STATUS:-unknown}"
DURATION="${CI_JOB_DURATION:-0}"
curl -s -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: gitlab-pipelines" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"project\": \"${CI_PROJECT_PATH}\",
\"project_id\": ${CI_PROJECT_ID},
\"pipeline_id\": ${CI_PIPELINE_ID},
\"pipeline_url\": \"${CI_PIPELINE_URL}\",
\"job_id\": ${CI_JOB_ID},
\"job_name\": \"${CI_JOB_NAME}\",
\"job_url\": \"${CI_JOB_URL}\",
\"stage\": \"${CI_JOB_STAGE}\",
\"ref\": \"${CI_COMMIT_REF_NAME}\",
\"sha\": \"${CI_COMMIT_SHA}\",
\"short_sha\": \"${CI_COMMIT_SHORT_SHA}\",
\"author\": \"${GITLAB_USER_LOGIN}\",
\"author_email\": \"${GITLAB_USER_EMAIL}\",
\"status\": \"${STATUS}\",
\"duration_seconds\": ${DURATION},
\"runner\": \"${CI_RUNNER_DESCRIPTION}\",
\"environment\": \"${CI_ENVIRONMENT_NAME:-none}\"
}]" || true
```
Use the template:
```yaml
include:
- local: templates/parseable.yml
build:
extends: .parseable_logging
stage: build
script:
- npm ci
- npm run build
test:
extends: .parseable_logging
stage: test
script:
- npm test
```
## Method 2: GitLab Webhooks
Use GitLab webhooks to capture pipeline events.
### Webhook Receiver
```javascript
// gitlab-webhook-to-parseable.js
const express = require('express');
const axios = require('axios');
const app = express();
app.use(express.json());
const GITLAB_TOKEN = process.env.GITLAB_WEBHOOK_TOKEN;
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;
app.post('/webhook', async (req, res) => {
// Verify token
if (req.headers['x-gitlab-token'] !== GITLAB_TOKEN) {
return res.status(401).json({ error: 'Invalid token' });
}
const event = req.headers['x-gitlab-event'];
const payload = req.body;
let logEntry = null;
if (event === 'Pipeline Hook') {
logEntry = {
timestamp: new Date().toISOString(),
event_type: 'pipeline',
project: payload.project?.path_with_namespace,
pipeline_id: payload.object_attributes?.id,
ref: payload.object_attributes?.ref,
sha: payload.object_attributes?.sha,
status: payload.object_attributes?.status,
duration: payload.object_attributes?.duration,
created_at: payload.object_attributes?.created_at,
finished_at: payload.object_attributes?.finished_at,
user: payload.user?.username
};
} else if (event === 'Job Hook') {
logEntry = {
timestamp: new Date().toISOString(),
event_type: 'job',
project: payload.project?.path_with_namespace,
pipeline_id: payload.pipeline_id,
job_id: payload.build_id,
job_name: payload.build_name,
stage: payload.build_stage,
status: payload.build_status,
duration: payload.build_duration,
runner: payload.runner?.description,
user: payload.user?.username
};
}
if (logEntry) {
try {
await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, [logEntry], {
headers: {
'Authorization': `Basic ${PARSEABLE_AUTH}`,
'X-P-Stream': 'gitlab-webhooks',
'Content-Type': 'application/json'
}
});
} catch (error) {
console.error('Error sending to Parseable:', error);
}
}
res.status(200).json({ status: 'received' });
});
app.listen(3000);
```
### Configure GitLab Webhook
1. Go to your project **Settings** → **Webhooks**
2. Enter your webhook URL
3. Enter a Secret Token
4. Select events: **Pipeline events** and **Job events**
5. Click **Add webhook**
## Method 3: OpenTelemetry Tracing
Use OpenTelemetry for distributed tracing of pipelines.
```yaml
variables:
OTEL_EXPORTER_OTLP_ENDPOINT: ${PARSEABLE_OTEL_ENDPOINT}
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${PARSEABLE_API_KEY},X-P-Stream=gitlab-traces"
OTEL_SERVICE_NAME: gitlab-ci
build:
stage: build
image: node:18
before_script:
- npm install -g @opentelemetry/auto-instrumentations-node
script:
- node --require @opentelemetry/auto-instrumentations-node npm run build
```
## GitLab CI/CD Variables
Configure these variables in GitLab:
| Variable | Type | Description |
| ------------------------- | ----------------- | ---------------------------------- |
| `PARSEABLE_URL` | Variable | Parseable instance URL |
| `PARSEABLE_AUTH` | Variable (masked) | Base64 encoded `username:password` |
| `PARSEABLE_API_KEY` | Variable (masked) | Parseable API key for OTLP tracing |
| `PARSEABLE_OTEL_ENDPOINT` | Variable | OTEL endpoint (for tracing) |
## Querying GitLab Logs
Query your GitLab CI/CD logs in Parseable:
```sql
-- Get recent pipeline runs
SELECT timestamp, project, pipeline_id, status, duration_seconds
FROM "gitlab-pipelines"
ORDER BY timestamp DESC
LIMIT 100
-- Find failed jobs
SELECT timestamp, project, job_name, stage, status, job_url
FROM "gitlab-pipelines"
WHERE status = 'failed'
ORDER BY timestamp DESC
-- Pipeline success rate by project
SELECT
project,
COUNT(*) as total_pipelines,
SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successful,
ROUND(SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)::float / COUNT(*) * 100, 2) as success_rate
FROM "gitlab-pipelines"
WHERE timestamp > NOW() - INTERVAL '7 days'
AND job_name IS NULL -- Pipeline-level events only
GROUP BY project
ORDER BY success_rate ASC
-- Average job duration by stage
SELECT
stage,
AVG(duration_seconds) as avg_duration,
MAX(duration_seconds) as max_duration,
COUNT(*) as job_count
FROM "gitlab-pipelines"
WHERE timestamp > NOW() - INTERVAL '7 days'
AND duration_seconds IS NOT NULL
GROUP BY stage
ORDER BY avg_duration DESC
```
## Best Practices
1. **Use Templates** - Centralize logging configuration
2. **Include Context** - Add project, pipeline, and job details
3. **Log All Stages** - Track each stage separately
4. **Handle Failures** - Use `|| true` to prevent logging failures from breaking builds
5. **Mask Secrets** - Always mask sensitive variables
## Troubleshooting
### Logs Not Appearing
1. Verify Parseable URL is accessible from GitLab runners
2. Check CI/CD variables are properly configured
3. Verify the dataset exists or auto-creation is enabled
4. Check job logs for curl errors
### Authentication Failures
1. Verify Base64 encoding of credentials
2. Check the Parseable user has ingest permissions
3. Ensure variables are not protected when running on unprotected branches
## Next Steps
* Set up [alerts](/user-guide/alerting) for failed pipelines
* Create [dashboards](/user-guide/dashboards) for CI/CD metrics
* Configure [GitHub Actions](/ingest-data/cicd/github-actions) for multi-platform CI
# Jenkins (/ingest-data/cicd/jenkins)
Collect and analyze Jenkins build logs in Parseable for comprehensive CI/CD observability.
## Overview
Integrate Jenkins with Parseable to:
* **Centralize Build Logs** - Collect all Jenkins logs in one place
* **Debug Failures** - Quickly find and analyze failed builds
* **Track Performance** - Monitor build times and trends
* **Audit Pipelines** - Maintain compliance with log retention
## Prerequisites
* Jenkins instance with Pipeline support
* Parseable instance accessible from Jenkins
* Jenkins HTTP Request plugin (optional)
## Method 1: Pipeline Script
Send logs directly from your Jenkinsfile.
### Declarative Pipeline
```groovy
pipeline {
agent any
environment {
PARSEABLE_URL = credentials('parseable-url')
PARSEABLE_AUTH = credentials('parseable-auth')
}
stages {
stage('Build') {
steps {
script {
sendToParseable('started', 'Build started')
}
sh 'npm ci'
sh 'npm run build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
}
}
post {
success {
script {
sendToParseable('success', 'Build completed successfully')
}
}
failure {
script {
sendToParseable('failure', 'Build failed')
}
}
always {
script {
sendToParseable(currentBuild.result ?: 'unknown', 'Build finished')
}
}
}
}
def sendToParseable(String status, String message) {
def payload = """[{
"timestamp": "${new Date().format("yyyy-MM-dd'T'HH:mm:ss'Z'", TimeZone.getTimeZone('UTC'))}",
"job_name": "${env.JOB_NAME}",
"build_number": ${env.BUILD_NUMBER},
"build_url": "${env.BUILD_URL}",
"node": "${env.NODE_NAME}",
"status": "${status}",
"message": "${message}",
"duration": ${currentBuild.duration},
"user": "${currentBuild.getBuildCauses()[0]?.userId ?: 'system'}"
}]"""
httpRequest(
url: "${PARSEABLE_URL}/api/v1/ingest",
httpMode: 'POST',
contentType: 'APPLICATION_JSON',
customHeaders: [
[name: 'Authorization', value: "Basic ${PARSEABLE_AUTH}"],
[name: 'X-P-Stream', value: 'jenkins-builds']
],
requestBody: payload,
validResponseCodes: '200:299'
)
}
```
### Scripted Pipeline
```groovy
node {
def parseableUrl = env.PARSEABLE_URL
def parseableAuth = env.PARSEABLE_AUTH
try {
stage('Build') {
sendLog('started', 'Build started')
sh 'npm ci && npm run build'
}
stage('Test') {
sh 'npm test'
}
sendLog('success', 'Build completed successfully')
} catch (Exception e) {
sendLog('failure', "Build failed: ${e.message}")
throw e
}
}
def sendLog(String status, String message) {
def payload = [
[
timestamp: new Date().format("yyyy-MM-dd'T'HH:mm:ss'Z'"),
job_name: env.JOB_NAME,
build_number: env.BUILD_NUMBER as Integer,
build_url: env.BUILD_URL,
status: status,
message: message
]
]
httpRequest(
url: "${env.PARSEABLE_URL}/api/v1/ingest",
httpMode: 'POST',
contentType: 'APPLICATION_JSON',
customHeaders: [
[name: 'Authorization', value: "Basic ${env.PARSEABLE_AUTH}"],
[name: 'X-P-Stream', value: 'jenkins-builds']
],
requestBody: groovy.json.JsonOutput.toJson(payload)
)
}
```
## Method 2: Fluent Bit Sidecar
Collect Jenkins logs using Fluent Bit.
### Fluent Bit Configuration
```yaml
service:
flush: 5
log_level: info
pipeline:
inputs:
- name: tail
path: /var/jenkins_home/jobs/*/builds/*/log
tag: jenkins.build
parser: jenkins
refresh_interval: 5
parsers:
- name: jenkins
format: regex
regex: ^(?\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+Z)\s+(?.*)$
time_key: timestamp
time_format: "%Y-%m-%dT%H:%M:%S.%LZ"
outputs:
- name: http
match: jenkins.*
host: parseable
port: 8000
uri: /api/v1/ingest
format: json
header: Authorization Basic YWRtaW46YWRtaW4=
header: X-P-Stream jenkins-logs
```
### Docker Compose
```yaml
version: '3.8'
services:
jenkins:
image: jenkins/jenkins:lts
volumes:
- jenkins_home:/var/jenkins_home
ports:
- "8080:8080"
fluent-bit:
image: fluent/fluent-bit:latest
volumes:
- jenkins_home:/var/jenkins_home:ro
- ./fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml
depends_on:
- jenkins
- parseable
volumes:
jenkins_home:
```
## Method 3: Logstash Integration
Use Logstash to collect and forward Jenkins logs.
### Logstash Configuration
```ruby
input {
file {
path => "/var/jenkins_home/jobs/*/builds/*/log"
start_position => "beginning"
sincedb_path => "/var/logstash/sincedb"
codec => multiline {
pattern => "^\d{4}-\d{2}-\d{2}"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => {
"path" => "/var/jenkins_home/jobs/(?[^/]+)/builds/(?\d+)/log"
}
}
mutate {
add_field => {
"source" => "jenkins"
}
}
}
output {
http {
url => "http://parseable:8000/api/v1/ingest"
http_method => "post"
format => "json"
headers => {
"Authorization" => "Basic YWRtaW46YWRtaW4="
"X-P-Stream" => "jenkins-logs"
}
}
}
```
## Method 4: Jenkins Webhook
Use Jenkins webhooks to send build notifications.
### Shared Library
Create a shared library for consistent logging:
```groovy
// vars/parseableNotify.groovy
def call(Map config = [:]) {
def status = config.status ?: currentBuild.result ?: 'UNKNOWN'
def message = config.message ?: "Build ${status}"
def payload = [
[
timestamp: new Date().format("yyyy-MM-dd'T'HH:mm:ss'Z'", TimeZone.getTimeZone('UTC')),
job_name: env.JOB_NAME,
build_number: env.BUILD_NUMBER as Integer,
build_url: env.BUILD_URL,
git_commit: env.GIT_COMMIT,
git_branch: env.GIT_BRANCH,
node: env.NODE_NAME,
status: status,
message: message,
duration_ms: currentBuild.duration,
executor: currentBuild.getBuildCauses()[0]?.userId ?: 'system'
]
]
httpRequest(
url: "${env.PARSEABLE_URL}/api/v1/ingest",
httpMode: 'POST',
contentType: 'APPLICATION_JSON',
customHeaders: [
[name: 'Authorization', value: "Basic ${env.PARSEABLE_AUTH}"],
[name: 'X-P-Stream', value: config.dataset ?: 'jenkins-builds']
],
requestBody: groovy.json.JsonOutput.toJson(payload),
validResponseCodes: '200:299',
quiet: true
)
}
```
Use in your pipeline:
```groovy
@Library('my-shared-library') _
pipeline {
agent any
stages {
stage('Build') {
steps {
parseableNotify(status: 'STARTED', message: 'Build started')
sh 'make build'
}
}
}
post {
success {
parseableNotify(status: 'SUCCESS')
}
failure {
parseableNotify(status: 'FAILURE')
}
}
}
```
## Jenkins Credentials
Add these credentials in Jenkins:
| Credential ID | Type | Description |
| ---------------- | ----------- | ---------------------------------- |
| `parseable-url` | Secret text | Parseable instance URL |
| `parseable-auth` | Secret text | Base64 encoded `username:password` |
## Querying Jenkins Logs
Query your Jenkins logs in Parseable:
```sql
-- Get recent builds
SELECT timestamp, job_name, build_number, status, duration
FROM "jenkins-builds"
ORDER BY timestamp DESC
LIMIT 100
-- Find failed builds
SELECT timestamp, job_name, build_number, message, build_url
FROM "jenkins-builds"
WHERE status = 'FAILURE'
ORDER BY timestamp DESC
-- Average build duration by job
SELECT
job_name,
COUNT(*) as total_builds,
AVG(duration_ms) / 1000 as avg_duration_seconds,
MAX(duration_ms) / 1000 as max_duration_seconds
FROM "jenkins-builds"
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY job_name
ORDER BY avg_duration_seconds DESC
-- Build success rate
SELECT
job_name,
COUNT(*) as total,
SUM(CASE WHEN status = 'SUCCESS' THEN 1 ELSE 0 END) as successful,
ROUND(SUM(CASE WHEN status = 'SUCCESS' THEN 1 ELSE 0 END)::float / COUNT(*) * 100, 2) as success_rate
FROM "jenkins-builds"
WHERE timestamp > NOW() - INTERVAL '30 days'
GROUP BY job_name
ORDER BY success_rate ASC
```
## Best Practices
1. **Use Shared Libraries** - Centralize logging logic
2. **Include Git Context** - Add commit, branch, and author info
3. **Log Stage Transitions** - Track each stage separately
4. **Capture Build Causes** - Know who/what triggered builds
5. **Set Up Alerts** - Notify on repeated failures
## Troubleshooting
### HTTP Request Plugin Errors
1. Install the HTTP Request plugin
2. Verify Parseable URL is accessible from Jenkins
3. Check credentials are properly configured
4. Review Jenkins console output for errors
### Missing Logs
1. Verify the log file paths are correct
2. Check Fluent Bit/Logstash has read permissions
3. Verify the Parseable dataset exists
## Next Steps
* Set up [alerts](/user-guide/alerting) for failed builds
* Create [dashboards](/user-guide/dashboards) for CI/CD metrics
* Configure [GitHub Actions](/ingest-data/cicd/github-actions) for additional pipelines
# Terraform (/ingest-data/cicd/terraform)
Collect Terraform execution logs and state changes in Parseable.
## Overview
Integrate Terraform with Parseable to:
* **Execution Logs** - Track plan and apply operations
* **State Changes** - Monitor infrastructure changes
* **Audit Trail** - Compliance and change history
* **Cost Tracking** - Monitor resource changes
## Prerequisites
* Terraform CLI or Terraform Cloud
* Parseable instance accessible
* CI/CD pipeline (recommended)
## Method 1: CI/CD Integration
Send Terraform logs from your CI/CD pipeline.
### GitHub Actions
```yaml
name: Terraform
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
terraform:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Terraform Init
id: init
run: terraform init
- name: Terraform Plan
id: plan
run: terraform plan -out=tfplan -json > plan.json 2>&1
continue-on-error: true
- name: Send Plan to Parseable
run: |
# Parse plan output
ADDS=$(jq '[.resource_changes[]? | select(.change.actions | contains(["create"]))] | length' plan.json 2>/dev/null || echo 0)
CHANGES=$(jq '[.resource_changes[]? | select(.change.actions | contains(["update"]))] | length' plan.json 2>/dev/null || echo 0)
DESTROYS=$(jq '[.resource_changes[]? | select(.change.actions | contains(["delete"]))] | length' plan.json 2>/dev/null || echo 0)
curl -X POST "${{ secrets.PARSEABLE_URL }}/api/v1/ingest" \
-H "Authorization: Basic ${{ secrets.PARSEABLE_AUTH }}" \
-H "X-P-Stream: terraform-logs" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"event\": \"plan\",
\"repository\": \"${{ github.repository }}\",
\"ref\": \"${{ github.ref }}\",
\"sha\": \"${{ github.sha }}\",
\"actor\": \"${{ github.actor }}\",
\"status\": \"${{ steps.plan.outcome }}\",
\"resources_to_add\": ${ADDS},
\"resources_to_change\": ${CHANGES},
\"resources_to_destroy\": ${DESTROYS}
}]"
- name: Terraform Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
id: apply
run: terraform apply -auto-approve tfplan
- name: Send Apply to Parseable
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: |
curl -X POST "${{ secrets.PARSEABLE_URL }}/api/v1/ingest" \
-H "Authorization: Basic ${{ secrets.PARSEABLE_AUTH }}" \
-H "X-P-Stream: terraform-logs" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"event\": \"apply\",
\"repository\": \"${{ github.repository }}\",
\"sha\": \"${{ github.sha }}\",
\"actor\": \"${{ github.actor }}\",
\"status\": \"${{ steps.apply.outcome }}\"
}]"
```
### GitLab CI
```yaml
stages:
- plan
- apply
variables:
TF_ROOT: ${CI_PROJECT_DIR}
plan:
stage: plan
image: hashicorp/terraform:latest
script:
- terraform init
- terraform plan -out=tfplan -json > plan.json
- |
ADDS=$(jq '[.resource_changes[]? | select(.change.actions | contains(["create"]))] | length' plan.json)
CHANGES=$(jq '[.resource_changes[]? | select(.change.actions | contains(["update"]))] | length' plan.json)
DESTROYS=$(jq '[.resource_changes[]? | select(.change.actions | contains(["delete"]))] | length' plan.json)
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: terraform-logs" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"event\": \"plan\",
\"project\": \"${CI_PROJECT_PATH}\",
\"ref\": \"${CI_COMMIT_REF_NAME}\",
\"sha\": \"${CI_COMMIT_SHA}\",
\"actor\": \"${GITLAB_USER_LOGIN}\",
\"resources_to_add\": ${ADDS},
\"resources_to_change\": ${CHANGES},
\"resources_to_destroy\": ${DESTROYS}
}]"
artifacts:
paths:
- tfplan
apply:
stage: apply
image: hashicorp/terraform:latest
script:
- terraform init
- terraform apply -auto-approve tfplan
- |
curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
-H "Authorization: Basic ${PARSEABLE_AUTH}" \
-H "X-P-Stream: terraform-logs" \
-H "Content-Type: application/json" \
-d "[{
\"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
\"event\": \"apply\",
\"project\": \"${CI_PROJECT_PATH}\",
\"sha\": \"${CI_COMMIT_SHA}\",
\"status\": \"success\"
}]"
only:
- main
```
## Method 2: Terraform Cloud Webhooks
Use Terraform Cloud run notifications.
### Webhook Receiver
```javascript
const express = require('express');
const axios = require('axios');
const app = express();
app.use(express.json());
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;
app.post('/webhook', async (req, res) => {
const notification = req.body;
const logEntry = {
timestamp: new Date().toISOString(),
event: 'terraform_cloud',
run_id: notification.run_id,
run_status: notification.run_status,
workspace: notification.workspace_name,
organization: notification.organization_name,
message: notification.run_message,
created_by: notification.run_created_by,
is_destroy: notification.run_is_destroy,
plan_resource_additions: notification.plan_resource_additions,
plan_resource_changes: notification.plan_resource_changes,
plan_resource_destructions: notification.plan_resource_destructions
};
try {
await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, [logEntry], {
headers: {
'Authorization': `Basic ${PARSEABLE_AUTH}`,
'X-P-Stream': 'terraform-cloud',
'Content-Type': 'application/json'
}
});
} catch (error) {
console.error('Error:', error);
}
res.status(200).json({ status: 'received' });
});
app.listen(3000);
```
### Configure in Terraform Cloud
1. Go to **Workspace Settings** → **Notifications**
2. Click **Create Notification**
3. Select **Webhook**
4. Enter your webhook URL
5. Select triggers (Run completed, needs attention, etc.)
## Querying Terraform Logs
```sql
-- Recent Terraform operations
SELECT timestamp, event, repository, actor, status,
resources_to_add, resources_to_change, resources_to_destroy
FROM "terraform-logs"
ORDER BY timestamp DESC
LIMIT 100
-- Failed applies
SELECT timestamp, repository, actor, status
FROM "terraform-logs"
WHERE event = 'apply' AND status != 'success'
ORDER BY timestamp DESC
-- Resource change summary
SELECT
DATE_TRUNC('day', timestamp) as day,
SUM(resources_to_add) as total_added,
SUM(resources_to_change) as total_changed,
SUM(resources_to_destroy) as total_destroyed
FROM "terraform-logs"
WHERE event = 'plan'
GROUP BY day
ORDER BY day DESC
-- Most active repositories
SELECT
repository,
COUNT(*) as operations,
SUM(resources_to_add + resources_to_change + resources_to_destroy) as total_changes
FROM "terraform-logs"
WHERE timestamp > NOW() - INTERVAL '30 days'
GROUP BY repository
ORDER BY operations DESC
```
## Best Practices
1. **Log Plans and Applies** - Track both operations
2. **Include Resource Counts** - Monitor change volume
3. **Track Actors** - Know who made changes
4. **Alert on Destroys** - Notify on resource deletions
5. **Store Plan Files** - Keep for audit purposes
## Troubleshooting
### Missing Logs
1. Check CI/CD pipeline logs
2. Verify Parseable endpoint
3. Check authentication
### Incorrect Counts
1. Verify JSON parsing
2. Check plan output format
3. Handle empty plans
## Next Steps
* Set up [alerts](/user-guide/alerting) for infrastructure changes
* Create [dashboards](/user-guide/dashboards) for IaC metrics
* Configure [ArgoCD](/ingest-data/cicd/argocd) for GitOps
# Anthropic (/ingest-data/ai-agents/anthropic)
Log Anthropic Claude API calls, responses, and token usage to Parseable for LLM observability.
## Overview
Integrate Anthropic with Parseable to:
* **API Logging** - Track all Claude API calls
* **Token Usage** - Monitor input/output tokens
* **Latency Tracking** - Measure response times
* **Error Analysis** - Debug failed requests
* **Model Comparison** - Compare Claude model performance
## Prerequisites
* Anthropic API key
* Parseable instance accessible
* Python application
## Python Integration
### Basic Logger
```python
import anthropic
import requests
import time
from datetime import datetime
from typing import Dict, Any, List
class AnthropicLogger:
def __init__(self, parseable_url: str, dataset: str, username: str, password: str):
self.parseable_url = parseable_url
self.dataset = dataset
self.auth = (username, password)
self.client = anthropic.Anthropic()
def _log(self, entry: Dict[str, Any]):
try:
requests.post(
f"{self.parseable_url}/api/v1/ingest",
json=[entry],
auth=self.auth,
headers={"X-P-Stream": self.dataset},
timeout=5
)
except Exception as e:
print(f"Logging failed: {e}")
def message(
self,
messages: List[Dict],
model: str = "claude-3-opus-20240229",
max_tokens: int = 1024,
system: str = None,
**kwargs
) -> Any:
start_time = datetime.utcnow()
log_entry = {
"timestamp": start_time.isoformat() + "Z",
"provider": "anthropic",
"type": "message",
"model": model,
"max_tokens": max_tokens,
"message_count": len(messages),
"has_system": system is not None,
"user_prompt_preview": messages[-1]["content"][:200] if messages else None
}
try:
response = self.client.messages.create(
model=model,
max_tokens=max_tokens,
system=system,
messages=messages,
**kwargs
)
end_time = datetime.utcnow()
log_entry.update({
"success": True,
"duration_ms": (end_time - start_time).total_seconds() * 1000,
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
"total_tokens": response.usage.input_tokens + response.usage.output_tokens,
"stop_reason": response.stop_reason,
"response_preview": response.content[0].text[:200] if response.content else None
})
self._log(log_entry)
return response
except Exception as e:
log_entry.update({
"success": False,
"error": str(e),
"error_type": type(e).__name__
})
self._log(log_entry)
raise
# Usage
logger = AnthropicLogger(
parseable_url="http://parseable:8000",
dataset="anthropic-logs",
username="admin",
password="admin"
)
response = logger.message(
messages=[{"role": "user", "content": "Explain quantum computing in simple terms."}],
model="claude-3-sonnet-20240229",
system="You are a helpful science teacher."
)
```
### Streaming Support
```python
def message_stream(
self,
messages: List[Dict],
model: str = "claude-3-opus-20240229",
max_tokens: int = 1024,
**kwargs
):
start_time = datetime.utcnow()
log_entry = {
"timestamp": start_time.isoformat() + "Z",
"provider": "anthropic",
"type": "message_stream",
"model": model,
"max_tokens": max_tokens
}
try:
with self.client.messages.dataset(
model=model,
max_tokens=max_tokens,
messages=messages,
**kwargs
) as dataset:
full_response = ""
for text in dataset.text_stream:
full_response += text
yield text
# Get final message for usage stats
final_message = dataset.get_final_message()
log_entry.update({
"success": True,
"duration_ms": (datetime.utcnow() - start_time).total_seconds() * 1000,
"input_tokens": final_message.usage.input_tokens,
"output_tokens": final_message.usage.output_tokens,
"response_length": len(full_response)
})
except Exception as e:
log_entry.update({
"success": False,
"error": str(e)
})
raise
finally:
self._log(log_entry)
```
## Node.js Integration
```javascript
const Anthropic = require('@anthropic-ai/sdk');
const axios = require('axios');
class AnthropicLogger {
constructor(parseableUrl, dataset, auth) {
this.parseableUrl = parseableUrl;
this.dataset = dataset;
this.auth = auth;
this.client = new Anthropic();
}
async log(entry) {
try {
await axios.post(`${this.parseableUrl}/api/v1/ingest`, [entry], {
headers: {
'Authorization': `Basic ${this.auth}`,
'X-P-Stream': this.dataset
}
});
} catch (error) {
console.error('Logging failed:', error.message);
}
}
async message(messages, options = {}) {
const startTime = Date.now();
const model = options.model || 'claude-3-sonnet-20240229';
const logEntry = {
timestamp: new Date().toISOString(),
provider: 'anthropic',
type: 'message',
model,
message_count: messages.length
};
try {
const response = await this.client.messages.create({
model,
max_tokens: options.max_tokens || 1024,
messages,
...options
});
logEntry.success = true;
logEntry.duration_ms = Date.now() - startTime;
logEntry.input_tokens = response.usage?.input_tokens;
logEntry.output_tokens = response.usage?.output_tokens;
logEntry.stop_reason = response.stop_reason;
await this.log(logEntry);
return response;
} catch (error) {
logEntry.success = false;
logEntry.error = error.message;
await this.log(logEntry);
throw error;
}
}
}
```
## Querying Anthropic Logs
```sql
-- Token usage by model
SELECT
model,
SUM(input_tokens) as total_input,
SUM(output_tokens) as total_output,
COUNT(*) as requests
FROM "anthropic-logs"
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY model
ORDER BY total_input + total_output DESC
-- Latency percentiles
SELECT
model,
AVG(duration_ms) as avg_latency,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY duration_ms) as p50,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) as p95,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY duration_ms) as p99
FROM "anthropic-logs"
WHERE success = true
GROUP BY model
-- Error analysis
SELECT
error_type,
error,
COUNT(*) as count
FROM "anthropic-logs"
WHERE success = false
GROUP BY error_type, error
ORDER BY count DESC
```
## Best Practices
1. **Log Both Providers** - Compare OpenAI vs Anthropic
2. **Track Stop Reasons** - Monitor truncations
3. **Monitor Streaming** - Track streaming vs non-streaming
4. **Cost Tracking** - Calculate costs per model
## Next Steps
* Configure [OpenAI](/ingest-data/ai-agents/openai) logging
* Set up [LangChain](/ingest-data/ai-agents/langchain) tracing
* Create [dashboards](/user-guide/dashboards) for LLM metrics
# AutoGen (/ingest-data/ai-agents/autogen)
Log AutoGen multi-agent conversations, tool calls, and LLM interactions to Parseable.
## Overview
Integrate AutoGen with Parseable to:
* **Conversation Logging** - Track multi-agent conversations
* **Agent Monitoring** - Monitor individual agent behavior
* **Tool Usage** - Track function/tool calls
* **Token Tracking** - Aggregate token usage
* **Debug Workflows** - Trace complex agent interactions
## Prerequisites
* AutoGen installed (`pip install pyautogen`)
* Parseable instance accessible
* Python application
## Custom Logger
```python
import autogen
import requests
from datetime import datetime
from typing import Dict, Any, List, Optional
import uuid
import json
class ParseableAutogenLogger:
def __init__(self, parseable_url: str, dataset: str, username: str, password: str):
self.parseable_url = parseable_url
self.dataset = dataset
self.auth = (username, password)
self.conversation_id = str(uuid.uuid4())
self.message_count = 0
def _log(self, entry: Dict[str, Any]):
entry["timestamp"] = datetime.utcnow().isoformat() + "Z"
entry["conversation_id"] = self.conversation_id
try:
requests.post(
f"{self.parseable_url}/api/v1/ingest",
json=[entry],
auth=self.auth,
headers={"X-P-Stream": self.dataset},
timeout=5
)
except Exception as e:
print(f"Logging failed: {e}")
def log_message(self, sender: str, recipient: str, message: Dict[str, Any]):
self.message_count += 1
content = message.get("content", "")
if isinstance(content, list):
content = json.dumps(content)
log_entry = {
"event": "message",
"message_number": self.message_count,
"sender": sender,
"recipient": recipient,
"content_preview": str(content)[:500],
"content_length": len(str(content)),
"role": message.get("role", "unknown"),
"has_function_call": "function_call" in message or "tool_calls" in message
}
# Log function calls
if "function_call" in message:
log_entry["function_name"] = message["function_call"].get("name")
log_entry["function_args"] = message["function_call"].get("arguments", "")[:200]
if "tool_calls" in message:
log_entry["tool_calls"] = [
{"name": tc.get("function", {}).get("name")}
for tc in message.get("tool_calls", [])
]
self._log(log_entry)
def log_llm_call(self, agent_name: str, model: str, messages: List[Dict], response: Any):
log_entry = {
"event": "llm_call",
"agent": agent_name,
"model": model,
"input_messages": len(messages)
}
if hasattr(response, "usage"):
log_entry["prompt_tokens"] = response.usage.prompt_tokens
log_entry["completion_tokens"] = response.usage.completion_tokens
log_entry["total_tokens"] = response.usage.total_tokens
self._log(log_entry)
def log_function_execution(self, function_name: str, args: Dict, result: Any, duration_ms: float):
self._log({
"event": "function_execution",
"function_name": function_name,
"args": str(args)[:200],
"result_preview": str(result)[:200],
"duration_ms": duration_ms
})
def log_conversation_start(self, agents: List[str], task: str):
self._log({
"event": "conversation_start",
"agents": agents,
"task_preview": task[:500]
})
def log_conversation_end(self, summary: str, total_messages: int):
self._log({
"event": "conversation_end",
"summary": summary[:500],
"total_messages": total_messages
})
def new_conversation(self):
self.conversation_id = str(uuid.uuid4())
self.message_count = 0
```
### Wrapped Agent
```python
from autogen import AssistantAgent, UserProxyAgent
import time
class LoggedAssistantAgent(AssistantAgent):
def __init__(self, logger: ParseableAutogenLogger, *args, **kwargs):
super().__init__(*args, **kwargs)
self.logger = logger
def send(self, message, recipient, request_reply=None, silent=False):
self.logger.log_message(self.name, recipient.name, message)
return super().send(message, recipient, request_reply, silent)
class LoggedUserProxyAgent(UserProxyAgent):
def __init__(self, logger: ParseableAutogenLogger, *args, **kwargs):
super().__init__(*args, **kwargs)
self.logger = logger
self._original_execute = self.execute_function
def send(self, message, recipient, request_reply=None, silent=False):
self.logger.log_message(self.name, recipient.name, message)
return super().send(message, recipient, request_reply, silent)
def execute_function(self, func_call):
start_time = time.time()
result = self._original_execute(func_call)
duration_ms = (time.time() - start_time) * 1000
self.logger.log_function_execution(
func_call.get("name", "unknown"),
func_call.get("arguments", {}),
result,
duration_ms
)
return result
```
### Usage
```python
# Create logger
logger = ParseableAutogenLogger(
parseable_url="http://parseable:8000",
dataset="autogen-logs",
username="admin",
password="admin"
)
# Create logged agents
assistant = LoggedAssistantAgent(
logger=logger,
name="assistant",
llm_config={"model": "gpt-4"}
)
user_proxy = LoggedUserProxyAgent(
logger=logger,
name="user_proxy",
human_input_mode="NEVER",
code_execution_config={"work_dir": "coding"}
)
# Log conversation start
logger.log_conversation_start(
agents=["assistant", "user_proxy"],
task="Write a Python function to calculate fibonacci numbers"
)
# Start conversation
user_proxy.initiate_chat(
assistant,
message="Write a Python function to calculate fibonacci numbers"
)
# Log conversation end
logger.log_conversation_end(
summary="Completed fibonacci function",
total_messages=logger.message_count
)
```
## Querying AutoGen Logs
```sql
-- Conversation overview
SELECT
conversation_id,
MIN(timestamp) as started,
MAX(timestamp) as ended,
COUNT(*) as total_events,
SUM(CASE WHEN event = 'message' THEN 1 ELSE 0 END) as messages
FROM "autogen-logs"
GROUP BY conversation_id
ORDER BY started DESC
-- Agent activity
SELECT
sender,
COUNT(*) as messages_sent,
AVG(content_length) as avg_content_length
FROM "autogen-logs"
WHERE event = 'message'
GROUP BY sender
-- Function usage
SELECT
function_name,
COUNT(*) as calls,
AVG(duration_ms) as avg_duration
FROM "autogen-logs"
WHERE event = 'function_execution'
GROUP BY function_name
ORDER BY calls DESC
-- Token usage per conversation
SELECT
conversation_id,
SUM(total_tokens) as total_tokens,
COUNT(CASE WHEN event = 'llm_call' THEN 1 END) as llm_calls
FROM "autogen-logs"
GROUP BY conversation_id
ORDER BY total_tokens DESC
```
## Best Practices
1. **Track Conversation IDs** - Correlate all events
2. **Log All Agents** - Monitor each agent's behavior
3. **Track Function Calls** - Monitor tool usage
4. **Truncate Content** - Don't log full messages
## Next Steps
* Configure [CrewAI](/ingest-data/ai-agents/crewai) logging
* Set up [LangChain](/ingest-data/ai-agents/langchain) tracing
* Create [dashboards](/user-guide/dashboards) for agent metrics
# Baseten (/ingest-data/ai-agents/baseten)
import { Step, Steps } from 'fumadocs-ui/components/steps';
[Baseten](https://www.baseten.co/) is an inference platform for deploying and serving machine learning models, including LLMs, behind production APIs. Teams commonly use Baseten Dedicated Inference with [Truss](https://docs.baseten.co/development/overview) to package a model, deploy it on managed GPU infrastructure, and expose an inference endpoint.
Parseable can store the telemetry that Baseten emits around those deployments. This guide shows how to send Baseten logs, built-in Truss traces, and deployment metrics to Parseable with an OpenTelemetry Collector.
## How it works
Baseten exports each signal differently:
* **Logs:** Baseten sends organization-level logs to an OTLP/HTTP logs endpoint.
* **Traces:** A Truss deployment can export built-in request spans to an OTLP/HTTP traces endpoint after you enable tracing in `config.yaml`.
* **Metrics:** Baseten exposes deployment metrics in Prometheus format at `https://app.baseten.co/metrics`; the Collector scrapes that endpoint and forwards the metrics to Parseable.
Use the Collector as the bridge between Baseten and Parseable. It keeps Parseable credentials out of the Truss config, receives OTLP from Baseten, scrapes Baseten metrics, and writes each signal into a separate Parseable dataset.
Baseten trace export should point at a Collector instead of directly at Parseable when authentication is required. Baseten's trace configuration uses `environment_variables`, and Baseten recommends not putting authentication credentials there.
## Prerequisites
Before you start, keep these ready:
* A Baseten account and a Truss model deployed with Dedicated Inference
* The Baseten CLI authenticated locally
* An API key for invoking your model, and a Baseten workspace metrics export key
* A Parseable instance reachable from the Collector, and an API key with ingest access
* An OpenTelemetry Collector endpoint reachable from Baseten over HTTPS
* Docker, if you want to run the Collector with the example below
## Set up Baseten with Parseable
### Create a Baseten metrics export key
Baseten deployment metrics are scraped separately from logs and traces. Create a metrics-only workspace key for the Collector:
```bash
baseten auth login
baseten org api-key create \
--type workspace-export-metrics \
--name metrics-export
```
To restrict the key to specific models, add one or more `--model-id` flags:
```bash
baseten org api-key create \
--type workspace-export-metrics \
--name metrics-export \
--model-id
```
Store the printed key securely. Baseten shows it once, and this key is only for exporting metrics.
### Configure the OpenTelemetry Collector
Create `otel-collector-config.yaml`:
This example exports OTLP JSON from the Collector to Parseable. `X-P-Stream` selects the Parseable dataset, and `X-API-Key` carries a Parseable API key with ingest access.
OTLP Protobuf ingestion is available in Parseable Cloud and Enterprise. If your Parseable deployment supports Protobuf, you can switch the Parseable exporters below to `encoding: proto` and `Content-Type: application/x-protobuf`.
```yaml
extensions:
health_check:
endpoint: 0.0.0.0:13133
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
prometheus/baseten:
config:
scrape_configs:
- job_name: baseten
scheme: https
metrics_path: /metrics
scrape_interval: 60s
static_configs:
- targets: [app.baseten.co]
authorization:
type: Bearer
credentials: ${env:BASETEN_METRICS_EXPORT_KEY}
processors:
memory_limiter:
check_interval: 1s
limit_mib: 384
spike_limit_mib: 96
batch:
timeout: 5s
send_batch_size: 512
exporters:
otlphttp/parseable_logs:
endpoint: ${env:PARSEABLE_ENDPOINT}
encoding: json
headers:
X-API-Key: ${env:PARSEABLE_API_KEY}
X-P-Stream: ${env:PARSEABLE_LOGS_STREAM}
Content-Type: application/json
otlphttp/parseable_traces:
endpoint: ${env:PARSEABLE_ENDPOINT}
encoding: json
headers:
X-API-Key: ${env:PARSEABLE_API_KEY}
X-P-Stream: ${env:PARSEABLE_TRACES_STREAM}
Content-Type: application/json
otlphttp/parseable_metrics:
endpoint: ${env:PARSEABLE_ENDPOINT}
encoding: json
headers:
X-API-Key: ${env:PARSEABLE_API_KEY}
X-P-Stream: ${env:PARSEABLE_METRICS_STREAM}
Content-Type: application/json
service:
extensions: [health_check]
pipelines:
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/parseable_logs]
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/parseable_traces]
metrics:
receivers: [prometheus/baseten]
processors: [memory_limiter, batch]
exporters: [otlphttp/parseable_metrics]
```
Configure the Collector environment:
```bash
export PARSEABLE_ENDPOINT="https://parseable.example.com"
export PARSEABLE_API_KEY=""
export PARSEABLE_LOGS_STREAM="baseten-logs"
export PARSEABLE_TRACES_STREAM="baseten-traces"
export PARSEABLE_METRICS_STREAM="baseten-metrics"
export BASETEN_METRICS_EXPORT_KEY=""
```
Run the Collector:
```bash
docker run --rm \
-p 0.0.0.0:4318:4318 \
-p 127.0.0.1:13133:13133 \
-e PARSEABLE_ENDPOINT \
-e PARSEABLE_API_KEY \
-e PARSEABLE_LOGS_STREAM \
-e PARSEABLE_TRACES_STREAM \
-e PARSEABLE_METRICS_STREAM \
-e BASETEN_METRICS_EXPORT_KEY \
-v "$PWD/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro" \
otel/opentelemetry-collector-contrib:0.130.1 \
--config=/etc/otelcol-contrib/config.yaml
```
Check the Collector health endpoint:
```bash
curl http://localhost:13133
```
The response should include `"status":"Server available"`.
Baseten must reach the Collector over HTTPS. For production, run the Collector behind a stable TLS endpoint and protect ingress with a reverse proxy, private networking, or an allowlist. Do not expose an unauthenticated OTLP receiver broadly on the internet.
For a short local proof of concept, put a temporary HTTPS tunnel in front of the local Collector:
```bash
ngrok http 4318
```
Use the generated HTTPS forwarding URL as the Collector base URL in the Baseten settings below.
### Export Baseten logs
Baseten log export is configured at the organization level, so one OTLP destination receives logs for the organization.
In the Baseten dashboard:
1. Open **Organization settings > General settings**.
2. In the **OTEL connection** card, choose **Add connection**.
3. Set **Endpoint URL** to the full Collector logs endpoint:
```text
https://otel.example.com/v1/logs
```
4. Set **Header name** to a header accepted by your Collector or edge proxy.
5. Set **Header value** to the credential for that header.
6. Add any additional headers required by your receiver.
7. Save the connection.
For the simple Collector example above, the Collector does not require an ingress credential. In a local proof of concept, you can use a harmless routing header to satisfy the Baseten form:
| Field | Value |
| ------------ | ---------------------------------- |
| Endpoint URL | `https:///v1/logs` |
| Header name | `X-P-Stream` |
| Header value | `baseten-logs` |
The Collector still decides the final Parseable dataset using `PARSEABLE_LOGS_STREAM`.
Baseten exports new log records only. Historical logs are not backfilled. Delivery is best effort: Baseten retries transient failures, but records can be dropped if retries are exhausted or the export buffer fills.
Baseten's connection test sends a probe only for supported vendor domains. Custom Collector or Parseable endpoints can still receive real logs even when the test action returns an error, so verify delivery with a real inference request.
If the OTEL connection card is not visible in your Baseten settings, contact Baseten support to enable log export for the organization.
### Export Truss traces
Built-in trace export applies to Dedicated Inference deployments that use the Truss server. For custom servers, configure OpenTelemetry instrumentation in your server or framework instead.
Enabling built-in trace export can add request overhead. Turn it on deliberately and check latency before using it for production traffic.
Update the Truss `config.yaml` for your model:
```yaml
environment_variables:
OTEL_EXPORTER_OTLP_ENDPOINT: https://otel.example.com/v1/traces
runtime:
enable_tracing_data: true
```
Use your public Collector trace endpoint and include `/v1/traces`.
Deploy the updated Truss:
```bash
truss push
```
If this creates a new deployment that is not serving production traffic yet, promote it from the Baseten dashboard or with your normal deployment workflow. Then send inference requests to generate traces.
In Parseable, Truss built-in spans usually have `service.name` set to `truss-server`.
### Export Baseten deployment metrics
The Collector configuration above scrapes:
```text
https://app.baseten.co/metrics
```
with `BASETEN_METRICS_EXPORT_KEY`, then forwards the data to the `baseten-metrics` dataset.
Keep the scrape interval at `60s`. Baseten rate limits the metrics endpoint to six requests per minute per organization across all keys and collectors.
After one scrape interval, verify scrape health before building dashboards. A healthy scrape should include an `up` sample for the `baseten` job, and the `baseten-metrics` dataset should start receiving deployment metric samples.
### Generate test telemetry
Send an inference request to the production environment:
```bash
export BASETEN_API_KEY=""
export BASETEN_MODEL_ID=""
export BASETEN_MODEL_NAME=""
curl -sD- \
"https://model-${BASETEN_MODEL_ID}.api.baseten.co/environments/production/sync/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${BASETEN_API_KEY}" \
-d '{
"model": "'"${BASETEN_MODEL_NAME}"'",
"messages": [
{
"role": "user",
"content": "Say hello from a Baseten deployment and emit normal serving telemetry."
}
]
}'
```
Keep the `X-Baseten-Request-Id` response header. It is useful for finding related Baseten logs.
## Explore data in Parseable
Open Parseable and confirm that the datasets exist: `baseten-metrics`, `baseten-logs`, and `baseten-traces`.
Run quick checks from the Parseable query UI:
```sql
SELECT * FROM "baseten-metrics"
ORDER BY p_timestamp DESC
LIMIT 20;
```
```sql
SELECT * FROM "baseten-logs"
ORDER BY p_timestamp DESC
LIMIT 20;
```
```sql
SELECT * FROM "baseten-traces"
ORDER BY p_timestamp DESC
LIMIT 20;
```
Open `baseten-metrics` from the Metrics page. The dataset contains the Prometheus samples scraped from Baseten and forwarded through the Collector.
Open `baseten-logs` from the Logs page. The dataset shows Baseten serving logs, deployment metadata, service filters, and log volume over time.
Open `baseten-traces` from the Traces page. The dataset shows Truss request spans and related OpenTelemetry attributes for the deployment.
## Dashboards
The reusable [Baseten Observability dashboard](https://github.com/parseablehq/dashboards/tree/main/baseten-observability) is available in the `parseablehq/dashboards` repository.
Import `baseten-observability-mixed.json`, then map its dataset variables to `baseten-metrics`, `baseten-logs`, and `baseten-traces`, or to your chosen dataset names. The dashboard combines request traffic, latency, concurrency, GPU capacity, pod readiness, logs, errors, and trace hotspots in one view.
## Troubleshooting
| Symptom | What to check |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No logs in Parseable | Confirm the Baseten OTEL connection uses `/v1/logs`, the Collector is reachable from Baseten, and new logs were generated after the connection was saved. |
| Baseten connection test fails for a custom domain | Baseten can still export logs to custom endpoints even when the dashboard test does not support that domain. Verify delivery in Parseable with a real prediction request. |
| No traces in Parseable | Confirm `runtime.enable_tracing_data: true`, the trace endpoint includes `/v1/traces`, the updated deployment is active, and the Collector is reachable over HTTPS. |
| No metrics in Parseable | Confirm `BASETEN_METRICS_EXPORT_KEY` is a `workspace-export-metrics` key and the Collector has been running for at least one scrape interval. |
| `401` from Parseable | Check `PARSEABLE_API_KEY`, the `X-API-Key` header, and that you are using the Parseable ingest endpoint. |
| `429` from Baseten metrics | Use a `60s` scrape interval and reduce the number of collectors scraping the organization metrics endpoint. |
| The deployment is costing money while testing | Deactivate or scale down the Baseten deployment when you are done generating telemetry. Reactivate it before sending more test inference requests. |
## References
* [Baseten logs](https://docs.baseten.co/observability/logs)
* [Baseten tracing](https://docs.baseten.co/observability/tracing)
* [Baseten metrics export](https://docs.baseten.co/observability/export-metrics/overview)
* [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/)
# CrewAI (/ingest-data/ai-agents/crewai)
import { Step, Steps } from 'fumadocs-ui/components/steps';
[CrewAI](https://www.crewai.com/) is a Python framework for building agentic workflows where agents take roles, receive tasks, call tools, and work together as a crew. That is useful, but it also makes debugging harder than a normal request-response application. A single run can include multiple agents, task handoffs, model calls, tool calls, retries, and failures before the final output is returned.
Parseable receives CrewAI telemetry as OpenTelemetry traces. In this guide, the CrewAI application is instrumented with [OpenInference](https://github.com/Arize-ai/openinference), exported over OTLP/HTTP, and stored in a Parseable traces dataset. Once the data is in Parseable, you can inspect the full crew run, follow the agent and task spans, review model calls, and debug failed runs from the same trace view.
## How it works
The integration has two layers of instrumentation.
CrewAI instrumentation captures the application workflow. This is where you see the crew, agents, tasks, tool calls, and execution flow. Provider instrumentation captures the underlying model calls. In this example we use OpenAI, so the setup includes `openinference-instrumentation-openai` to capture LLM spans and model attributes.
```text
CrewAI application
|
| OpenInference CrewAI instrumentation
| OpenInference OpenAI instrumentation
v
OpenTelemetry SDK
|
| OTLP/HTTP traces
v
Parseable /v1/traces
|
+--> crewai-traces
Parseable traces dataset
```
Use the LLM instrumentor that matches the provider your CrewAI app calls. This guide uses OpenAI. If your crew uses Anthropic, LiteLLM, or another provider, pair `openinference-instrumentation-crewai` with that provider's OpenInference instrumentor.
## Prerequisites
Before you start, keep these ready:
* A running Parseable instance
* A Parseable API key with ingest access
* Python 3.10 or later
* A CrewAI 1.0 or later application
* An OpenAI API key for the example in this guide
## Set up CrewAI with Parseable
### Create a traces dataset
Set the Parseable connection values:
```bash
export PARSEABLE_URL="https://"
export PARSEABLE_API_KEY=""
export PARSEABLE_STREAM="crewai-traces"
```
Create a dataset for CrewAI spans and tag it for Agent Observability:
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/$PARSEABLE_STREAM" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: traces" \
-H "X-P-Dataset-Tag: agent-observability"
```
The dataset will be available in the Traces view. Because it is tagged for Agent Observability, it can also be used from the Agents page.
### Install the packages
Install CrewAI, OpenInference instrumentation, and the OpenTelemetry OTLP HTTP exporter:
```bash
pip install \
crewai \
openinference-instrumentation-crewai \
openinference-instrumentation-openai \
opentelemetry-sdk \
opentelemetry-exporter-otlp-proto-http
```
If your crew uses a different model provider, install the matching OpenInference package for that provider as well.
### Configure the environment
Set the OpenAI API key used by the sample crew:
```bash
export OPENAI_API_KEY=""
```
Keep `PARSEABLE_URL`, `PARSEABLE_API_KEY`, and `PARSEABLE_STREAM` in the same shell. The Python code below appends `/v1/traces` to `PARSEABLE_URL`.
### Instrument and run the crew
Set up the OpenTelemetry tracer provider first, then instrument CrewAI and the model provider before creating the crew. This order matters because the instrumentors patch the libraries they observe.
```python
import os
from openinference.instrumentation.crewai import CrewAIInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
parseable_url = os.environ["PARSEABLE_URL"].rstrip("/")
parseable_stream = os.getenv("PARSEABLE_STREAM", "crewai-traces")
provider = TracerProvider(
resource=Resource.create(
{
"service.name": "crewai-research-crew",
"deployment.environment.name": "development",
}
)
)
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=f"{parseable_url}/v1/traces",
headers={
"Authorization": f"Bearer {os.environ['PARSEABLE_API_KEY']}",
"X-P-Stream": parseable_stream,
},
)
)
)
CrewAIInstrumentor().instrument(tracer_provider=provider)
OpenAIInstrumentor().instrument(tracer_provider=provider)
from crewai import Agent, Crew, Process, Task
researcher = Agent(
role="Researcher",
goal="Find accurate facts about {topic}",
backstory="You verify claims and summarize the strongest evidence.",
llm="openai/gpt-4o-mini",
verbose=True,
)
writer = Agent(
role="Writer",
goal="Turn research about {topic} into a concise briefing",
backstory="You write clear technical summaries for engineering teams.",
llm="openai/gpt-4o-mini",
verbose=True,
)
research_task = Task(
description="Research three important facts about {topic}.",
expected_output="Three verified facts with short explanations.",
agent=researcher,
)
writing_task = Task(
description="Use the research to write a five-paragraph briefing about {topic}.",
expected_output="A concise five-paragraph technical briefing.",
agent=writer,
context=[research_task],
)
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, writing_task],
process=Process.sequential,
verbose=True,
)
try:
result = crew.kickoff(inputs={"topic": "OpenTelemetry for AI agents"})
print(result.raw)
finally:
provider.shutdown()
```
`provider.shutdown()` is useful for scripts and short-running jobs. It flushes buffered spans before the process exits, so the final crew run is not left behind in memory.
Pass the tracer provider directly to the CrewAI and OpenAI instrumentors as shown above. Avoid calling `set_tracer_provider()` in this setup because the OpenTelemetry global provider can only be set once in a process.
## What you should see
One `crew.kickoff()` usually becomes one trace. Inside that trace, you should see spans for the crew run, agent execution, task execution, and the underlying LLM calls.
```text
Crew kickoff
├── Agent and task execution
│ └── OpenAI LLM call
└── Agent and task execution
└── OpenAI LLM call
```
Crews that use tools, delegation, retries, or additional model calls will have more child spans. Depending on what runs, OpenInference records attributes such as span kind, model name, token usage, inputs, outputs, agent role, task description, and error details.
## View in Parseable
Open the Parseable console and go to **Traces**. Select `crewai-traces`, filter by the service name `crewai-research-crew`, and open a trace to inspect the waterfall and span attributes.
The trace overview helps you follow the crew run from start to finish. This is usually the first place to check when a run is slow, when a task took longer than expected, or when you want to see how many model and tool calls were made during the run.
The LLM span shows the model call made under the crew workflow. This is where model name, provider attributes, prompt or response metadata, token usage, and latency become easier to inspect without separating the model call from the agent run that caused it.
When something fails, open the error span and inspect its attributes. This helps you see whether the failure came from a task, tool, model provider, or application code.
## Use with Agent Observability
If the dataset is tagged with `X-P-Dataset-Tag: agent-observability`, you can use the Agents page to explore CrewAI runs from an agent-focused view. This is useful when you want to compare runs, look at model usage, inspect tool behavior, or move from a high-level run into the trace details.
## Production notes
Direct export from the Python process is a good way to get started. For production workloads, send OTLP from the application to an OpenTelemetry Collector and let the Collector forward traces to Parseable. A Collector gives you central batching, retries, filtering, sampling, and redaction without changing the CrewAI code again.
Do not enable multiple instrumentation paths for the same library. For example, avoid using the explicit OpenInference instrumentor in this guide and OpenTelemetry no-code auto-instrumentation for CrewAI at the same time. That can create duplicate spans.
## Protect sensitive data
Agent goals, backstories, task descriptions, prompts, completions, and tool inputs or outputs can contain sensitive data. Review what your spans contain before enabling this in production. If required, use OpenTelemetry processors or OpenInference configuration to redact or suppress content before it reaches Parseable.
## Troubleshooting
* **No traces appear:** Confirm that `PARSEABLE_URL` is reachable from the application, the API key has ingest access, and the exporter endpoint is `/v1/traces`.
* **Only CrewAI spans appear:** CrewAI instrumentation captures the workflow, but provider-level instrumentation captures the detailed model call. Make sure the matching provider instrumentor runs before the crew executes.
* **LLM spans are missing:** Check that your CrewAI `llm` value matches the provider instrumentor you installed. The example uses OpenAI, so it uses `openinference-instrumentation-openai`.
* **Spans are duplicated:** Use only one instrumentation path per library. Disable no-code auto-instrumentation when using the explicit instrumentor calls shown here.
* **CrewAI says tracing is disabled:** That message refers to CrewAI's own tracing product. It does not mean OpenInference has stopped exporting OpenTelemetry spans to Parseable.
## Next steps
* Explore traces with the [Parseable Traces UI](/user-guide/traces)
* Learn how Parseable presents [Agent Observability](/user-guide/agent-observability)
* Build or customize [dashboards](/user-guide/dashboards)
# Daytona (/ingest-data/ai-agents/daytona)
[Daytona](https://www.daytona.io/) provides secure, isolated sandboxes for running code and agent workloads. A typical Daytona workflow might create a sandbox, upload files, execute code, stream logs, inspect results, and then clean up the sandbox when the work is done.
That flow is easy to understand while testing locally, but production sandboxes need more visibility. When a sandbox is slow, fails during execution, or starts using more CPU or memory than expected, you need to see both sides of the run:
* what your application did through the Daytona SDK
* what happened inside the sandbox runtime
Parseable can receive this telemetry over OpenTelemetry, so SDK traces, sandbox logs, sandbox traces, and sandbox metrics can be inspected with the rest of your observability data.
## How the integration works
Daytona has two OpenTelemetry paths.
| Path | Where you configure it | What it sends to Parseable |
| ---------------------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| SDK tracing | In the application that uses the Daytona SDK | Traces for SDK operations such as sandbox creation, file operations, process execution, and API calls |
| Sandbox telemetry collection | Daytona Dashboard, Settings, OpenTelemetry | Logs, traces, sandbox resource metrics, and organization resource metrics |
These paths are independent. You can enable SDK tracing when you only need to understand how your application calls Daytona. You can enable sandbox telemetry when you need to see what happened inside sandboxes. In most production setups, you will want both.
```text
Application using Daytona SDK
-> SDK traces
-> Parseable dataset: daytona-sdk-traces
Daytona sandbox telemetry
-> OpenTelemetry Collector
-> logs -> Parseable dataset: daytona-logs
-> traces -> Parseable dataset: daytona-traces
-> metrics -> Parseable dataset: daytona-metrics
```
The OpenTelemetry Collector is recommended for sandbox telemetry because Daytona uses one organization-level OTLP configuration. Each Parseable dataset needs its own `X-P-Stream` header. The Collector receives all signals from Daytona and routes each one to the correct Parseable dataset.
## Prerequisites
Before you start, keep these ready:
* A Daytona account and API key
* A Parseable Cloud or Enterprise instance reachable from your application and from the OpenTelemetry Collector. These examples use OTLP over HTTP with Protocol Buffers.
* A Parseable API key with ingest access, sent as a Bearer token
* Python 3 if you want to run the SDK example below
* An OpenTelemetry Collector for routing sandbox logs, traces, and metrics
If your Parseable deployment still uses Basic Auth, replace the Bearer token with a Basic Auth value in the examples.
## Send SDK traces to Parseable
SDK tracing shows the operations your application performs through the Daytona SDK. This includes sandbox lifecycle calls, file system operations, code execution, process management, and HTTP calls made by the SDK.
Install the Daytona SDK:
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install daytona
```
Set the Daytona and OpenTelemetry environment variables:
```bash
export DAYTONA_API_KEY=""
export DAYTONA_OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT="https://parseable.example.com"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ,X-P-Stream=daytona-sdk-traces"
```
Use the Parseable base URL without `/v1/traces`. The OTLP HTTP exporter appends the signal path.
Keep `OTEL_EXPORTER_OTLP_HEADERS` quoted. It contains commas and values that shells can parse incorrectly when loaded from an environment file.
Create a small traced operation:
```python
import asyncio
import os
from daytona import AsyncDaytona, DaytonaConfig
async def main() -> None:
config = DaytonaConfig(
api_key=os.environ["DAYTONA_API_KEY"],
otel_enabled=True,
)
async with AsyncDaytona(config) as daytona:
sandbox = await daytona.create()
try:
result = await sandbox.process.code_run(
'print("hello from a Daytona sandbox")'
)
print(result.result)
finally:
await daytona.delete(sandbox)
if __name__ == "__main__":
asyncio.run(main())
```
Run it:
```bash
python test_daytona.py
```
Open the `daytona-sdk-traces` dataset in Parseable. You should see traces for the SDK operations that created the sandbox, executed code, and deleted the sandbox.
The line printed inside the sandbox is not exported as a log by SDK tracing. SDK tracing only describes the SDK calls. To collect stdout, stderr, runtime logs, and resource metrics from sandboxes, configure sandbox telemetry.
## Send sandbox telemetry to Parseable
Sandbox telemetry flow
Daytona sandboxes
Runtime logs, traces, and resource metrics
->
OpenTelemetry Collector
Receives OTLP and routes each signal
->
logs todaytona-logs
traces todaytona-traces
metrics todaytona-metrics
Sandbox telemetry comes from Daytona's organization-level OpenTelemetry configuration. Daytona can collect:
* application stdout and stderr
* system logs, runtime errors, and warnings
* HTTP request traces and custom spans from inside sandboxes
* CPU, memory, and filesystem metrics for sandboxes
* organization-level resource usage and quota metrics
Daytona documents this configuration under **Daytona Dashboard > Settings > OpenTelemetry**. The setting is available to organization owners. After it is saved, allow a few minutes for the change to apply.
Because Daytona sends all sandbox telemetry to one OTLP endpoint, put an OpenTelemetry Collector between Daytona and Parseable.
When this is set up, the `daytona-logs` dataset gives you the runtime side of the sandbox run. You can inspect the log volume over time, filter by log category, and open the matching records to see phase changes, stdout checkpoints, and other sandbox output in one place.
### Configure the Collector
Create `otel-collector-config.yaml`:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
otlphttp/parseable_logs:
endpoint: "${env:PARSEABLE_ENDPOINT}"
encoding: proto
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: "daytona-logs"
Content-Type: "application/x-protobuf"
otlphttp/parseable_traces:
endpoint: "${env:PARSEABLE_ENDPOINT}"
encoding: proto
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: "daytona-traces"
Content-Type: "application/x-protobuf"
otlphttp/parseable_metrics:
endpoint: "${env:PARSEABLE_ENDPOINT}"
encoding: proto
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: "daytona-metrics"
Content-Type: "application/x-protobuf"
service:
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_logs]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_traces]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_metrics]
```
Run the Collector:
```bash
export PARSEABLE_ENDPOINT="https://parseable.example.com"
export PARSEABLE_API_KEY=""
docker run --rm \
-p 4317:4317 \
-p 4318:4318 \
-e PARSEABLE_ENDPOINT \
-e PARSEABLE_API_KEY \
-v "$PWD/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro" \
otel/opentelemetry-collector-contrib:latest
```
Expose the Collector through a public HTTPS endpoint that Daytona can reach, for example `https://otel.example.com`. Protect this endpoint with authentication at your ingress, proxy, or load balancer. Do not expose an unauthenticated OTLP receiver to the internet.
### Configure Daytona
In Daytona, open **Settings > OpenTelemetry** and enter the Collector base URL:
```text
OTLP Endpoint: https://otel.example.com
```
Do not append `/v1/logs`, `/v1/traces`, or `/v1/metrics`. Daytona sends each signal to the matching OTLP path.
If your Collector endpoint requires an auth header, add it in the Daytona OpenTelemetry settings:
```text
Authorization: Bearer
```
Once saved, Daytona will export sandbox telemetry to the Collector, and the Collector will forward each signal to the matching Parseable dataset.
## Add useful labels
Daytona adds resource attributes such as organization, region, and snapshot information to sandbox telemetry. You can add your own labels with `DAYTONA_SANDBOX_OTEL_EXTRA_LABELS`.
For example:
```bash
DAYTONA_SANDBOX_OTEL_EXTRA_LABELS="team=agents,env=staging,app=code-runner"
```
These labels become OpenTelemetry resource attributes. In Parseable, they help you filter by team, environment, application, or any other dimension that matters for your setup.
## Validate in Parseable
After the setup is saved, run a sandbox workload and keep it active for at least a minute so Daytona can collect resource samples.
```python
import asyncio
import os
from daytona import AsyncDaytona, DaytonaConfig
async def main() -> None:
config = DaytonaConfig(api_key=os.environ["DAYTONA_API_KEY"])
async with AsyncDaytona(config) as daytona:
sandbox = await daytona.create()
try:
result = await sandbox.process.code_run(
"""
import time
print("starting Daytona telemetry workload")
end = time.time() + 90
while time.time() < end:
sum(i * i for i in range(250000))
print("finished Daytona telemetry workload")
""",
timeout=150,
)
print(result.result)
finally:
await daytona.delete(sandbox)
if __name__ == "__main__":
asyncio.run(main())
```
Then check these datasets in Parseable:
| Dataset | What to expect |
| -------------------- | -------------------------------------------------------------------------- |
| `daytona-sdk-traces` | SDK operations from your application, if SDK tracing is enabled |
| `daytona-logs` | stdout, stderr, system logs, runtime errors, and warnings from sandboxes |
| `daytona-traces` | HTTP request traces and custom spans emitted inside sandboxes |
| `daytona-metrics` | sandbox CPU, memory, filesystem metrics, and organization resource metrics |
Daytona organization metrics are pushed periodically, so allow at least 60 to 90 seconds before checking the metrics dataset.
Useful metric names include:
```text
daytona.sandbox.cpu.utilization
daytona.sandbox.memory.utilization
daytona.sandbox.filesystem.utilization
daytona.sandbox.used_cpu
daytona.sandbox.used_ram
daytona.sandbox.used_storage
```
The `daytona-metrics` dataset is useful when you want to see how a sandbox behaved over time instead of only checking whether it succeeded. In practice, this is where CPU bursts, memory spikes, and filesystem changes are easiest to spot during longer code runs.
## Direct export or Collector
Direct export to Parseable is fine for SDK traces because the SDK trace exporter only needs one signal-specific configuration.
For sandbox telemetry, use the Collector. Daytona's organization setting uses one endpoint and one header set for logs, traces, and metrics. The Collector lets you send each signal to the correct Parseable dataset with its own `X-P-Stream` header.
## Troubleshooting
* If SDK traces do not appear, confirm `DAYTONA_OTEL_ENABLED=true`, use `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`, and close the Daytona client so buffered spans flush.
* If the exporter reaches the wrong path, use the Parseable base URL in `OTEL_EXPORTER_OTLP_ENDPOINT`. Do not append `/v1/traces`.
* If authentication fails, confirm the API key has ingest access and is sent as `Authorization: Bearer `.
* If sandbox logs or metrics are missing from SDK tracing, that is expected. Configure Daytona sandbox telemetry in the dashboard.
* If no sandbox telemetry appears, confirm the Daytona organization setting has applied and the Collector is reachable over HTTPS from Daytona.
* If only one signal appears, check that the Collector has separate logs, traces, and metrics pipelines, and that each exporter uses the intended `X-P-Stream`.
* If metrics are delayed, wait at least 60 to 90 seconds after the workload starts.
For more protocol details, see [Daytona OpenTelemetry collection](https://www.daytona.io/docs/en/observability/otel-collection/), [Parseable OTLP logs](/ingest-data/otel/logs), [Parseable OTLP traces](/ingest-data/otel/traces), and [Parseable OTLP metrics](/ingest-data/otel/metrics).
# DBOS (/ingest-data/ai-agents/dbos)
DBOS applications can send workflow observability data to Parseable in two paths:
* **Logs and traces** are exported directly from the DBOS application through OpenTelemetry OTLP HTTP.
* **Metrics** are scraped from DBOS Conductor's Prometheus/OpenMetrics endpoint and forwarded to Parseable.
## Requirements
* A DBOS TypeScript application using `@dbos-inc/dbos-sdk`
* DBOS TypeScript `4.19` or later for the Conductor metrics endpoint
* A reachable Parseable instance for OTLP logs and traces
* Parseable Enterprise for PromQL metrics storage and querying
* DBOS Conductor for metrics. DBOS Conductor is part of the DBOS Enterprise offering.
Metrics require both Parseable Enterprise and DBOS Conductor. Without DBOS Conductor, there is no DBOS `/v1/metrics` endpoint to scrape. Without Parseable Enterprise, metrics cannot be queried through Parseable's PromQL support.
## Logs and traces
Enable OTLP export in the DBOS app and point logs and traces to Parseable's OTLP ingest endpoints.
```ts
import { DBOS } from '@dbos-inc/dbos-sdk';
DBOS.setConfig({
name: 'dbos-node-starter',
systemDatabaseUrl: process.env.DBOS_SYSTEM_DATABASE_URL,
applicationVersion: '0.1.0',
enableOTLP: true,
otlpLogsEndpoints: [`${process.env.PARSEABLE_OTLP_ENDPOINT}/v1/logs`],
otlpTracesEndpoints: [`${process.env.PARSEABLE_OTLP_ENDPOINT}/v1/traces`],
otelAttributeFormat: 'semconv',
});
await DBOS.launch({
conductorURL: process.env.DBOS_CONDUCTOR_URL,
conductorKey: process.env.DBOS_CONDUCTOR_KEY,
});
```
Configure Parseable OTLP headers separately for logs and traces. Use `X-P-Stream` for the destination stream name; Parseable derives the signal type from the endpoint path.
```bash
PARSEABLE_OTLP_ENDPOINT=http://parseable.example:8000
OTEL_EXPORTER_OTLP_LOGS_HEADERS=Authorization=Bearer ,X-P-Stream=dbos-logs
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer ,X-P-Stream=dbos-traces
```
After restarting the DBOS app, trigger a workflow and check the `dbos-logs` and `dbos-traces` streams in Parseable.
## Metrics
DBOS metrics are exposed by DBOS Conductor, not by the application process directly. The app connects to Conductor using the Conductor URL and API key:
```bash
DBOS_CONDUCTOR_URL=ws://localhost:8090
DBOS_CONDUCTOR_KEY=
```
Conductor exposes metrics at:
```text
/v1/metrics
```
The endpoint is Prometheus/OpenMetrics compatible and requires the Conductor API key as a bearer token.
```bash
curl \
-H "Authorization: Bearer " \
-H "Accept: application/openmetrics-text" \
http://localhost:8090/v1/metrics
```
Example metrics include:
```text
dbos_conductor_v1_executor_count
dbos_conductor_v1_workflow_pending_count
dbos_conductor_v1_workflow_oldest_pending_timestamp_seconds
```
## Scrape with Prometheus
Use Prometheus to scrape DBOS Conductor and forward metrics to Parseable.
```yaml
scrape_configs:
- job_name: dbos-conductor
metrics_path: /v1/metrics
authorization:
type: Bearer
credentials_file: /etc/prometheus/dbos_api_key
static_configs:
- targets:
- conductor:8090
remote_write:
- url: http://parseable.example:8000/v1/prometheus/write
authorization:
type: Bearer
credentials:
headers:
X-P-Stream: dbos-conductor-metrics
```
## Validate
Check that Conductor is available:
```bash
curl http://localhost:8090/healthz
```
Check that Prometheus is scraping Conductor:
```bash
curl 'http://localhost:9090/api/v1/query?query=dbos_conductor_v1_executor_count'
```
Check Parseable:
* `dbos-logs` should contain DBOS application log records.
* `dbos-traces` should contain DBOS workflow and step spans.
* `dbos-conductor-metrics` should contain DBOS Conductor metrics when Parseable Enterprise metrics ingest is enabled.
## Troubleshooting
* **Logs do not appear** - Confirm the logs exporter uses the `/v1/logs` endpoint and restart the DBOS application after changing environment variables.
* **Traces do not appear** - Confirm the traces exporter uses the `/v1/traces` endpoint and restart the DBOS application after changing environment variables.
* **Metrics endpoint returns unauthorized** - Pass the DBOS Conductor API key as a bearer token.
* **No metrics endpoint exists** - Metrics require DBOS Conductor. The regular DBOS application server does not expose `/v1/metrics`.
* **PromQL is unavailable in Parseable** - PromQL metrics querying requires Parseable Enterprise.
# DSPy (/ingest-data/ai-agents/dspy)
Log DSPy programs, module executions, and optimization runs to Parseable.
## Overview
Integrate DSPy with Parseable to:
* **Program Tracing** - Track DSPy program executions
* **Module Monitoring** - Log individual module calls
* **Optimization Tracking** - Monitor optimization progress
* **Token Usage** - Track LLM token consumption
## Prerequisites
* DSPy installed (`pip install dspy-ai`)
* Parseable instance accessible
* Python application
## Custom Logger
```python
import dspy
import requests
from datetime import datetime
from typing import Dict, Any
import uuid
import time
class ParseableDSPyLogger:
def __init__(self, parseable_url: str, dataset: str, username: str, password: str):
self.parseable_url = parseable_url
self.dataset = dataset
self.auth = (username, password)
self.program_id = str(uuid.uuid4())
def _log(self, entry: Dict[str, Any]):
entry["timestamp"] = datetime.utcnow().isoformat() + "Z"
entry["program_id"] = self.program_id
try:
requests.post(
f"{self.parseable_url}/api/v1/ingest",
json=[entry],
auth=self.auth,
headers={"X-P-Stream": self.dataset},
timeout=5
)
except Exception as e:
print(f"Logging failed: {e}")
def log_module_call(self, module_name: str, inputs: Dict, outputs: Dict, duration_ms: float):
self._log({
"event": "module_call",
"module_name": module_name,
"inputs": {k: str(v)[:100] for k, v in inputs.items()},
"outputs": {k: str(v)[:200] for k, v in outputs.items()},
"duration_ms": duration_ms
})
def log_optimization_trial(self, trial_num: int, score: float):
self._log({
"event": "optimization_trial",
"trial_num": trial_num,
"score": score
})
```
## Querying DSPy Logs
```sql
-- Module performance
SELECT module_name, AVG(duration_ms) as avg_duration, COUNT(*) as calls
FROM "dspy-logs"
WHERE event = 'module_call'
GROUP BY module_name
-- Optimization progress
SELECT trial_num, score
FROM "dspy-logs"
WHERE event = 'optimization_trial'
ORDER BY trial_num
```
## Next Steps
* Configure [LangChain](/ingest-data/ai-agents/langchain) tracing
* Set up [OpenAI](/ingest-data/ai-agents/openai) direct logging
# Hermes Agent (/ingest-data/ai-agents/hermes)
import { Step, Steps } from 'fumadocs-ui/components/steps';
[Hermes Agent](https://github.com/NousResearch/hermes-agent) is an AI agent with model-provider, tool, skill, gateway, and scheduled-task support. The [`hermes-otel`](https://github.com/briancaffey/hermes-otel) plugin observes Hermes lifecycle hooks and exports agent, model, API, tool, and skill activity through OpenTelemetry.
You can send each signal to a separate Parseable dataset:
```text
Hermes Agent hooks
|
v
hermes-otel plugin
|
+--> OTLP/HTTP traces --> hermes-traces
+--> OTLP/HTTP logs --> hermes-logs
+--> OTLP/HTTP metrics --> hermes-metrics
|
v
Parseable
```
The plugin can send OTLP/HTTP data to Parseable without an OpenTelemetry Collector. Add a Collector when you need central retry policy, redaction, sampling, or routing across several applications.
## Prerequisites
* A working Hermes Agent installation
* A Parseable instance that Hermes can reach
* A Parseable API key with dataset creation and ingest access
* A model-provider API key for Hermes
* Python package access in the environment where Hermes runs
Use the Parseable ingestor base URL, such as `https://parseable.example.com`. In a distributed deployment, the query/UI endpoint might reject ingestion.
## Set up Hermes Agent with Parseable
### Create the datasets
Set the Parseable connection values:
```bash
export PARSEABLE_URL="https://parseable.example.com"
export PARSEABLE_API_KEY=""
```
Create one dataset for each OpenTelemetry signal:
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/hermes-traces" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: traces" \
-H "X-P-Dataset-Tag: agent-observability"
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/hermes-logs" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: logs"
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/hermes-metrics" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: metrics"
```
Parseable lists the tagged `hermes-traces` dataset on the Agents page.
### Install the plugin
Install `hermes-otel` with the Hermes plugin manager:
```bash
hermes plugins install briancaffey/hermes-otel
```
Hermes places the plugin at `~/.hermes/plugins/hermes_otel/`. Install the plugin and its OpenTelemetry dependencies into the Python environment that runs Hermes:
```bash
HERMES_PYTHON="$(dirname "$(command -v hermes)")/python"
uv pip install --python "$HERMES_PYTHON" \
-e ~/.hermes/plugins/hermes_otel
```
If your Hermes installation uses a named virtual environment, set `HERMES_PYTHON` to that environment's Python executable.
### Configure direct OTLP export
Copy the plugin configuration template:
```bash
cp ~/.hermes/plugins/hermes_otel/config.yaml.example \
~/.hermes/plugins/hermes_otel/config.yaml
chmod 600 ~/.hermes/plugins/hermes_otel/config.yaml
```
Replace the file contents with the configuration below. Set the Parseable ingestor URL and API key before starting Hermes.
```yaml
enabled: true
project_name: hermes-agent
resource_attributes:
service.name: hermes-agent
deployment.environment.name: production
capture_previews: false
capture_conversation_history: false
capture_sender_id: false
capture_logs: true
log_level: INFO
emit_genai_metrics: true
force_flush_on_session_end: true
span_batch_export_timeout_ms: 30000
backends:
- type: otlp
name: parseable-traces
endpoint: https://parseable.example.com/v1/traces
headers:
Authorization: "Bearer "
X-P-Stream: hermes-traces
X-P-Dataset-Tag: agent-observability
traces: true
metrics: false
logs: false
- type: otlp
name: parseable-metrics
endpoint: https://parseable.example.com/v1/traces
headers:
Authorization: "Bearer "
X-P-Stream: hermes-metrics
traces: false
metrics: true
logs: false
- type: otlp
name: parseable-logs
endpoint: https://parseable.example.com/v1/traces
headers:
Authorization: "Bearer "
X-P-Stream: hermes-logs
traces: false
metrics: false
logs: true
```
`hermes-otel` derives the metrics and logs OTLP paths from each backend's traces endpoint. Keep `/v1/traces` on all three entries.
The generic OTLP backend stores custom headers in `config.yaml`. Keep the file out of source control and restrict it to the Hermes account. Render the file from your secret manager at startup when you run Hermes in production.
This configuration disables prompt, response, conversation-history, and sender-ID capture. Enable those fields after you define redaction, access, and retention rules for message content.
### Run an agent invocation
Start Hermes through your usual CLI or gateway path. For a CLI check, run:
```bash
hermes chat \
--provider openai-api \
--toolsets terminal \
--max-turns 4 \
-q "Use the terminal tool once to print HERMES_OTEL_OK, then return the output."
```
Use the provider and model configured for your Hermes profile. The invocation should create one `agent` span with nested `llm.*`, `api.*`, and `tool.*` spans.
### Verify ingestion
Run several Hermes invocations, then open **Traces** in Parseable and select `hermes-traces`. Open a recent trace and confirm that it contains:
* one root `agent` span;
* child `llm.*` and `api.*` spans for model activity;
* `tool.*` and `skill.*` spans when the invocation used tools or skills;
* model, token, duration, status, trace ID, and parent span fields.
Open **Agents**, select `hermes-traces`, and use the **Overview**, **Models**, **Tools**, and **Agent Runs** tabs to verify the aggregated agent telemetry. The Overview should show agent runs, errors, total tokens, LLM calls, tool calls, cost, and latency charts.
Check the other signal datasets from their matching Parseable views:
| Dataset | Records to check |
| ---------------- | ----------------------------------------------------------------------------------------- |
| `hermes-traces` | `agent`, `llm.*`, `api.*`, `tool.*`, and `skill.*` spans |
| `hermes-logs` | Hermes Python logs with severity, service name, and trace context when available |
| `hermes-metrics` | `hermes.*` and `gen_ai.*` token, duration, session, model, tool, retry, and error metrics |
## Import the dashboard
The [Hermes Agent Observability dashboard](https://github.com/parseablehq/dashboards/tree/main/hermes-agent-observability) combines the three datasets in one view. Its 12 panels cover agent invocations, token usage, model calls, tool outcomes, latency, errors, recent traces, logs, and native metrics.
Download the [dashboard JSON](https://github.com/parseablehq/dashboards/blob/main/hermes-agent-observability/hermes-agent-observability-sql.json) and import it into Parseable. Map its variables to `hermes-traces`, `hermes-logs`, and `hermes-metrics`.
## Content capture
`hermes-otel` can attach prompts, responses, conversation history, tool input and output previews, and sender IDs to telemetry. Those fields can contain credentials, personal data, or proprietary text.
Keep these settings disabled for production until your team approves the data policy:
```yaml
capture_previews: false
capture_conversation_history: false
capture_sender_id: false
```
You can still query token counts, model names, durations, tool names, statuses, and trace relationships after you disable preview capture.
## Troubleshooting
* **Hermes does not load the plugin:** Run `hermes plugins list`, confirm that `hermes_otel` shows `enabled`, and check that the plugin directory contains `plugin.yaml`.
* **OpenTelemetry imports fail:** Install the plugin into the same Python environment that provides the `hermes` command.
* **OTLP requests return 404 or 405:** Point the backend entries at the Parseable ingestor. Keep `/v1/traces` in each configured endpoint.
* **Few traces arrive from short CLI runs:** Keep `force_flush_on_session_end: true` and raise `span_batch_export_timeout_ms` to `30000`.
* **Traces arrive but logs do not:** Set `capture_logs: true`, keep a log-capable OTLP backend entry, and verify the backend sends logs to `/v1/logs`.
* **Agent Observability reports missing message fields:** Enable the content fields required by your use case after reviewing their privacy impact. The dashboard still works with content capture disabled.
* **Token totals look doubled:** Sum token attributes from the root `agent` spans or from the `api.*` spans. Do not add both sets together.
## Related resources
* [Hermes Agent documentation](https://hermes-agent.nousresearch.com/)
* [`hermes-otel` plugin](https://github.com/briancaffey/hermes-otel)
* [Hermes Agent Observability dashboard](https://github.com/parseablehq/dashboards/tree/main/hermes-agent-observability)
* [Send OpenTelemetry traces to Parseable](/ingest-data/otel/traces)
* [Send OpenTelemetry logs to Parseable](/ingest-data/otel/logs)
* [Send OpenTelemetry metrics to Parseable](/ingest-data/otel/metrics)
# AI Infrastructure (/ingest-data/ai-agents)
AI systems are usually made of more than one model call. A request can move through an agent framework, a gateway, a sandbox, a workflow runtime, a hosted model API, or a self-hosted inference server before the user sees the final answer.
This section groups Parseable integrations by the part of the AI stack they observe. Use the framework guides when you want agent, model, and tool-call traces. Use the sandbox and runtime guides when you need visibility into execution environments and long-running workflows. Use the LLM and gateway guides when the main question is about model traffic, token usage, latency, and errors.
## Agent frameworks
Agent frameworks usually carry the highest-level application context. They can show which agent ran, which prompt or task started the run, which model was called, which tool was used, and where the run failed or slowed down.
}>
Observe CrewAI crews, agents, tasks, LLM calls, and errors.
}>
Trace durable agent execution, model calls, tools, retries, and recovery.
}>
Send Mastra agent logs, traces, and metrics to Parseable.
}>
Observe n8n workflow automation through traces and related telemetry.
}>
Observe Hermes Agent sessions and tool activity.
}>
Capture agent, model, and tool-call telemetry from Pydantic AI applications.
}>
Send OpenLIT traces and metrics to Parseable.
}>
Send LangChain application telemetry to Parseable.
}>
Observe LlamaIndex retrieval and generation workflows.
}>
Track multi-agent AutoGen activity and failures.
}>
Monitor DSPy programs and prompt optimization workflows.
## Coding assistants
Coding assistants emit agent-level telemetry from interactive development sessions. Use these guides to inspect model calls, token usage, tool execution, latency, errors, and session activity.
}>
Export Claude Code events, metrics and optional traces to Parseable.
}>
Export GitHub Copilot agent traces to Parseable.
}>
Export Codex logs, traces and metrics directly to Parseable.
## Sandboxes
Sandboxes show what happens when agents execute code and jobs in isolated environments. These guides cover sandbox activity, logs, traces, and metrics.
}>
Capture SDK traces and sandbox logs, traces, and metrics.
}>
Send serverless Python logs, traces, and metrics through OpenTelemetry.
## Agent runtimes
Agent runtimes carry work across steps, retries, and long-running workflows. Use these guides to follow execution and diagnose failures across the workflow lifecycle.
}>
Observe Temporal workflows, activities, SDK telemetry, and metrics.
}>
Observe durable workflows with logs, traces, and Conductor metrics.
}>
Send durable execution workflow traces and service logs to Parseable.
## LLMs
LLM integrations help when you want to observe model-facing traffic directly. This can mean hosted provider APIs, SDK-level calls, or self-hosted inference runtimes such as vLLM.
}>
Observe LiteLLM SDK usage from application code.
}>
Log OpenAI API calls, responses, latency, and token usage.
}>
Capture Claude API calls, responses, errors, and token usage.
}>
Send Baseten deployment logs, traces, and metrics to Parseable.
}>
Monitor self-hosted vLLM inference serving.
## Gateways
Gateways sit between applications and model providers. They are useful when teams want one place to manage routing, keys, budgets, policies, fallbacks, and model-provider telemetry.
}>
Send OpenRouter traces and dashboard telemetry to Parseable.
}>
Observe LiteLLM Gateway traces and Prometheus metrics.
}>
Observe Cloudflare AI Gateway traces, model usage, latency, cost, and errors.
# LangChain (/ingest-data/ai-agents/langchain)
Trace LangChain chains, agents, and tool calls with Parseable for comprehensive LLM observability.
## Overview
Integrate LangChain with Parseable to:
* **Chain Tracing** - Track entire chain executions
* **Agent Monitoring** - Monitor agent decisions and tool usage
* **Token Tracking** - Aggregate token usage across chains
* **Latency Analysis** - Identify slow components
* **Error Debugging** - Trace failures through chains
## Prerequisites
* LangChain installed
* Parseable instance accessible
* Python application
## Custom Callback Handler
### Basic Handler
```python
from langchain.callbacks.base import BaseCallbackHandler
from langchain.schema import LLMResult
import requests
from datetime import datetime
from typing import Dict, Any, List, Optional
import uuid
class ParseableCallbackHandler(BaseCallbackHandler):
def __init__(self, parseable_url: str, dataset: str, username: str, password: str):
self.parseable_url = parseable_url
self.dataset = dataset
self.auth = (username, password)
self.run_id = str(uuid.uuid4())
self.chain_stack = []
def _log(self, entry: Dict[str, Any]):
entry["run_id"] = self.run_id
entry["timestamp"] = datetime.utcnow().isoformat() + "Z"
try:
requests.post(
f"{self.parseable_url}/api/v1/ingest",
json=[entry],
auth=self.auth,
headers={"X-P-Stream": self.dataset},
timeout=5
)
except Exception as e:
print(f"Logging failed: {e}")
def on_chain_start(self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs):
chain_id = str(uuid.uuid4())
self.chain_stack.append({
"chain_id": chain_id,
"start_time": datetime.utcnow(),
"name": serialized.get("name", "unknown")
})
self._log({
"event": "chain_start",
"chain_id": chain_id,
"chain_name": serialized.get("name"),
"inputs": str(inputs)[:500]
})
def on_chain_end(self, outputs: Dict[str, Any], **kwargs):
if self.chain_stack:
chain = self.chain_stack.pop()
duration = (datetime.utcnow() - chain["start_time"]).total_seconds() * 1000
self._log({
"event": "chain_end",
"chain_id": chain["chain_id"],
"chain_name": chain["name"],
"duration_ms": duration,
"outputs": str(outputs)[:500]
})
def on_chain_error(self, error: Exception, **kwargs):
if self.chain_stack:
chain = self.chain_stack.pop()
self._log({
"event": "chain_error",
"chain_id": chain["chain_id"],
"chain_name": chain["name"],
"error": str(error),
"error_type": type(error).__name__
})
def on_llm_start(self, serialized: Dict[str, Any], prompts: List[str], **kwargs):
self._log({
"event": "llm_start",
"model": serialized.get("name", "unknown"),
"prompt_count": len(prompts),
"prompt_preview": prompts[0][:200] if prompts else None
})
def on_llm_end(self, response: LLMResult, **kwargs):
usage = {}
if response.llm_output:
token_usage = response.llm_output.get("token_usage", {})
usage = {
"prompt_tokens": token_usage.get("prompt_tokens"),
"completion_tokens": token_usage.get("completion_tokens"),
"total_tokens": token_usage.get("total_tokens")
}
self._log({
"event": "llm_end",
"generations": len(response.generations),
**usage
})
def on_llm_error(self, error: Exception, **kwargs):
self._log({
"event": "llm_error",
"error": str(error),
"error_type": type(error).__name__
})
def on_tool_start(self, serialized: Dict[str, Any], input_str: str, **kwargs):
self._log({
"event": "tool_start",
"tool_name": serialized.get("name", "unknown"),
"input": input_str[:500]
})
def on_tool_end(self, output: str, **kwargs):
self._log({
"event": "tool_end",
"output": output[:500]
})
def on_tool_error(self, error: Exception, **kwargs):
self._log({
"event": "tool_error",
"error": str(error),
"error_type": type(error).__name__
})
def on_agent_action(self, action, **kwargs):
self._log({
"event": "agent_action",
"tool": action.tool,
"tool_input": str(action.tool_input)[:500],
"log": action.log[:500]
})
def on_agent_finish(self, finish, **kwargs):
self._log({
"event": "agent_finish",
"output": str(finish.return_values)[:500]
})
```
### Usage with Chains
```python
from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
# Create handler
handler = ParseableCallbackHandler(
parseable_url="http://parseable:8000",
dataset="langchain-traces",
username="admin",
password="admin"
)
# Create chain with callback
llm = ChatOpenAI(model="gpt-4", callbacks=[handler])
prompt = PromptTemplate.from_template("Tell me about {topic}")
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])
# Run chain
result = chain.run(topic="machine learning")
```
### Usage with Agents
```python
from langchain.agents import initialize_agent, Tool, AgentType
from langchain.chat_models import ChatOpenAI
# Define tools
tools = [
Tool(
name="Calculator",
func=lambda x: eval(x),
description="Useful for math calculations"
)
]
# Create agent with callback
llm = ChatOpenAI(model="gpt-4")
agent = initialize_agent(
tools,
llm,
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
callbacks=[handler],
verbose=True
)
# Run agent
result = agent.run("What is 25 * 4 + 10?")
```
## LCEL Integration
```python
from langchain_core.runnables import RunnableConfig
# Create config with callbacks
config = RunnableConfig(callbacks=[handler])
# Use with LCEL chains
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
prompt = ChatPromptTemplate.from_template("Tell me about {topic}")
model = ChatOpenAI()
chain = prompt | model
# Invoke with config
result = chain.invoke({"topic": "AI"}, config=config)
```
## Querying LangChain Traces
```sql
-- Chain execution times
SELECT
chain_name,
AVG(duration_ms) as avg_duration,
COUNT(*) as executions
FROM "langchain-traces"
WHERE event = 'chain_end'
GROUP BY chain_name
ORDER BY avg_duration DESC
-- Token usage by run
SELECT
run_id,
SUM(total_tokens) as total_tokens,
COUNT(CASE WHEN event = 'tool_start' THEN 1 END) as tool_calls
FROM "langchain-traces"
GROUP BY run_id
ORDER BY total_tokens DESC
-- Agent tool usage
SELECT
tool,
COUNT(*) as usage_count
FROM "langchain-traces"
WHERE event = 'agent_action'
GROUP BY tool
ORDER BY usage_count DESC
-- Error analysis
SELECT
event,
error_type,
error,
COUNT(*) as count
FROM "langchain-traces"
WHERE event LIKE '%error%'
GROUP BY event, error_type, error
ORDER BY count DESC
```
## Best Practices
1. **Use Run IDs** - Correlate events in a single execution
2. **Truncate Content** - Don't log full prompts/responses
3. **Track All Events** - Log starts, ends, and errors
4. **Monitor Agents** - Pay attention to tool usage patterns
## Next Steps
* Configure [LlamaIndex](/ingest-data/ai-agents/llamaindex) tracing
* Set up [OpenAI](/ingest-data/ai-agents/openai) direct logging
* Create [dashboards](/user-guide/dashboards) for chain metrics
# LiteLLM SDK (/ingest-data/ai-agents/litellm-sdk)
import { Step, Steps } from 'fumadocs-ui/components/steps';
The LiteLLM Python SDK lets your application call models from OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, and other providers through one interface. Use this guide when your Python application imports `litellm` directly and you want to observe those model calls without running the LiteLLM Gateway.
Parseable receives the SDK telemetry over OpenTelemetry. Traces show each model call and how it sits inside your application flow. Logs show the GenAI events emitted by the LiteLLM OpenTelemetry integration. Metrics show request duration, token usage, cost, and streaming latency as OpenTelemetry histograms.
For the standalone gateway or proxy setup, see [LiteLLM Gateway](/ingest-data/gateways/litellm).
## How it works
In production, keep Parseable credentials in the OpenTelemetry Collector instead of putting them in application code. The application sends OTLP data to the Collector, and the Collector forwards each signal to a separate Parseable dataset.
```text
Python application using LiteLLM SDK
|
| OTLP traces, logs/events, and metrics
v
OpenTelemetry Collector
|
+--> litellm-sdk-traces traces dataset in Parseable
+--> litellm-sdk-logs logs dataset in Parseable
+--> litellm-sdk-metrics metrics dataset in Parseable
```
This is different from the gateway path. The SDK does not expose a `/metrics` endpoint for Prometheus scraping. Its traces, logs, and metrics come from the OpenTelemetry integration inside the Python process.
## Prerequisites
Before you start, keep these ready:
* A running Parseable instance
* A Parseable API key with ingest access
* Python 3.9 or newer
* A model-provider API key, such as `OPENAI_API_KEY`
* An OpenTelemetry Collector that your application can reach
## Set up LiteLLM SDK with Parseable
### Create Parseable datasets
Create one dataset for each signal. This keeps the views in Prism clean and lets Parseable understand whether the incoming records are logs, traces, or metrics.
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/litellm-sdk-traces" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: traces" \
-H "X-P-Dataset-Tag: agent-observability"
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/litellm-sdk-logs" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: logs"
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/litellm-sdk-metrics" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: metrics"
```
### Configure the OpenTelemetry Collector
Create `otel-collector-config.yaml`:
```yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch: {}
exporters:
otlphttp/parseable_traces:
endpoint: "${env:PARSEABLE_URL}"
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: litellm-sdk-traces
otlphttp/parseable_logs:
endpoint: "${env:PARSEABLE_URL}"
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: litellm-sdk-logs
otlphttp/parseable_metrics:
endpoint: "${env:PARSEABLE_URL}"
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: litellm-sdk-metrics
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_traces]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_logs]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_metrics]
```
Start the Collector with your Parseable endpoint and API key:
```bash
export PARSEABLE_URL="https://:8000"
export PARSEABLE_API_KEY=""
otelcol-contrib --config otel-collector-config.yaml
```
### Configure the Python application
Install LiteLLM and the OpenTelemetry packages:
```bash
pip install litellm \
opentelemetry-api \
opentelemetry-sdk \
opentelemetry-exporter-otlp
```
Then configure LiteLLM's OpenTelemetry callback. The callback is what emits the SDK spans, GenAI events, and metrics.
```python
import os
import litellm
from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig
os.environ["OPENAI_API_KEY"] = ""
os.environ["USE_OTEL_LITELLM_REQUEST_SPAN"] = "true"
litellm.callbacks = [
OpenTelemetry(
config=OpenTelemetryConfig(
exporter="otlp_http",
endpoint="http://localhost:4318",
enable_metrics=True,
enable_events=True,
capture_message_content="NO_CONTENT",
semconv_stability="gen_ai_latest_experimental",
)
)
]
response = litellm.completion(
model="openai/gpt-4o-mini",
messages=[
{
"role": "user",
"content": "Explain distributed tracing in one paragraph.",
}
],
metadata={
"scenario": "documentation_example",
"mask_input": True,
"mask_output": True,
},
)
print(response.choices[0].message.content)
```
`USE_OTEL_LITELLM_REQUEST_SPAN=true` gives each SDK call its own model-call span. This is useful when one application span makes more than one LiteLLM call, because each model call keeps its own attributes and duration.
The example uses `capture_message_content="NO_CONTENT"` and request-level masking. This keeps model, provider, token, latency, cost, and error metadata available while avoiding raw prompt and response capture.
### Send a few requests
Run the application a few times with different models, prompts, and outcomes. For streaming latency metrics, make at least one streaming request and consume the full response. LiteLLM records time-to-first-token and time-per-output-token only after the stream finishes.
## What you get in Parseable
Open `litellm-sdk-traces` from the Traces page. Each SDK call shows the provider, requested model, response model, token usage, duration, status, and error details when a call fails. If your application already has its own OpenTelemetry spans, LiteLLM can attach the model call under the current trace context, which makes it easier to connect the LLM call back to the user action or background job that triggered it.
Open `litellm-sdk-logs` from the Logs page. When `enable_events=True`, LiteLLM emits GenAI events as OTLP logs. In semantic-convention mode, those events use the consolidated `gen_ai.client.inference.operation.details` shape. With message content disabled, the log records still help you inspect the request lifecycle without storing raw prompts or completions.
Open `litellm-sdk-metrics` from the Metrics page. The SDK metrics are OpenTelemetry histograms, so the records include fields such as `data_point_sum`, `data_point_count`, and bucket data. Use them to build dashboards for latency, token usage, request volume, streaming behavior, and provider reliability.
## Useful fields
These are the fields you will usually start with while exploring the trace dataset:
| Field | Meaning |
| ---------------------------- | ------------------------------------------- |
| `gen_ai.operation.name` | The GenAI operation, such as `chat` |
| `gen_ai.provider.name` | The model provider |
| `gen_ai.request.model` | The model requested by the application |
| `gen_ai.response.model` | The model returned by the provider |
| `gen_ai.usage.input_tokens` | Input token count |
| `gen_ai.usage.output_tokens` | Output token count |
| `span_status_code` | Whether the SDK span completed successfully |
LiteLLM emits these GenAI metrics when metrics are enabled:
| Metric | Description |
| ------------------------------------- | ---------------------------------------------------------------- |
| `gen_ai.client.operation.duration` | End-to-end SDK operation duration |
| `gen_ai.client.token.usage` | Input and output token usage |
| `gen_ai.usage.cost` | Computed request cost when LiteLLM can calculate it |
| `gen_ai.server.time_to_first_token` | Time from request start to the first streamed token |
| `gen_ai.server.time_per_output_token` | Average generation time per output token for streaming responses |
| `gen_ai.client.response.duration` | Provider response-generation duration |
## Query SDK metrics
Because the SDK metrics are histograms, use `data_point_sum` for totals and `data_point_count` for sample counts.
Example total token usage:
```sql
SELECT
attributes->>'gen_ai.token.type' AS token_type,
SUM(data_point_sum) AS tokens
FROM "litellm-sdk-metrics"
WHERE metric_name = 'gen_ai.client.token.usage'
GROUP BY token_type;
```
Example average operation latency:
```sql
SELECT
SUM(data_point_sum) / NULLIF(SUM(data_point_count), 0) AS avg_latency_seconds
FROM "litellm-sdk-metrics"
WHERE metric_name = 'gen_ai.client.operation.duration';
```
Example total computed cost:
```sql
SELECT SUM(data_point_sum) AS total_cost_usd
FROM "litellm-sdk-metrics"
WHERE metric_name = 'gen_ai.usage.cost';
```
## SDK or Gateway
Use this SDK integration when the application calls model providers directly through `litellm.completion()` or related SDK methods. It gives you telemetry from inside that Python process.
Use [LiteLLM Gateway](/ingest-data/gateways/litellm) when traffic goes through the LiteLLM Proxy or Gateway. The gateway path adds proxy-level visibility such as authentication, routing, budgets, rate limits, Redis, PostgreSQL, and Prometheus `/metrics` scraping.
## Troubleshooting
* **Traces appear, but logs do not**
Confirm `enable_events=True` in `OpenTelemetryConfig`, and check that the Collector has a `logs` pipeline exporting to `litellm-sdk-logs`.
* **Metrics do not appear**
Confirm `enable_metrics=True` and check that the Collector has a `metrics` pipeline exporting to `litellm-sdk-metrics`.
* **Query reports `No field named data_point_value`**
LiteLLM SDK metrics are histograms. Use `data_point_sum`, `data_point_count`, and bucket fields instead of `data_point_value`.
* **Streaming metrics are missing**
Make sure the request uses streaming and the client consumes the stream until it ends. LiteLLM records streaming latency after the final chunk.
* **Prompt or response text appears in telemetry**
Keep `capture_message_content="NO_CONTENT"` and pass `metadata={"mask_input": True, "mask_output": True}` for requests that may contain sensitive content.
## See also
* [LiteLLM Gateway](/ingest-data/gateways/litellm)
* [Pydantic AI](/ingest-data/ai-agents/pydantic-ai)
* [Traces](/user-guide/traces)
* [Logs](/user-guide/logs)
* [Metrics](/user-guide/metrics)
# LlamaIndex (/ingest-data/ai-agents/llamaindex)
Trace LlamaIndex queries, retrievals, and LLM calls with Parseable for RAG observability.
## Overview
Integrate LlamaIndex with Parseable to:
* **Query Tracing** - Track query engine executions
* **Retrieval Monitoring** - Monitor document retrieval
* **LLM Tracking** - Log LLM calls and token usage
* **Index Operations** - Track indexing and embedding
* **RAG Debugging** - Debug retrieval-augmented generation
## Prerequisites
* LlamaIndex installed
* Parseable instance accessible
* Python application
## Custom Callback Handler
```python
from llama_index.core.callbacks import CallbackManager, CBEventType
from llama_index.core.callbacks.base_handler import BaseCallbackHandler
import requests
from datetime import datetime
from typing import Dict, Any, List, Optional
import uuid
class ParseableCallbackHandler(BaseCallbackHandler):
def __init__(self, parseable_url: str, dataset: str, username: str, password: str):
self.parseable_url = parseable_url
self.dataset = dataset
self.auth = (username, password)
self.event_starts = {}
super().__init__(
event_starts_to_ignore=[],
event_ends_to_ignore=[]
)
def _log(self, entry: Dict[str, Any]):
entry["timestamp"] = datetime.utcnow().isoformat() + "Z"
try:
requests.post(
f"{self.parseable_url}/api/v1/ingest",
json=[entry],
auth=self.auth,
headers={"X-P-Stream": self.dataset},
timeout=5
)
except Exception as e:
print(f"Logging failed: {e}")
def on_event_start(
self,
event_type: CBEventType,
payload: Optional[Dict[str, Any]] = None,
event_id: str = "",
parent_id: str = "",
**kwargs
) -> str:
self.event_starts[event_id] = datetime.utcnow()
log_entry = {
"event": "start",
"event_type": event_type.value,
"event_id": event_id,
"parent_id": parent_id
}
if payload:
if event_type == CBEventType.QUERY:
log_entry["query"] = str(payload.get("query_str", ""))[:500]
elif event_type == CBEventType.RETRIEVE:
log_entry["query"] = str(payload.get("query_str", ""))[:500]
elif event_type == CBEventType.LLM:
messages = payload.get("messages", [])
log_entry["message_count"] = len(messages)
log_entry["model"] = payload.get("model_name", "unknown")
elif event_type == CBEventType.EMBEDDING:
log_entry["chunk_count"] = len(payload.get("chunks", []))
self._log(log_entry)
return event_id
def on_event_end(
self,
event_type: CBEventType,
payload: Optional[Dict[str, Any]] = None,
event_id: str = "",
**kwargs
) -> None:
start_time = self.event_starts.pop(event_id, None)
duration_ms = None
if start_time:
duration_ms = (datetime.utcnow() - start_time).total_seconds() * 1000
log_entry = {
"event": "end",
"event_type": event_type.value,
"event_id": event_id,
"duration_ms": duration_ms
}
if payload:
if event_type == CBEventType.QUERY:
response = payload.get("response")
if response:
log_entry["response_preview"] = str(response)[:500]
elif event_type == CBEventType.RETRIEVE:
nodes = payload.get("nodes", [])
log_entry["nodes_retrieved"] = len(nodes)
log_entry["scores"] = [n.score for n in nodes[:5]] if nodes else []
elif event_type == CBEventType.LLM:
response = payload.get("response")
if response:
log_entry["response_preview"] = str(response)[:200]
# Token usage if available
if hasattr(response, "raw"):
usage = getattr(response.raw, "usage", None)
if usage:
log_entry["prompt_tokens"] = usage.prompt_tokens
log_entry["completion_tokens"] = usage.completion_tokens
elif event_type == CBEventType.EMBEDDING:
log_entry["embeddings_created"] = len(payload.get("embeddings", []))
self._log(log_entry)
def start_trace(self, trace_id: Optional[str] = None) -> None:
self.trace_id = trace_id or str(uuid.uuid4())
def end_trace(
self,
trace_id: Optional[str] = None,
trace_map: Optional[Dict[str, List[str]]] = None
) -> None:
pass
```
### Usage
```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.callbacks import CallbackManager
# Create handler
handler = ParseableCallbackHandler(
parseable_url="http://parseable:8000",
dataset="llamaindex-traces",
username="admin",
password="admin"
)
# Create callback manager
callback_manager = CallbackManager([handler])
# Load documents
documents = SimpleDirectoryReader("data").load_data()
# Create index with callbacks
index = VectorStoreIndex.from_documents(
documents,
callback_manager=callback_manager
)
# Query with callbacks
query_engine = index.as_query_engine(callback_manager=callback_manager)
response = query_engine.query("What is the main topic?")
```
## Global Handler Setup
```python
from llama_index.core import Settings
# Set global callback manager
Settings.callback_manager = CallbackManager([handler])
# All LlamaIndex operations will now be traced
```
## Querying LlamaIndex Traces
```sql
-- Query performance
SELECT
event_id,
duration_ms,
nodes_retrieved,
response_preview
FROM "llamaindex-traces"
WHERE event_type = 'query' AND event = 'end'
ORDER BY timestamp DESC
LIMIT 50
-- Retrieval quality
SELECT
AVG(nodes_retrieved) as avg_nodes,
AVG(scores[1]) as avg_top_score
FROM "llamaindex-traces"
WHERE event_type = 'retrieve' AND event = 'end'
-- LLM token usage
SELECT
model,
SUM(prompt_tokens) as total_prompt,
SUM(completion_tokens) as total_completion,
COUNT(*) as calls
FROM "llamaindex-traces"
WHERE event_type = 'llm' AND event = 'end'
GROUP BY model
-- Embedding operations
SELECT
DATE_TRUNC('hour', timestamp) as hour,
SUM(embeddings_created) as embeddings,
COUNT(*) as operations
FROM "llamaindex-traces"
WHERE event_type = 'embedding' AND event = 'end'
GROUP BY hour
ORDER BY hour DESC
```
## Best Practices
1. **Track All Events** - Monitor queries, retrievals, and LLM calls
2. **Log Retrieval Scores** - Analyze retrieval quality
3. **Monitor Token Usage** - Track costs across operations
4. **Use Parent IDs** - Correlate nested events
## Next Steps
* Configure [LangChain](/ingest-data/ai-agents/langchain) tracing
* Set up [OpenAI](/ingest-data/ai-agents/openai) direct logging
* Create [dashboards](/user-guide/dashboards) for RAG metrics
# Mastra (/ingest-data/ai-agents/mastra)
import { Step, Steps } from 'fumadocs-ui/components/steps';
[Mastra](https://mastra.ai/) is a TypeScript framework for building AI agents and workflows. Its observability event bus captures agent, model, and tool activity, including latency, token usage, errors, and correlated logs.
This guide sends each OpenTelemetry signal directly to a dedicated Parseable dataset:
```text
Mastra agent
|
| traces -> OTLP/HTTP -> mastra-traces
| logs -> OTLP/HTTP -> mastra-logs
| metrics -> OTLP/HTTP -> mastra-metrics
|
v
Parseable datasets
```
Traces and logs are sent through Mastra's OpenTelemetry exporter. Metrics come from Mastra observability events and are sent to the metrics dataset using OpenTelemetry metrics.
Keeping the signals in separate datasets makes retention, access control, and dashboard queries easier to manage.
## Prerequisites
* Node.js 22.13 or later
* An existing Mastra application
* A Parseable instance reachable from the application
* A Parseable API key with ingest access
* An LLM provider API key for the agent
Use the Parseable ingestor base URL, for example `https://parseable.example.com`. The exporters append the appropriate OTLP signal path.
In a distributed Parseable deployment, the query/UI endpoint might not accept ingestion. Use the ingestor URL returned by the Parseable About API or the URL configured for your ingestor.
## Set up Mastra with Parseable
### Install the packages
From the Mastra project, install the observability and OpenTelemetry metric packages:
```bash
npm install \
@mastra/observability \
@mastra/otel-exporter \
@opentelemetry/exporter-metrics-otlp-http \
@opentelemetry/resources \
@opentelemetry/sdk-metrics
```
The Mastra OpenTelemetry exporter handles traces and logs. The OpenTelemetry metrics packages are used to send Mastra metric events to Parseable.
The example agent below uses OpenAI. Install the provider package if it is not already present:
```bash
npm install @ai-sdk/openai
```
### Configure the environment
Add these variables to the application environment:
```bash
export PARSEABLE_OTLP_ENDPOINT="https://parseable.example.com"
export PARSEABLE_API_KEY=""
export OPENAI_API_KEY=""
export OPENAI_MODEL="gpt-5-mini"
```
Do not commit API keys to source control.
### Configure Mastra observability
Create `src/mastra/observability.ts`:
```ts
import type { MetricEvent, TracingEvent } from '@mastra/core/observability';
import { BaseExporter, Observability } from '@mastra/observability';
import { OtelExporter } from '@mastra/otel-exporter';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { resourceFromAttributes } from '@opentelemetry/resources';
import {
MeterProvider,
PeriodicExportingMetricReader,
type Counter,
type Histogram,
} from '@opentelemetry/sdk-metrics';
const endpoint = process.env.PARSEABLE_OTLP_ENDPOINT?.replace(/\/$/, '');
const apiKey = process.env.PARSEABLE_API_KEY;
if (!endpoint || !apiKey) {
throw new Error('PARSEABLE_OTLP_ENDPOINT and PARSEABLE_API_KEY are required');
}
const serviceName = 'mastra-agent';
const resourceAttributes = {
'service.name': serviceName,
'service.version': '1.0.0',
'deployment.environment': process.env.NODE_ENV ?? 'development',
};
const headers = (stream: string) => ({
'Authorization': `Bearer ${apiKey}`,
'X-P-Stream': stream,
});
const traceExporter = new OtelExporter({
provider: {
custom: {
endpoint,
protocol: 'http/json',
headers: headers('mastra-traces'),
},
},
signals: { traces: true, logs: false },
resourceAttributes,
});
const logExporter = new OtelExporter({
provider: {
custom: {
endpoint,
protocol: 'http/json',
headers: headers('mastra-logs'),
},
},
signals: { traces: false, logs: true },
resourceAttributes,
});
class ParseableMetricExporter extends BaseExporter {
name = 'parseable-otlp-metrics';
private readonly meterProvider: MeterProvider;
private readonly meter;
private readonly histograms = new Map();
private readonly counters = new Map();
// Optional application metric used by the dashboard's workload panel.
readonly workloadRequests;
constructor() {
super();
const metricExporter = new OTLPMetricExporter({
url: `${endpoint}/v1/metrics`,
headers: headers('mastra-metrics'),
});
const reader = new PeriodicExportingMetricReader({
exporter: metricExporter,
exportIntervalMillis: 5_000,
exportTimeoutMillis: 4_000,
});
this.meterProvider = new MeterProvider({
resource: resourceFromAttributes(resourceAttributes),
readers: [reader],
});
this.meter = this.meterProvider.getMeter('@mastra/observability');
this.workloadRequests = this.meter.createCounter(
'mastra_workload_requests_total',
{ description: 'Number of Mastra workload requests' },
);
}
protected async _exportTracingEvent(_event: TracingEvent): Promise {}
async onMetricEvent(event: MetricEvent): Promise {
const metric = event.metric;
const attributes: Record = {
...metric.labels,
'service.name': serviceName,
};
if (metric.costContext?.provider) {
attributes.provider = metric.costContext.provider;
}
if (metric.costContext?.model) {
attributes.model = metric.costContext.model;
}
if (metric.costContext?.costUnit) {
attributes.cost_unit = metric.costContext.costUnit;
}
if (metric.name.endsWith('_duration_ms')) {
let histogram = this.histograms.get(metric.name);
if (!histogram) {
histogram = this.meter.createHistogram(metric.name, { unit: 'ms' });
this.histograms.set(metric.name, histogram);
}
histogram.record(metric.value, attributes);
return;
}
let counter = this.counters.get(metric.name);
if (!counter) {
counter = this.meter.createCounter(metric.name);
this.counters.set(metric.name, counter);
}
counter.add(metric.value, attributes);
}
async flush(): Promise {
await this.meterProvider.forceFlush({ timeoutMillis: 20_000 });
}
async shutdown(): Promise {
await this.meterProvider.shutdown({ timeoutMillis: 20_000 });
}
}
export const parseableMetrics = new ParseableMetricExporter();
export const observability = new Observability({
configs: {
default: {
serviceName,
exporters: [traceExporter, logExporter, parseableMetrics],
includeInternalSpans: true,
logging: { enabled: true, level: 'info' },
serializationOptions: {
maxStringLength: 4_096,
maxArrayLength: 50,
maxObjectKeys: 100,
},
},
},
});
```
Mastra emits native metric events through its observability bus. The custom exporter above converts duration events to OpenTelemetry histograms and other numeric events to counters before sending them to `mastra-metrics`. Default cardinality protection prevents identifiers such as trace IDs and request IDs from becoming metric labels.
Typical native metric names include:
* `mastra_agent_duration_ms`
* `mastra_tool_duration_ms`
* `mastra_model_duration_ms`
* `mastra_model_total_input_tokens`
* `mastra_model_total_output_tokens`
* `mastra_model_output_reasoning_tokens`
### Register the agent
Use the observability instance when creating Mastra. For example, in `src/mastra/index.ts`:
```ts
import { openai } from '@ai-sdk/openai';
import { Agent } from '@mastra/core/agent';
import { Mastra } from '@mastra/core/mastra';
import { observability, parseableMetrics } from './observability';
const supportAgent = new Agent({
id: 'support-agent',
name: 'Support Agent',
instructions: 'Answer support questions and use available tools when needed.',
model: openai(process.env.OPENAI_MODEL ?? 'gpt-5-mini'),
});
export const mastra = new Mastra({
agents: { supportAgent },
observability,
});
// Optional: record application workload outcomes for the dashboard.
export function recordWorkload(status: 'started' | 'completed' | 'failed') {
parseableMetrics.workloadRequests.add(1, {
agent: 'support-agent',
status,
});
}
```
Agent runs, model calls, tool calls, token use, failures, and duration events are recorded automatically. Application-specific counters and structured logs can add business context such as a scenario, tenant, or outcome.
For a short-lived process or serverless function, call `await observability.flush()` before the process exits. Call `await observability.shutdown()` during final application shutdown so buffered telemetry is delivered.
### Verify ingestion
Run the application and invoke the agent a few times. In Parseable, confirm that all three datasets receive recent events:
| Dataset | What to verify |
| ---------------- | -------------------------------------------------------------------- |
| `mastra-traces` | Agent, model, and tool spans with trace and span IDs |
| `mastra-logs` | Structured Mastra logs with severity, service, and trace correlation |
| `mastra-metrics` | Duration histograms and model token counters |
The traces view should show agent, model, and tool spans for each run.
The logs view should show structured agent logs with the same service and trace context.
The metrics view should show duration and token metrics emitted from Mastra's observability events.
Useful PromQL checks for the metrics dataset include:
```text
sum(increase(mastra_agent_duration_ms_count[5m]))
```
```text
histogram_quantile(
0.95,
sum by (le) (rate(mastra_agent_duration_ms_bucket[5m]))
)
```
If a dataset remains empty, check the application logs for OTLP errors and verify the endpoint, API key, and `X-P-Stream` header.
## Import the dashboard
The [Mastra Agent Observability dashboard](https://github.com/parseablehq/dashboards/tree/main/mastra-agent-observability) includes panels for:
* Overview
* Agent Activity & Reliability
* Models, Tokens & Cost
* Tools
* Performance & Latency
* Logs & Errors
* Trace Explorer
* Metrics & Telemetry
Download the [Mastra dashboard JSON](https://github.com/parseablehq/dashboards/blob/main/mastra-agent-observability/mastra-agent-observability-mixed.json) and import it into Parseable. The template has three dataset variables only. Map them to `mastra-traces`, `mastra-logs`, and `mastra-metrics`, or to your chosen dataset names.
Cost panels use Parseable's `agent_cost()` function with the provider and model recorded on model spans. Unsupported provider/model combinations do not produce an estimate, so keep the model identifier and provider attributes intact during ingestion.
The dashboard gives a ready-made view for token usage, latency, and error investigation once the three datasets are mapped.
## Troubleshooting
* **OTLP requests return 404 or 405**: The application is probably sending telemetry to a query/UI node instead of an ingestor. Change `PARSEABLE_OTLP_ENDPOINT` to the Parseable ingestor base URL.
* **Traces arrive but logs do not**: Keep `logging.enabled` set to `true`, include the log exporter, and verify that its signal configuration is `traces: false, logs: true`.
* **Metrics do not arrive**: Register the metric exporter in the same observability configuration as the trace and log exporters. Keep the metric export interval greater than its export timeout, and flush telemetry before a short-lived process exits.
* **Metric series are too numerous**: Do not add trace IDs, request IDs, user IDs, session IDs, or other unbounded values as metric labels. Keep those values on logs and traces, where they can still be used for correlation.
## Related resources
* [Mastra observability](https://mastra.ai/ai-agent-observability)
* [Mastra OpenTelemetry exporter](https://mastra.ai/docs/observability/integrations/exporters/otel)
* [Send OpenTelemetry traces to Parseable](/ingest-data/otel/traces)
* [Send OpenTelemetry logs to Parseable](/ingest-data/otel/logs)
* [Send OpenTelemetry metrics to Parseable](/ingest-data/otel/metrics)
* [Mastra Agent Observability dashboard](https://github.com/parseablehq/dashboards/tree/main/mastra-agent-observability)
# Modal (/ingest-data/ai-agents/modal)
[Modal](https://modal.com/) is a serverless platform for running Python workloads such as AI inference, batch jobs, web endpoints, and sandboxed code. When those workloads move fast or scale across many containers, the useful question is usually simple: what ran, how long did it take, and what happened inside the function when something failed?
Parseable can receive Modal telemetry through OpenTelemetry. In this setup, Modal sends telemetry to an OpenTelemetry Collector, and the Collector forwards logs, traces, and metrics into separate Parseable datasets.
## How it works
There are two useful telemetry paths for Modal.
The first path is Modal's managed OpenTelemetry integration. This is the easiest way to export Modal Function logs and container metrics. Audit logs can also be exported when they are available for your Modal workspace plan.
The second path is application-level OpenTelemetry from your Modal code. Use this when you want custom spans around important work, structured application logs, or metrics that are specific to your own workload.
Both paths send OTLP over HTTP to the same Collector. The Collector keeps the Parseable API key outside your Modal code, receives telemetry from Modal with a bearer token, and forwards each signal to a dedicated Parseable dataset.
```text
Modal managed telemetry ---\
+--> OTLP/HTTP --> OpenTelemetry Collector
Modal app OpenTelemetry ----/ |
+--> modal-logs
+--> modal-traces
+--> modal-metrics
Parseable
```
OTLP means OpenTelemetry Protocol. Modal sends telemetry in OpenTelemetry format, and Parseable stores it in datasets that you can query, filter, and explore.
## Prerequisites
Before you start, keep these ready:
* A Modal account and workspace
* Python 3.10 through 3.13
* The Modal CLI installed locally
* A Parseable instance reachable from the Collector
* A Parseable API key with ingest access
* An OpenTelemetry Collector reachable from Modal over HTTPS
* Docker, if you want to run the Collector with the example below
Authenticate the Modal CLI first:
```bash
python3.12 -m venv .venv
source .venv/bin/activate
pip install modal
modal setup
modal app list
```
For a new workspace, `modal app list` can return an empty list. That is fine. The important part is that the command completes without an authentication error.
## Configure the OpenTelemetry Collector
Create `otel-collector-config.yaml`:
```yaml
extensions:
health_check:
endpoint: 0.0.0.0:13133
bearertokenauth/ingress:
token: ${env:OTLP_INGEST_TOKEN}
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
auth:
authenticator: bearertokenauth/ingress
processors:
memory_limiter:
check_interval: 1s
limit_mib: 384
spike_limit_mib: 96
batch:
timeout: 5s
send_batch_size: 512
exporters:
otlphttp/parseable_logs:
endpoint: ${env:PARSEABLE_ENDPOINT}
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: ${env:PARSEABLE_LOGS_STREAM}
Content-Type: application/json
otlphttp/parseable_traces:
endpoint: ${env:PARSEABLE_ENDPOINT}
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: ${env:PARSEABLE_TRACES_STREAM}
Content-Type: application/json
otlphttp/parseable_metrics:
endpoint: ${env:PARSEABLE_ENDPOINT}
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: ${env:PARSEABLE_METRICS_STREAM}
Content-Type: application/json
service:
extensions: [health_check, bearertokenauth/ingress]
pipelines:
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/parseable_logs]
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/parseable_traces]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/parseable_metrics]
```
This Collector accepts OTLP/HTTP on port `4318`. Modal authenticates to the Collector with `OTLP_INGEST_TOKEN`, and the Collector authenticates to Parseable with `PARSEABLE_API_KEY`.
Configure the Collector environment:
```bash
export PARSEABLE_ENDPOINT="https://parseable.example.com"
export PARSEABLE_API_KEY=""
export PARSEABLE_LOGS_STREAM="modal-logs"
export PARSEABLE_TRACES_STREAM="modal-traces"
export PARSEABLE_METRICS_STREAM="modal-metrics"
export OTLP_INGEST_TOKEN="$(openssl rand -hex 32)"
```
Run the Collector:
```bash
docker run --rm \
-p 127.0.0.1:4318:4318 \
-p 127.0.0.1:13133:13133 \
-e PARSEABLE_ENDPOINT \
-e PARSEABLE_API_KEY \
-e PARSEABLE_LOGS_STREAM \
-e PARSEABLE_TRACES_STREAM \
-e PARSEABLE_METRICS_STREAM \
-e OTLP_INGEST_TOKEN \
-v "$PWD/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro" \
otel/opentelemetry-collector-contrib:0.130.1 \
--config=/etc/otelcol-contrib/config.yaml
```
Modal must be able to reach the Collector over HTTPS. For production, run the Collector behind a stable TLS endpoint such as `https://otel.example.com`.
For a short local proof of concept, you can put a temporary tunnel in front of the local Collector:
```bash
cloudflared tunnel --url http://localhost:4318
```
Do not expose an unauthenticated OTLP receiver to the internet. The example protects Collector ingress with a bearer token. Use a durable Collector deployment and a stable TLS hostname for production.
## Create the Modal Secret
Create a Modal Secret named `parseable-otel`. This Secret is used in two places: Modal's managed OpenTelemetry integration and the sample Modal application later in this guide.
In the [Modal Secrets dashboard](https://modal.com/secrets), add these keys:
| Key | Value |
| ----------------------------- | ------------------------------------------------------ |
| `OTEL_HEADER_Authorization` | `Bearer ` |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector base URL, such as `https://otel.example.com` |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` |
| `OTEL_EXPORTER_OTLP_HEADERS` | `Authorization=Bearer%20` |
`OTEL_HEADER_Authorization` is used by Modal's managed OpenTelemetry integration. The `OTEL_EXPORTER_*` variables are picked up by OpenTelemetry SDKs running inside your Modal functions. In `OTEL_EXPORTER_OTLP_HEADERS`, `%20` is the encoded space between `Bearer` and the token.
You can also create the Secret from the CLI:
```bash
modal secret create parseable-otel \
OTEL_HEADER_Authorization="Bearer ${OTLP_INGEST_TOKEN}" \
OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.example.com" \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20${OTLP_INGEST_TOKEN}"
```
Only the Collector needs the Parseable API key. Modal only needs the Collector ingress token.
## Enable Modal managed telemetry
In the Modal dashboard:
1. Open **Settings > Metrics > OpenTelemetry**.
2. Enter the Collector base URL, such as `https://otel.example.com`.
3. Do not append `/v1/logs`, `/v1/traces`, or `/v1/metrics`.
4. Select the `parseable-otel` Secret.
5. Save the integration.
6. Use **Send test** to confirm the path is working.
A successful test sends a log record from the `modal.test_logs` service. Open the `modal-logs` dataset in Parseable and search for that service name.
Modal's managed integration exports Function logs and container metrics. Audit-log export depends on your Modal plan. Modal documents platform metrics such as CPU utilization, memory usage, GPU usage, running containers, input latency, pending inputs, running inputs, and success counts.
## Add custom application telemetry
Managed telemetry is useful for the platform view. Application telemetry is useful when you want to see what your own code did inside a Modal function.
The following Modal class emits two log records, two spans, a counter, and a histogram. It also flushes the OpenTelemetry providers during container shutdown, which matters because Modal containers can scale down or be preempted.
```python
import logging
import time
import modal
app = modal.App("parseable-otel-demo")
image = modal.Image.debian_slim(python_version="3.12").pip_install(
"opentelemetry-api==1.36.0",
"opentelemetry-sdk==1.36.0",
"opentelemetry-exporter-otlp-proto-http==1.36.0",
)
@app.cls(
image=image,
secrets=[modal.Secret.from_name("parseable-otel")],
)
class InstrumentedWorker:
@modal.enter()
def setup_telemetry(self):
from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
resource = Resource.create({"service.name": "modal-parseable-demo"})
self.tracer_provider = TracerProvider(resource=resource)
self.tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(self.tracer_provider)
self.logger_provider = LoggerProvider(resource=resource)
self.logger_provider.add_log_record_processor(
BatchLogRecordProcessor(OTLPLogExporter())
)
set_logger_provider(self.logger_provider)
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(), export_interval_millis=5_000
)
self.meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(self.meter_provider)
self.tracer = trace.get_tracer("modal.demo")
meter = metrics.get_meter("modal.demo")
self.invocations = meter.create_counter("demo.invocations")
self.duration = meter.create_histogram("demo.work.duration", unit="ms")
self.logger = logging.getLogger("modal.demo")
self.logger.setLevel(logging.INFO)
self.logger.addHandler(LoggingHandler(logger_provider=self.logger_provider))
self.logger.propagate = False
@modal.method()
def run(self, name="Modal"):
started = time.perf_counter()
with self.tracer.start_as_current_span("demo.run") as span:
span.set_attribute("demo.name", name)
self.logger.info("Starting demo work for %s", name)
with self.tracer.start_as_current_span("demo.simulated_work"):
time.sleep(0.1)
elapsed_ms = (time.perf_counter() - started) * 1_000
self.invocations.add(1, {"result": "success"})
self.duration.record(elapsed_ms, {"operation": "demo.run"})
self.logger.info("Finished demo work in %.2f ms", elapsed_ms)
return {"message": f"Hello, {name}!", "telemetry": "exported"}
@modal.exit()
def shutdown_telemetry(self):
self.tracer_provider.force_flush(timeout_millis=5_000)
self.logger_provider.force_flush(timeout_millis=5_000)
self.meter_provider.force_flush(timeout_millis=5_000)
self.logger_provider.shutdown()
self.meter_provider.shutdown()
self.tracer_provider.shutdown()
@app.local_entrypoint()
def main(name="Modal"):
print(InstrumentedWorker().run.remote(name))
```
Save the file as `modal_app.py`, then run it:
```bash
modal run modal_app.py --name Parseable
```
Run it a few times if you want metric charts to show visible points in Parseable.
## Validate in Parseable
After you send the Modal test event and run the sample app, open the three datasets in Parseable.
**Logs**
Open `modal-logs` to check Modal Function output, the `modal.test_logs` test event, and application log messages from `modal-parseable-demo`.
**Traces**
Open `modal-traces` to inspect custom spans such as `demo.run` and `demo.simulated_work`. This is where you can confirm that the application-level spans from your Modal function reached Parseable.
**Metrics**
Open `modal-metrics` to view Modal platform metrics along with custom metrics such as `demo.invocations` and `demo.work.duration`.
Metric names can appear before there are enough points to draw a useful chart. If a metric card says `No data`, run the workload again, wait for the Collector batch interval, and refresh the selected time range.
You can also check the Collector health endpoint:
```bash
curl http://localhost:13133
```
A healthy Collector response tells you the process is running. If Parseable is still empty, check the Collector logs for authentication errors, exporter failures, or connection errors to the Parseable endpoint.
## Troubleshooting
* **The Modal test does not appear in Parseable**: Confirm the Collector URL is public HTTPS, the tunnel or gateway is running, and `OTEL_HEADER_Authorization` matches `OTLP_INGEST_TOKEN`.
* **The Collector returns `401`**: Check the bearer token used by Modal. Managed telemetry uses `OTEL_HEADER_Authorization`; application telemetry uses `OTEL_EXPORTER_OTLP_HEADERS`.
* **The Collector receives data but Parseable is empty**: Verify `PARSEABLE_ENDPOINT`, `PARSEABLE_API_KEY`, and `X-P-Stream` in the Collector exporter config.
* **Logs arrive but traces or metrics do not**: Confirm the application has the matching OTLP exporters installed and that the app calls `force_flush` before shutdown.
* **Metric charts show no visible points**: Run the workload more than once, wait at least one batch interval, and refresh the time range in Parseable.
* **Telemetry stops after a while**: Temporary tunnel URLs stop working when the tunnel process exits. Create a new URL and update the Modal Secret and managed integration.
* **You see duplicate logs**: Modal-managed Function logs and application OpenTelemetry logs are separate sources. Filter by `service.name` or disable one path if you only need one copy.
## Production guidance
For production, run the Collector on durable infrastructure with a stable TLS hostname. Keep Collector ingress authenticated, rotate the ingress token periodically, and store the Parseable API key only in the Collector environment or secret store.
Keep logs, traces, and metrics in separate Parseable datasets. This makes it easier to browse each signal in Prism and keeps retention, access, and query behavior easier to reason about.
Finally, treat application log bodies as sensitive data. Avoid recording credentials, prompts, user identifiers, or payloads unless your team has explicitly decided those fields are safe to collect.
# n8n (/ingest-data/ai-agents/n8n)
n8n is often where a lot of small decisions in a system come together. A workflow might start from a webhook, call a few APIs, run an AI node, update a database, and then wait for another event before it continues.
To query or ingest Parseable data inside a workflow, use the [Parseable community node for n8n](/integrations/application-platforms/n8n).
When that workflow is simple, the execution page in n8n is usually enough. But once workflows become production paths, you need to see them beside the rest of your application telemetry. Parseable can receive n8n workflow traces through OpenTelemetry, so you can query and inspect workflow behavior from the same place where you already work with logs and traces.
## What n8n exports
n8n now has native OpenTelemetry tracing for workflow and node executions. When tracing is enabled, n8n exports:
* `workflow.execute`: one span for the workflow run
* `node.execute`: one child span for each node execution
The workflow span carries details such as workflow ID, workflow name, execution ID, execution mode, status, node count, and error type when the run fails. Node spans carry node ID, node name, node type, node version, and item counts.
n8n also supports W3C trace context. If a webhook request includes a `traceparent` header, n8n can use that as the parent for the workflow span. For outbound HTTP requests, n8n can inject trace context so downstream services can continue the same trace.
> Native OpenTelemetry support in n8n is currently focused on traces. n8n's documentation notes that OpenTelemetry-formatted metrics are still under development. For logs, collect the n8n process or container logs separately if you want them in Parseable.
## Telemetry paths
Think of the integration as three paths, not one single stream where every signal behaves the same way.
Native n8n workflow traces go through OpenTelemetry:
```text
n8n workflow and node executions
-> OpenTelemetry Collector
-> Parseable /v1/traces
-> n8n-traces dataset
```
Logs are collected separately from the n8n process or container:
```text
n8n process or container logs
-> log shipper or Collector filelog receiver
-> Parseable /v1/logs
-> n8n-logs dataset
```
Metrics are separate for now:
```text
runtime or infrastructure metrics
-> metrics collector or Prometheus pipeline
-> Parseable metrics dataset
```
The trace path is the main integration in this guide. Logs and metrics can still live in Parseable, but they should not be described as native n8n OpenTelemetry output today.
## How the setup works
The clean path is to send n8n traces to an OpenTelemetry Collector and let the Collector export them to Parseable.
```text
n8n
-> OpenTelemetry Collector
-> Parseable /v1/traces
-> n8n-traces dataset
```
This keeps n8n configured with standard OpenTelemetry settings while Parseable-specific headers stay in the Collector. It also gives you one place to add batching, retries, filtering, or redaction later.
## Prerequisites
You need a self-hosted n8n instance with OpenTelemetry tracing support, an OpenTelemetry Collector, and a Parseable instance that can receive OTLP traces.
If you run n8n in queue mode, set the OpenTelemetry configuration on every n8n process where tracing should be enabled, including main, worker, and webhook processors.
## Enable tracing in n8n
Set these environment variables on your n8n instance:
```bash
export N8N_OTEL_ENABLED=true
export N8N_OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
export N8N_OTEL_EXPORTER_SERVICE_NAME=n8n
```
Restart n8n after setting the variables.
The endpoint should point to the base OTLP HTTP endpoint of your Collector. n8n appends `/v1/traces` by default, so do not include `/v1/traces` in `N8N_OTEL_EXPORTER_OTLP_ENDPOINT`.
By default, n8n exports production executions. If you also want traces from manual or test executions while validating the setup, set:
```bash
export N8N_OTEL_TRACES_PRODUCTION_ONLY=false
```
For high-volume instances, you can reduce trace volume with sampling:
```bash
export N8N_OTEL_TRACES_SAMPLE_RATE=0.1
```
This exports roughly 10 percent of traces. n8n uses trace ID ratio sampling, so a trace is either kept with all its spans or dropped together.
## Send traces to Parseable
Use the OpenTelemetry Collector to receive traces from n8n and export them to Parseable.
```yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
otlphttp/parseabletraces:
endpoint: "https://"
headers:
Authorization: "Bearer "
X-P-Stream: "n8n-traces"
Content-Type: "application/x-protobuf"
encoding: proto
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseabletraces]
```
Replace `` with your Parseable base URL, for example `http://parseable:8000` or your Parseable Cloud endpoint.
If your Parseable deployment still uses basic authentication, replace the Bearer token with a Basic Auth value.
```yaml
headers:
Authorization: "Basic "
X-P-Stream: "n8n-traces"
Content-Type: "application/x-protobuf"
```
## Docker Compose example
Here is a small example showing n8n and the Collector together. Keep the Parseable endpoint and API key in environment variables in your real deployment.
```yaml
services:
n8n:
image: n8nio/n8n:latest
ports:
- "5678:5678"
environment:
N8N_OTEL_ENABLED: "true"
N8N_OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
N8N_OTEL_EXPORTER_SERVICE_NAME: "n8n"
N8N_OTEL_TRACES_PRODUCTION_ONLY: "false"
depends_on:
- otel-collector
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
command: ["--config=/etc/otelcol/config.yaml"]
volumes:
- ./otel-collector.yaml:/etc/otelcol/config.yaml
ports:
- "4318:4318"
```
Run a workflow after the stack starts. You should then see a dataset such as `n8n-traces` in Parseable.
## What to look for in Parseable
Once traces arrive, start with the `n8n-traces` dataset. The useful fields are usually the ones that describe the workflow run and the node that ran inside it.
For recent failed executions:
```sql
SELECT
p_timestamp,
span_name,
"n8n.workflow.name",
"n8n.execution.id",
"n8n.execution.status",
"n8n.execution.error_type"
FROM "n8n-traces"
WHERE "n8n.execution.status" = 'error'
ORDER BY p_timestamp DESC
LIMIT 50;
```
For slow workflow spans:
```sql
SELECT
p_timestamp,
span_name,
"n8n.workflow.name",
"n8n.execution.id",
ROUND(("span_end_time_unix_nano" - "span_start_time_unix_nano") / 1000000.0, 2) AS duration_ms
FROM "n8n-traces"
WHERE span_name = 'workflow.execute'
ORDER BY duration_ms DESC
LIMIT 20;
```
For slow node spans:
```sql
SELECT
p_timestamp,
"n8n.workflow.name",
"n8n.node.name",
"n8n.node.type",
ROUND(("span_end_time_unix_nano" - "span_start_time_unix_nano") / 1000000.0, 2) AS duration_ms
FROM "n8n-traces"
WHERE span_name = 'node.execute'
ORDER BY duration_ms DESC
LIMIT 20;
```
These queries are meant as starting points. Field names can vary slightly depending on the n8n version and how the spans are flattened after ingestion, so check a few recent records in Parseable first and adjust the column names if required.
## Dashboards
After traces are flowing, you can use a dashboard to keep the main workflow signals in one place instead of writing ad hoc queries each time. Parseable provides an n8n workflow observability dashboard in [parseablehq/dashboards](https://github.com/parseablehq/dashboards/tree/main/n8n-workflow-observability). Use it as a starting point, then map the dataset variable to your `n8n-traces` dataset.
The dashboard covers execution volume, success rate, average duration, the slowest nodes, node execution counts by type, a per-execution summary table and a recent errors table - all from the `workflow.execute` and `node.execute` spans described above.
## Logs and metrics
n8n's native OpenTelemetry path currently sends workflow traces. If you also want logs in Parseable, collect n8n process or container logs with your existing log shipper or with the OpenTelemetry Collector `filelog` receiver, then export them to Parseable's `/v1/logs` endpoint.
The log path usually looks like this:
```text
n8n logs
-> filelog receiver or log shipper
-> Parseable /v1/logs
-> n8n-logs
```
Metrics should be treated separately for now. n8n's own OpenTelemetry documentation says OpenTelemetry-formatted metrics are still coming, so avoid documenting them as native n8n OpenTelemetry output until that support is available.
For metrics, use the metric source you already trust for your deployment, such as container, host, or Kubernetes metrics. Those can be sent to Parseable through your metrics pipeline, but they are not the same thing as n8n native OpenTelemetry metrics.
## Troubleshooting
If traces do not appear in Parseable, first check that n8n can reach the Collector:
```bash
docker compose logs n8n | grep -i otel
docker compose logs otel-collector
```
Then verify the endpoint shape. n8n should point to the Collector base URL, such as `http://otel-collector:4318`. The Collector should point to the Parseable base URL, such as `https://`, and the exporter should send trace data through the OTLP HTTP exporter.
Also check sampling and execution type. If `N8N_OTEL_TRACES_PRODUCTION_ONLY=true`, manual executions may not show up. During setup, temporarily set it to `false`, run a workflow, and then switch it back if production-only traces are what you want.
## References
* [n8n OpenTelemetry tracing](https://docs.n8n.io/deploy/host-n8n/keep-n8n-running/trace-executions-with-opentelemetry/)
* [n8n OpenTelemetry environment variables](https://docs.n8n.io/deploy/host-n8n/configure-n8n/basic-configuration/use-environment-variables/opentelemetry/)
* [Parseable OpenTelemetry traces](/ingest-data/otel/traces)
* [Parseable OpenTelemetry logs](/ingest-data/otel/logs)
# OpenAI (/ingest-data/ai-agents/openai)
import { Step, Steps } from 'fumadocs-ui/components/steps';
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
The OpenAI Python SDK does not emit OpenTelemetry data on its own. There is no built-in `openai.instrument()` method for plain SDK calls. The separate `openai-agents` package has its own tracing, but that is meant for applications built with the Agents SDK, not for instrumenting a regular `openai` client.
To get spans from a call such as `client.chat.completions.create()`, use an instrumentor that patches the OpenAI client before it makes requests. This page shows two common options:
* **[OpenLIT](https://github.com/openlit/openlit):** a short `openlit.init()` setup, its own OTLP exporter configuration and token cost calculation out of the box. This is also the pattern used by the [CrewAI](/ingest-data/ai-agents/crewai) and [LiteLLM SDK](/ingest-data/ai-agents/litellm-sdk) docs in this hub.
* **[OpenInference](https://github.com/Arize-ai/openinference):** a standard OpenTelemetry instrumentor through `OpenAIInstrumentor().instrument(tracer_provider=...)`. You create the `TracerProvider` and exporter yourself, which makes it easier to combine with other OpenInference instrumentors on the same provider.
Both approaches emit GenAI semantic convention spans that can be stored in the same Parseable traces dataset. Pick one for a given OpenAI client. Running both together can double-instrument the same request.
## How it works
```text
Python application using OpenAI SDK
|
| OpenLIT or OpenInference patches the OpenAI client
|
| OTLP traces
v
Parseable
|
+--> openai-sdk-traces traces dataset in Parseable
```
Each chat completion produces a `chat ` span with GenAI attributes such as prompt, response, token usage and cost when the instrumentor provides it. The trace also includes a child `POST` span for the underlying HTTP call to `api.openai.com`. If the model responds with tool calls, the follow-up request that sends tool results back is captured as another span in the same trace.
## Prerequisites
Before you start, keep these ready:
* A running Parseable instance
* A Parseable API key with ingest access
* Python 3.10 or newer
* An `OPENAI_API_KEY`
## Set up OpenAI SDK with Parseable
### Install dependencies
```bash
pip install openai openlit
```
```bash
pip install openai \
openinference-instrumentation-openai \
opentelemetry-sdk \
opentelemetry-exporter-otlp-proto-http
```
### Instrument the client before making requests
Whichever instrumentor you pick, it must run before you construct the `OpenAI` client, so the patch is in place when the client makes its first call.
`openlit.init()` sets up its own `TracerProvider` and OTLP exporter, so you do not need a separate OpenTelemetry setup for this path.
```python
import os
import openlit
from openai import OpenAI
openlit.init(
otlp_endpoint=os.environ["PARSEABLE_URL"], # e.g. http://:8010
otlp_headers={
"Authorization": f"Bearer {os.environ['PARSEABLE_API_KEY']}",
"X-P-Stream": "openai-sdk-traces",
},
service_name="openai-sdk-demo",
environment="production",
disable_batch=True,
disable_metrics=True,
disable_events=True,
)
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Say hello in five words."}],
)
print(response.choices[0].message.content)
```
`disable_batch=True` exports each span as soon as it finishes, which is useful for short-lived scripts. Remove it for long-running services so spans batch and export on a timer instead.
OpenInference is a standard OpenTelemetry instrumentor. You build the `TracerProvider` and exporter yourself, then pass that provider to `OpenAIInstrumentor().instrument(...)`. This is the same pattern the [CrewAI](/ingest-data/ai-agents/crewai) integration uses when it combines `CrewAIInstrumentor` and `OpenAIInstrumentor` on one provider.
```python
import os
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider(
resource=Resource.create({"service.name": "openai-sdk-demo"})
)
exporter = OTLPSpanExporter(
endpoint=f"{os.environ['PARSEABLE_URL']}/v1/traces",
headers={
"Authorization": f"Bearer {os.environ['PARSEABLE_API_KEY']}",
"X-P-Stream": "openai-sdk-traces",
},
)
provider.add_span_processor(BatchSpanProcessor(exporter))
OpenAIInstrumentor().instrument(tracer_provider=provider)
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Say hello in five words."}],
)
print(response.choices[0].message.content)
```
`OTLPSpanExporter` builds the URL as `{PARSEABLE_URL}/v1/traces` explicitly. Unlike OpenLIT, it does not append the path for you. `BatchSpanProcessor` batches on a timer by default. For short scripts, call `provider.force_flush()` or `provider.shutdown()` before exit so spans are exported before the process ends.
The dataset named in `X-P-Stream` is created automatically on first ingest if it does not already exist.
### Tool calls
Tool-calling requests use the same instrumentation path under both instrumentors. OpenAI SDK-level tool call and result messages are captured as part of the same trace.
```python
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What's the weather in Bengaluru?"}],
tools=tools,
tool_choice="auto",
)
message = response.choices[0].message
if message.tool_calls:
messages = [{"role": "user", "content": "What's the weather in Bengaluru?"}, message]
for call in message.tool_calls:
# ... execute the tool, then append its result ...
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": '{"city": "Bengaluru", "temp_c": 28}',
})
client.chat.completions.create(model="gpt-4o-mini", messages=messages)
```
### Send a few requests
Run the application a few times with different models and prompts, including at least one tool-calling request. This gives you enough data to inspect normal chat spans, HTTP spans and tool-call follow-up spans in Parseable.
## What you get in Parseable
Open `openai-sdk-traces` from the Traces page. Each chat completion appears as a `chat ` span with GenAI attributes, plus a child `POST` span for the HTTP call. Multiple calls in one process share `service.instance.id`, and tool-call follow-up requests can be followed through the same trace identifiers.
## Useful fields
| Field | Meaning |
| --------------------------------------- | ------------------------------------------------------------------------------------------ |
| `gen_ai.provider.name` | Always `openai` for this integration |
| `gen_ai.request.model` | The model requested by the application |
| `gen_ai.response.model` | The model version that actually served the request |
| `gen_ai.operation.name` | The GenAI operation, such as `chat` |
| `gen_ai.input.messages` | The request messages, including system/user/tool roles |
| `gen_ai.output.messages` | The response messages and finish reason |
| `gen_ai.usage.input_tokens` | Input token count |
| `gen_ai.usage.output_tokens` | Output token count |
| `gen_ai.usage.cost` | Computed request cost (OpenLIT computes this; OpenInference may not, depending on version) |
| `gen_ai.server.time_to_first_token` | Time to first token |
| `span_status_code` | Whether the span completed successfully (`1` = OK) |
| `span_trace_id` / `span_parent_span_id` | Use these to reconstruct the chat span and its child HTTP span |
## Query examples
Total requests and tokens by model:
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS requests,
SUM(CAST("gen_ai.usage.input_tokens" AS BIGINT)) AS input_tokens,
SUM(CAST("gen_ai.usage.output_tokens" AS BIGINT)) AS output_tokens
FROM "openai-sdk-traces"
WHERE "gen_ai.operation.name" = 'chat'
GROUP BY model;
```
Error rate by model:
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS total,
SUM(CASE WHEN span_status_code != 1 THEN 1 ELSE 0 END) AS errors
FROM "openai-sdk-traces"
WHERE "gen_ai.operation.name" = 'chat'
GROUP BY model;
```
## OpenLIT or OpenInference
Use **OpenLIT** when you want a short `init()` call, built-in cost calculation and a setup that does not require you to build the OpenTelemetry provider yourself.
Use **OpenInference** when you are already building an OpenTelemetry `TracerProvider` for other instrumentors, such as combining `CrewAIInstrumentor` and `OpenAIInstrumentor` on one provider, or when you want direct control over the exporter and processors.
## Troubleshooting
* **No traces appear**
Confirm the instrumentor runs before the `OpenAI` client is constructed. If the client is imported and instantiated at module load time before instrumentation runs, it cannot be patched.
* **Traces appear late or not at all in short scripts**
OpenLIT: set `disable_batch=True` in `openlit.init()`. OpenInference: call `provider.force_flush()` or `provider.shutdown()` before the process exits. `BatchSpanProcessor` batches on a timer by default.
* **Prompt or response text appears in telemetry and that's a concern**
OpenLIT: pass `capture_message_content=False` to `openlit.init()`. OpenInference: check the instrumentor's config for a content-masking option, or set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false` (or the instrumentor-specific env var) before instrumenting.
## See also
* [LiteLLM SDK](/ingest-data/ai-agents/litellm-sdk)
* [CrewAI](/ingest-data/ai-agents/crewai)
* [Pydantic AI](/ingest-data/ai-agents/pydantic-ai)
* [Traces](/user-guide/traces)
# OpenLIT Agent Observability (/ingest-data/ai-agents/openlit)
import { Step, Steps } from 'fumadocs-ui/components/steps';
[OpenLIT](https://openlit.io/) is an OpenTelemetry-native observability SDK for
LLM applications and AI agents. It can instrument supported agent frameworks,
model providers, and tools so one trace shows the complete path from an agent
invocation to its workflow steps, tool executions, and model calls.
This integration is framework-neutral. Use the same Parseable and OpenTelemetry
Collector configuration with CrewAI, OpenAI Agents SDK, LangChain or LangGraph,
Agno, LlamaIndex, and other frameworks supported by OpenLIT.
```text
Agent application
|
| OpenLIT instrumentation
v
Agent invocation -> workflow -> tool calls -> model calls
|
| OTLP/HTTP
v
OpenTelemetry Collector
|
+--> openlit-agent-traces Agent Observability and trace dataset
+--> openlit-agent-metrics Metrics dataset
```
## Prerequisites
* A running Parseable instance
* A Parseable API key with dataset creation and ingest access
* Python 3.9 or newer
* An OpenTelemetry Collector Contrib binary or container
* A supported agent framework and its model-provider API key
## Set up OpenLIT Agent Observability
### Create the Parseable datasets
Set the connection values:
```bash
export PARSEABLE_URL="https://:8000"
export PARSEABLE_API_KEY=""
export OPENLIT_TRACE_STREAM="openlit-agent-traces"
export OPENLIT_METRIC_STREAM="openlit-agent-metrics"
```
Create the trace dataset and tag it for Agent Observability:
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/$OPENLIT_TRACE_STREAM" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: traces" \
-H "X-P-Dataset-Tag: agent-observability"
```
Create the metrics dataset:
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/$OPENLIT_METRIC_STREAM" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: metrics"
```
`X-P-Dataset-Tag: agent-observability` makes the trace dataset available from
Parseable's Agents page. Create the dataset explicitly because an automatically
created OTLP trace dataset does not receive this tag.
### Configure the OpenTelemetry Collector
Create `otel-collector-config.yaml`:
```yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
exporters:
otlphttp/parseable_traces:
endpoint: "${env:PARSEABLE_URL}"
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: "${env:OPENLIT_TRACE_STREAM}"
otlphttp/parseable_metrics:
endpoint: "${env:PARSEABLE_URL}"
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: "${env:OPENLIT_METRIC_STREAM}"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_traces]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable_metrics]
```
Start the Collector in the shell containing the exported variables:
```bash
otelcol-contrib --config otel-collector-config.yaml
```
Use the Parseable base URL. The `otlphttp` exporter appends `/v1/traces` and
`/v1/metrics` for the corresponding pipelines.
### Instrument the agent application
Install OpenLIT and the packages required by your chosen agent framework:
```bash
pip install openlit
```
Initialize OpenLIT before importing the framework or provider SDK:
```python
import openlit
openlit.init(
otlp_endpoint="http://localhost:4318",
service_name="support-agent",
environment="production",
capture_message_content=False,
)
# Import and run your supported agent framework only after initialization.
```
No Parseable-specific instrumentation belongs in the agent implementation. The
Collector determines which trace and metric datasets receive the OTLP signals.
Choose the OpenLIT integration matching your framework:
* [OpenAI Agents SDK](https://docs.openlit.io/latest/sdk/integrations/openai-agents)
* [CrewAI](https://docs.openlit.io/latest/sdk/integrations/crewai)
* [LangChain and LangGraph](https://docs.openlit.io/latest/sdk/integrations/langchain)
* [All OpenLIT integrations](https://docs.openlit.io/latest/sdk/integrations/overview)
The application must execute an agent framework operation. A direct model SDK
call produces model-operation spans such as `chat`, but it does not create an
agent invocation, workflow, or tool-execution hierarchy.
### Run and verify an agent
Run an agent workflow that makes at least one model call and, when possible,
executes a tool. Allow the SDK and Collector batch processors to flush.
In Parseable, verify:
1. `openlit-agent-traces` appears in both Traces and Agents.
2. One trace contains the complete agent run and its child spans.
3. Agent or workflow operations appear alongside model operations such as `chat`.
4. Tool spans represent actual executions, not only tool-call requests returned by a model.
5. All related spans share a trace ID and have coherent parent span IDs.
6. `openlit-agent-metrics` appears in Metrics.
For a short-lived smoke test, reduce the metric export interval:
```bash
export OTEL_METRIC_EXPORT_INTERVAL=1000
python agent.py
```
## What you get in Parseable
Exact span names and attributes depend on the framework, provider, and OpenLIT
version. A complete agent trace can contain:
| Telemetry | What it represents |
| ---------------- | ------------------------------------------------------------------- |
| Agent invocation | Top-level agent execution |
| Workflow or task | Chain, graph, crew, task, or handoff operation |
| Tool execution | Tool name, arguments, result, duration, and status |
| Model operation | Provider request, model, tokens, cost, latency, and response status |
| Error event | Framework, tool, or provider exception with trace context |
Common fields include:
| Field | Meaning |
| ---------------------------- | -------------------------------------------------------------- |
| `service.name` | Service configured in `openlit.init()` |
| `deployment.environment` | Deployment environment |
| `gen_ai.operation.name` | Operation such as `invoke_agent`, `invoke_workflow`, or `chat` |
| `gen_ai.agent.name` | Agent name when supplied by the framework |
| `gen_ai.workflow.name` | Workflow name when supplied by the framework |
| `gen_ai.tool.name` | Tool name on supported tool spans or model tool-call data |
| `gen_ai.request.model` | Requested model |
| `gen_ai.usage.input_tokens` | Input token count |
| `gen_ai.usage.output_tokens` | Output token count |
| `gen_ai.usage.cost` | Computed model-request cost |
| `span_trace_id` | Identifier joining all spans in one run |
| `span_parent_span_id` | Parent relationship used by the waterfall |
| `span_status_code` | Operation status |
OpenLIT also emits metrics such as `gen_ai.client.operation.duration`,
`gen_ai.client.token.usage`, `gen_ai.usage.cost`,
`gen_ai.server.time_to_first_token`, and
`gen_ai.server.time_per_output_token`.
## Query agent telemetry
Agent and workflow operations:
```sql
SELECT
"gen_ai.operation.name" AS operation,
COUNT(*) AS spans,
COUNT(DISTINCT "span_trace_id") AS runs
FROM "openlit-agent-traces"
WHERE p_timestamp > NOW() - INTERVAL '24 hours'
GROUP BY operation
ORDER BY spans DESC;
```
Model usage caused by agent runs:
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS calls,
SUM(CAST("gen_ai.usage.input_tokens" AS BIGINT)) AS input_tokens,
SUM(CAST("gen_ai.usage.output_tokens" AS BIGINT)) AS output_tokens
FROM "openlit-agent-traces"
WHERE "gen_ai.operation.name" = 'chat'
AND p_timestamp > NOW() - INTERVAL '24 hours'
GROUP BY model
ORDER BY calls DESC;
```
## Troubleshoot
* **Dataset appears in Traces but not Agents:** The dataset was probably auto-created without the Agent Observability tag. Recreate an empty dataset with `X-P-Telemetry-Type: traces` and `X-P-Dataset-Tag: agent-observability` before ingesting data.
* **Traces contain only `chat` operations:** OpenLIT is observing direct model calls, not an agent-framework execution. Confirm that the installed framework is supported, initialize OpenLIT before importing it, and execute the framework's agent runner rather than only its underlying model client.
* **Tool calls appear without tool execution spans:** A model can request a tool without the application executing it. Confirm that the agent framework dispatches the tool and that OpenLIT supports instrumenting that framework's tool runtime.
* **No traces appear:** Check Collector logs, confirm the application exports OTLP/HTTP to port `4318`, and verify the Parseable exporter includes `X-P-Stream`.
* **Traces appear but metrics are empty:** Metrics use a periodic exporter. Keep the application alive until the first export or temporarily set `OTEL_METRIC_EXPORT_INTERVAL=1000`.
* **Prompts or responses appear in traces:** Set `capture_message_content=False`. This affects only newly generated telemetry; review or remove older records separately.
## Related documentation
* [OpenLIT SDK](https://docs.openlit.io/latest/sdk/quickstart-ai-observability)
* [Parseable Agent Observability](/user-guide/agent-observability)
* [Parseable OpenTelemetry traces](/ingest-data/otel/traces)
* [Parseable OpenTelemetry metrics](/ingest-data/otel/metrics)
* [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/)
# OpenRouter (/ingest-data/ai-agents/openrouter)
[OpenRouter](https://openrouter.ai/) gives you one API for calling models from different providers. That makes it useful for AI applications that switch models, compare providers, or use fallback routing. Once those requests are running in production, you still need to answer a few basic questions: which model was called, how long it took, how many tokens were used, which provider served the request, and whether a request failed.
OpenRouter Broadcast can send traces for OpenRouter requests to an OpenTelemetry-compatible destination. Parseable can receive those traces on its native OTLP traces endpoint and store them in a trace dataset that you can inspect, query, and use in dashboards.
This guide focuses on traces because that is what OpenRouter Broadcast exports. Once the setup is working, each OpenRouter request can show up in Parseable with model, provider, timing, token usage, cost, status, and request context.
```text
Application
|
| OpenRouter API request
v
OpenRouter
|
| Broadcast traces over OTLP/HTTP
v
Parseable
|
+--> openrouter-traces
```
Use the direct setup when your Parseable endpoint is reachable over HTTPS from OpenRouter. Use the Collector setup when Parseable is private, HTTP-only, or when you want the Collector to handle authentication and routing.
| Setup path | Use when |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Direct to Parseable | Parseable has a public HTTPS endpoint and you want the shortest path |
| Through a Collector | Parseable is private, HTTP-only, or you want batching, filtering, retry handling, or header injection outside OpenRouter |
## Prerequisites
Before you start, keep these ready:
* An OpenRouter account with access to **Settings > Observability**
* Organization admin access if Broadcast is configured for an OpenRouter organization
* An OpenRouter API key for sending a test request
* A Parseable instance reachable over HTTPS, or an OpenTelemetry Collector Contrib deployment that is reachable over HTTPS
* A Parseable API key with ingest access
* A trace dataset name, for example `openrouter-traces`
OpenRouter Broadcast sends traces. The dataset you create in Parseable should be a traces dataset, not a regular JSON log dataset.
## Send traces directly to Parseable
If Parseable is reachable from the public internet over HTTPS, you can point OpenRouter Broadcast directly at the Parseable OTLP traces endpoint.
Use this endpoint in OpenRouter:
```text
https://:8000/v1/traces
```
Add these headers in the OpenRouter destination configuration:
```json
{
"Authorization": "Bearer ",
"X-P-Stream": "openrouter-traces"
}
```
The Bearer token authenticates the request with Parseable. `X-P-Stream` tells Parseable which dataset should receive the traces. Parseable identifies the signal as traces from the `/v1/traces` endpoint.
In the OpenRouter dashboard, open **Settings > Observability**, enable **Broadcast**, and add an **OpenTelemetry Collector** destination. Set the endpoint to the Parseable `/v1/traces` URL and paste the headers JSON above. Use **Test Connection** before saving the destination.
OpenRouter only saves the destination after the test succeeds. The test sends a small synthetic trace, so you may see a connection-test record before your real application traffic arrives. After a successful test, send one real OpenRouter request and check the `openrouter-traces` dataset in Parseable.
If `openrouter-traces` already exists as a regular log dataset, OTLP trace ingestion can fail because the dataset was created with the wrong telemetry type. Use a fresh dataset name or recreate the dataset through the `/v1/traces` path.
## Send traces through an OpenTelemetry Collector
Use a Collector when the Parseable endpoint should stay private, or when you want one place to add batching, filtering, retries, or routing later. OpenRouter still needs to reach the Collector over HTTPS.
The example below uses `otelcol-contrib` because it includes the `bearertokenauth` extension. The token is only used between OpenRouter and your Collector. It is not your OpenRouter API key and it is not your Parseable API key.
Create `otel-collector-config.yaml`:
```yaml
extensions:
bearertokenauth/openrouter:
token: ${env:OPENROUTER_BROADCAST_TOKEN}
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
auth:
authenticator: bearertokenauth/openrouter
processors:
batch:
timeout: 5s
send_batch_size: 512
exporters:
otlphttp/parseable:
traces_endpoint: https://:8000/v1/traces
encoding: json
headers:
Authorization: "Bearer ${env:PARSEABLE_API_KEY}"
X-P-Stream: openrouter-traces
service:
extensions: [bearertokenauth/openrouter]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/parseable]
```
If the Collector and Parseable run inside the same private network, `traces_endpoint` can point to the internal Parseable URL instead. The public HTTPS requirement applies to the endpoint that OpenRouter calls, not necessarily to the private Collector to Parseable hop.
Run the Collector:
```bash
export PARSEABLE_API_KEY=""
export OPENROUTER_BROADCAST_TOKEN="$(openssl rand -hex 32)"
otelcol-contrib --config otel-collector-config.yaml
```
Expose the Collector over HTTPS. For local testing, a temporary tunnel is enough:
```bash
cloudflared tunnel --url http://localhost:4318
```
In OpenRouter, configure the Broadcast destination with:
```text
https:///v1/traces
```
Set the destination headers to:
```json
{
"Authorization": "Bearer "
}
```
The Collector receives the trace from OpenRouter, validates the bearer token, and forwards the trace to Parseable with the Parseable API key and dataset headers.
## Broadcast options
OpenRouter Broadcast has a few options that are worth setting before production traffic starts flowing.
Use **API key filtering** when you want only selected OpenRouter keys to send traces to Parseable. This is useful when development and production use different OpenRouter keys, or when you want one destination to receive only a subset of traffic.
Use **Sampling Rate** when request volume is high. A sampling rate of `1.0` sends all traces. A value such as `0.1` sends a smaller sample. If you send `session_id` with requests, OpenRouter keeps sampling consistent for that session so the traces are not split randomly across the same conversation.
Use **Privacy Mode** if prompts and completions should not be exported. With Privacy Mode enabled, request and response content is removed from the trace, while metadata such as token usage, cost, timing, model information, and custom metadata is still sent.
If you do not need prompts or completions in Parseable, start with **Privacy Mode** enabled. You still get the operational fields needed for usage, latency, model, and cost analysis, and you can decide later whether a debugging environment should receive fuller traces.
## Send a test request
After Broadcast is saved, send a request through OpenRouter:
```bash
curl https://openrouter.ai/api/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d '{
"model": "openai/gpt-4o-mini",
"messages": [
{
"role": "user",
"content": "Say OK"
}
]
}'
```
OpenRouter uses `Authorization: Bearer ` for its own API. Parseable uses `Authorization: Bearer ` on the OTLP destination or inside the Collector exporter.
Broadcast delivery is asynchronous, so give it a few seconds after the model response is returned.
## Add request context
OpenRouter traces are more useful when requests carry enough context to connect them back to a user, session, or workflow. You can add a `user`, a `session_id`, and a `trace` object to the OpenRouter request body.
```json
{
"model": "openai/gpt-4o-mini",
"messages": [
{
"role": "user",
"content": "Summarize this support ticket"
}
],
"user": "user_12345",
"session_id": "ticket_789",
"trace": {
"trace_name": "Support ticket summarization",
"environment": "production",
"feature": "support-assistant"
}
}
```
Use this for stable, non-sensitive identifiers. Avoid putting emails, raw customer names, or secrets in metadata unless your data handling policy allows it.
## View traces in Parseable
Open the `openrouter-traces` dataset in Parseable and select the **Traces** view. You should see OpenRouter requests as trace records. A trace usually includes the model request, the upstream provider attempt, timing, status, token usage, and the generation ID returned by OpenRouter.
Useful fields to look for include:
* `span_trace_id` and `span_span_id`
* `span_name`
* `span_duration_ns`
* `span_status_code`
* `gen_ai.request.model`
* `gen_ai.response.model`
* `gen_ai.response.id`
* `gen_ai.usage.input_tokens`
* `gen_ai.usage.output_tokens`
* `gen_ai.usage.total_tokens`
* `gen_ai.usage.total_cost`
The exact fields can vary based on the request and OpenRouter response. Start with the trace detail view first, then use SQL once you know which fields are present in your dataset.
To verify the setup, start with three checks:
1. Confirm that the `openrouter-traces` dataset exists.
2. Open the **Traces** view and check that new records appear after an OpenRouter request.
3. Open one trace and confirm that model, token, latency, provider, and status fields are present.
## Query the trace dataset
You can query the trace dataset from the SQL editor.
Token usage by requested model:
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS requests,
SUM(CAST("gen_ai.usage.input_tokens" AS BIGINT)) AS input_tokens,
SUM(CAST("gen_ai.usage.output_tokens" AS BIGINT)) AS output_tokens,
SUM(CAST("gen_ai.usage.total_tokens" AS BIGINT)) AS total_tokens
FROM "openrouter-traces"
WHERE p_timestamp > NOW() - INTERVAL '24 hours'
GROUP BY model
ORDER BY total_tokens DESC;
```
Latency by model:
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS requests,
AVG(span_duration_ns / 1000000.0) AS avg_latency_ms,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY span_duration_ns / 1000000.0) AS p95_latency_ms
FROM "openrouter-traces"
WHERE p_timestamp > NOW() - INTERVAL '24 hours'
GROUP BY model
ORDER BY p95_latency_ms DESC;
```
Cost by model:
```sql
SELECT
"gen_ai.request.model" AS model,
COUNT(*) AS requests,
SUM(CAST("gen_ai.usage.total_cost" AS DOUBLE)) AS total_cost_usd
FROM "openrouter-traces"
WHERE p_timestamp > NOW() - INTERVAL '7 days'
GROUP BY model
ORDER BY total_cost_usd DESC;
```
## Dashboards
After traces are flowing, you can build a dashboard for request volume, token usage, cost, latency, and provider behavior. Public Parseable dashboard examples live in [parseablehq/dashboards](https://github.com/parseablehq/dashboards/tree/main/openrouter-observability). Use them as a starting point, then update dataset names and queries for your `openrouter-traces` dataset.
The request dashboard helps you see how request volume changes over time and which models or providers are active.
The token and cost dashboard gives you a quick view of input tokens, output tokens, and cost trends.
## Troubleshooting
* If OpenRouter **Test Connection** fails, check that the endpoint is public HTTPS and points to `/v1/traces`.
* If the dataset already exists but ingestion fails, make sure it was created as a traces dataset. If it was created earlier as a plain JSON log dataset, use a new dataset name or recreate it through OTLP traces ingestion.
* If traces reach the Collector but not Parseable, check the Collector logs and confirm `Authorization` and `X-P-Stream` are present in the exporter headers.
* If no real requests appear after a successful test, confirm that your application is calling OpenRouter and not a provider API directly. Broadcast only runs for requests that pass through OpenRouter.
## Next steps
* Read the [OpenRouter Broadcast docs](https://openrouter.ai/docs/guides/features/broadcast)
* Read the [OpenRouter OpenTelemetry Collector docs](https://openrouter.ai/docs/guides/features/broadcast/otel-collector)
* Build a Parseable dashboard from the `openrouter-traces` dataset
# Pydantic AI (/ingest-data/ai-agents/pydantic-ai)
import { Step, Steps } from 'fumadocs-ui/components/steps';
Pydantic AI is a Python framework from the Pydantic team for building typed agents. It is useful when you want agent code to feel like normal Python, with tools, structured outputs, retries, validation, and model calls kept close to your application logic.
When an agent runs in production, the final answer is usually the least interesting part of the story. You also need to know which model was called, which tools ran, how long each step took, how many tokens were used, and where a retry or failure happened. Pydantic AI has built-in OpenTelemetry instrumentation for that path, and Parseable can receive the emitted telemetry directly.
This guide shows how to send Pydantic AI traces and metrics to Parseable. The traces power Parseable Agent Observability, while the metrics help you track usage and latency over time.
## How it works
Pydantic AI emits OpenTelemetry spans for agent runs, model requests, and tool execution after instrumentation is enabled. A single agent run usually becomes one trace with the agent span at the top, model-call spans below it, and tool spans where the agent calls Python functions.
```text
Pydantic AI application
|
| OpenTelemetry traces and metrics
v
Parseable
|
+--> pydantic-ai-traces traces dataset and Agent Observability
+--> pydantic-ai-metrics metrics dataset
```
You can send telemetry through an OpenTelemetry Collector if that is already part of your deployment. For a small application or first test, sending directly to Parseable is also fine.
## Prerequisites
Before you start, keep these ready:
* Python 3.10 or later
* A running Parseable instance
* A Parseable API key with ingest access
* An OpenAI API key, or another model-provider key supported by Pydantic AI
## Set up Pydantic AI with Parseable
### Create Parseable datasets
Create a trace dataset and tag it for Agent Observability:
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/pydantic-ai-traces" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: traces" \
-H "X-P-Dataset-Tag: agent-observability"
```
Create a metrics dataset when you also want token usage, latency, and streaming metrics:
```bash
curl -X PUT "$PARSEABLE_URL/api/v1/logstream/pydantic-ai-metrics" \
-H "Authorization: Bearer ${PARSEABLE_API_KEY}" \
-H "X-P-Telemetry-Type: metrics"
```
`X-P-Dataset-Tag` tags the dataset itself. Do not use `X-P-Tag-*` for this step, because those headers add fields to ingested records instead.
### Install dependencies
Install Pydantic AI with OpenAI support and the OpenTelemetry OTLP HTTP exporter:
```bash
pip install "pydantic-ai-slim[openai]" \
opentelemetry-sdk \
opentelemetry-exporter-otlp-proto-http
```
### Configure OpenTelemetry
Create `agent.py`:
```python
import os
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from pydantic_ai import Agent, InstrumentationSettings, RunContext
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
parseable_url = os.environ["PARSEABLE_URL"].rstrip("/")
parseable_api_key = os.environ["PARSEABLE_API_KEY"]
resource = Resource.create(
{
"service.name": "pydantic-ai-agent",
"deployment.environment.name": "production",
}
)
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=f"{parseable_url}/v1/traces",
headers={
"Authorization": f"Bearer {parseable_api_key}",
"X-P-Stream": "pydantic-ai-traces",
},
)
)
)
trace.set_tracer_provider(tracer_provider)
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=f"{parseable_url}/v1/metrics",
headers={
"Authorization": f"Bearer {parseable_api_key}",
"X-P-Stream": "pydantic-ai-metrics",
},
),
export_interval_millis=5000,
)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
Agent.instrument_all(
InstrumentationSettings(
version=5,
include_content=True,
tracer_provider=tracer_provider,
meter_provider=meter_provider,
)
)
model = OpenAIChatModel(
"gpt-4o-mini",
provider=OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]),
)
agent = Agent(model, name="weather_agent")
@agent.tool
def get_weather(ctx: RunContext[None], city: str) -> str:
return f"The weather in {city} is sunny and 24 C."
try:
result = agent.run_sync("What is the weather in Bengaluru?")
print(result.output)
finally:
tracer_provider.force_flush()
tracer_provider.shutdown()
meter_provider.force_flush()
meter_provider.shutdown()
```
Run the application:
```bash
export PARSEABLE_URL="https://:8000"
export PARSEABLE_API_KEY=""
export OPENAI_API_KEY=""
python agent.py
```
## What the instrumentation captures
With `InstrumentationSettings(version=5)`, Pydantic AI emits the current instrumentation format. In Parseable, a normal tool-using run should look like this:
```text
invoke_agent weather_agent
├── chat gpt-4o-mini
├── execute_tool get_weather
└── chat gpt-4o-mini
```
The agent span represents the full run. Model spans show the provider, requested model, response model, token usage, latency, and errors. Tool spans show the tool name, tool call ID, arguments, result, and duration. This is the structure that makes it possible to move from a high-level agent run into the exact model or tool step that changed the outcome.
For Agent Observability, the most important tool fields are:
| Field | Meaning |
| ---------------------------- | --------------------------------- |
| `gen_ai.tool.name` | Tool function name |
| `gen_ai.tool.call.id` | Tool call identifier |
| `gen_ai.tool.call.arguments` | Arguments passed to the tool |
| `gen_ai.tool.call.result` | Tool result returned to the agent |
If the Agent Observability setup page reports missing tool arguments or results, confirm that you are using instrumentation `version=5`, `include_content=True`, and that the run actually invoked a tool.
## Metrics
Pydantic AI records OpenTelemetry histograms for model usage and streaming behavior:
| Metric | Unit | When emitted |
| --------------------------------------------- | ------- | --------------------------------------------------------- |
| `gen_ai.client.token.usage` | tokens | Model or embedding requests, split by `gen_ai.token.type` |
| `operation.cost` | USD | When the model price is known |
| `gen_ai.client.operation.time_to_first_chunk` | seconds | Streaming requests only |
Each metric point carries model and provider attributes such as `gen_ai.provider.name`, `gen_ai.operation.name`, `gen_ai.request.model`, and `gen_ai.response.model`. Streaming latency appears only when your code consumes the stream far enough for Pydantic AI to observe the first chunk.
Since these are histogram metrics, query `data_point_sum` and `data_point_count` rather than `data_point_value`.
Example token usage query:
```sql
SELECT
attributes->>'gen_ai.token.type' AS token_type,
SUM(data_point_sum) AS tokens
FROM "pydantic-ai-metrics"
WHERE metric_name = 'gen_ai.client.token.usage'
GROUP BY token_type;
```
Example average time to first chunk:
```sql
SELECT
SUM(data_point_sum) / NULLIF(SUM(data_point_count), 0) AS avg_ttf_chunk_seconds
FROM "pydantic-ai-metrics"
WHERE metric_name = 'gen_ai.client.operation.time_to_first_chunk';
```
## Content and privacy
`include_content=True` captures prompts, completions, tool arguments, and tool results. This is useful for understanding agent behavior, and it gives Agent Observability enough detail to show tool input and output. It can also capture sensitive data, so review this setting before using it in production.
If your production policy is to avoid content capture, use:
```python
Agent.instrument_all(
InstrumentationSettings(
version=5,
include_content=False,
include_binary_content=False,
include_model_request_parameters=False,
tracer_provider=tracer_provider,
meter_provider=meter_provider,
)
)
```
With `include_content=False`, you still get span structure, model metadata, usage, latency, and errors. The tradeoff is that prompt content, tool arguments, and tool results will not be available in Parseable.
## Instrument all agents or one agent
`Agent.instrument_all(...)` applies to every agent that does not provide its own instrumentation capability. This is the simplest setup for most applications.
If different agents need different privacy settings or telemetry destinations, configure instrumentation per agent:
```python
from pydantic_ai import Agent, InstrumentationSettings
from pydantic_ai.capabilities import Instrumentation
settings = InstrumentationSettings(
version=5,
include_content=False,
tracer_provider=tracer_provider,
meter_provider=meter_provider,
)
agent = Agent(model, capabilities=[Instrumentation(settings=settings)])
```
## Token aggregation
Model request spans use standard token attributes such as `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Agent run spans use `gen_ai.aggregated_usage.*` by default, because they contain totals from their child model calls. This avoids double-counting when dashboards aggregate token usage across parent and child spans.
If your own queries already separate agent spans from model spans, you can disable the aggregated names:
```python
InstrumentationSettings(
version=5,
use_aggregated_usage_attribute_names=False,
)
```
Change this only if you know how your dashboards count usage, since it can affect totals.
## Verify in Parseable
Open the `pydantic-ai-traces` dataset from the Traces page. You should see agent, model, and tool spans in one trace. If the dataset is tagged with `agent-observability`, it will also appear in the Agent Observability flow.
Open the `pydantic-ai-metrics` dataset from the Metrics page. You should see metrics such as `gen_ai.client.token.usage`, `operation.cost`, and `gen_ai.client.operation.time_to_first_chunk` after the application has emitted enough model calls.
## Troubleshooting
* **No traces appear**
Confirm `Agent.instrument_all(...)` runs before the agent call, and check that the trace exporter points to `${PARSEABLE_URL}/v1/traces` with `X-P-Stream: pydantic-ai-traces`.
* **Metrics are missing**
Confirm the `MeterProvider` is passed to `InstrumentationSettings`, and check that the metric exporter points to `${PARSEABLE_URL}/v1/metrics` with `X-P-Stream: pydantic-ai-metrics`.
* **Tool arguments or results are missing**
Confirm `include_content=True` and run an agent path that actually calls a tool. If `include_content=False`, Pydantic AI intentionally removes prompts, completions, tool arguments, and tool results from telemetry.
* **Streaming latency is missing**
Use a streaming run and consume or close the stream. Pydantic AI records time-to-first-chunk only after the streaming path produces the required timing data.
* **Cost is missing**
Cost is recorded only when Pydantic AI can resolve pricing for the model. Token usage and latency should still be available.
## See also
* [Parseable Agent Observability](/user-guide/agent-observability)
* [LiteLLM SDK](/ingest-data/ai-agents/litellm-sdk)
* [Traces](/user-guide/traces)
* [Metrics](/user-guide/metrics)
# Restate (/ingest-data/ai-agents/restate-openai-agents)
import { Step, Steps } from 'fumadocs-ui/components/steps';
[Restate](https://restate.dev/) journals model calls and tool operations so interrupted agents can resume without repeating completed work. Parseable receives Restate runtime spans and OpenInference spans from the OpenAI Agents SDK in one trace.
## What the integration captures
* Restate ingress, invocation, retry, and journal spans
* OpenAI Agents workflow, agent, turn, and generation spans
* Tool calls and their arguments, results, and duration
* Model names, prompts, responses, token usage, and errors
`RestateTracer` assigns Restate's active invocation context to each OpenInference span. Parseable renders the complete execution as one trace.
## Prerequisites
* A running [Parseable](https://parseable.com/) instance
* Python 3.11 or later
* The [Restate Server and CLI](https://docs.restate.dev/develop/local_dev)
* An OpenAI API key
## Set up Restate with Parseable
### Install dependencies
```bash
pip install \
restate-sdk \
openai-agents \
openinference-instrumentation-openai-agents \
opentelemetry-sdk \
opentelemetry-exporter-otlp-proto-http \
hypercorn
```
### Configure the environment
Set the Parseable connection values and OpenAI API key:
```bash
export PARSEABLE_URL="https://"
export PARSEABLE_API_KEY=""
export PARSEABLE_STREAM="restate-openai-agent-traces"
export OPENAI_API_KEY=""
```
Parseable creates `restate-openai-agent-traces` when it receives the first spans.
### Define the agent
Create `agent.py`:
```python
import restate
from agents import Agent
from restate.ext.openai import (
DurableRunner,
durable_function_tool,
restate_context,
)
@durable_function_tool
async def get_weather(city: str) -> dict:
"""Get the current weather for a city."""
async def call_weather_api(city: str) -> dict:
return {
"city": city,
"temperature_celsius": 23,
"description": "Sunny and warm",
}
return await restate_context().run_typed(
"Get weather", call_weather_api, city=city
)
weather_agent = Agent(
name="WeatherAgent",
instructions=(
"Provide weather updates. Call get_weather before answering. "
"Include the city, temperature, and conditions."
),
tools=[get_weather],
)
agent_service = restate.Service("agent")
@agent_service.handler()
async def run(_ctx: restate.Context, message: str) -> str:
result = await DurableRunner.run(weather_agent, message)
return result.final_output
```
`durable_function_tool` journals the tool call. `restate_context().run_typed()` records the external operation so Restate can recover it without running it twice.
### Enable Parseable tracing
Create `telemetry.py`:
```python
import os
from agents import set_trace_processors
from openinference.instrumentation import OITracer, TraceConfig
from openinference.instrumentation.openai_agents._processor import (
OpenInferenceTracingProcessor,
)
from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from restate.ext.tracing import RestateTracer
parseable_url = os.environ["PARSEABLE_URL"].rstrip("/")
provider = TracerProvider(
resource=Resource.create(
{"service.name": "restate-openai-agents-demo"}
)
)
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=f"{parseable_url}/v1/traces",
headers={
"Authorization": f"Bearer {os.environ['PARSEABLE_API_KEY']}",
"X-P-Stream": os.getenv(
"PARSEABLE_STREAM", "restate-openai-agent-traces"
),
},
)
)
)
trace_api.set_tracer_provider(provider)
tracer = OITracer(
RestateTracer(trace_api.get_tracer("openinference.openai_agents")),
config=TraceConfig(),
)
set_trace_processors([OpenInferenceTracingProcessor(tracer)])
```
Add the following to the end of `agent.py`. Importing `telemetry` before creating the Restate application initializes the exporter and connects OpenInference spans to the active Restate invocation.
```python
import asyncio
import hypercorn.asyncio
from hypercorn.config import Config
import telemetry
app = restate.app(services=[agent_service])
async def serve() -> None:
config = Config()
config.bind = ["0.0.0.0:9080"]
await hypercorn.asyncio.serve(app, config)
if __name__ == "__main__":
try:
asyncio.run(serve())
finally:
telemetry.provider.shutdown()
```
The Python process exports the OpenInference spans. Restate Server exports ingress, invocation, retry, and recovery spans. Start Restate with the same Parseable dataset and authorization header:
```bash
export RESTATE_TRACING_HEADERS__AUTHORIZATION="Bearer ${PARSEABLE_API_KEY}"
export RESTATE_TRACING_HEADERS__X_P_STREAM="$PARSEABLE_STREAM"
restate-server \
--tracing-endpoint "otlp+${PARSEABLE_URL%/}/v1/traces" \
--tracing-filter "restate=info"
```
Send both exporters to the same dataset. Separate datasets break the trace hierarchy in Parseable.
### Run the agent
Start the application on port `9080`:
```bash
python agent.py
```
Register it with Restate:
```bash
restate deployments add --yes http://localhost:9080
```
Invoke the agent through Restate:
```bash
curl --fail-with-body \
http://localhost:8080/agent/run \
--json '"What is the weather in Bengaluru?"'
```
## View the run in Parseable
After the first invocation, add the `agent-observability` tag to `restate-openai-agent-traces` from the dataset settings. Open **Agents** and select the dataset to inspect prompts, responses, model details, token counts, tool calls, and duration.
## Production configuration
Route both OTLP streams through an OpenTelemetry Collector in production. Configure a sending queue and retry policy to handle temporary network failures. Keep the Python service and Restate Server on the same trace dataset.
Disable prompt and response capture when those fields may contain sensitive data. An OpenTelemetry Collector processor can remove specific attributes before export.
## Troubleshooting
* **Traces appear but the dataset is missing from Agents:** Add the `agent-observability` dataset tag. The Agents page only lists tagged datasets.
* **Only Restate spans appear:** Import `telemetry` before creating the Restate application and confirm that `OpenInferenceTracingProcessor` is registered.
* **Only agent spans appear:** Start Restate with `--tracing-endpoint`. Set `RESTATE_TRACING_HEADERS__AUTHORIZATION` and `RESTATE_TRACING_HEADERS__X_P_STREAM`.
* **The hierarchy splits across traces:** Wrap the OpenInference tracer with `RestateTracer` and send both exporters to the same dataset.
* **Duplicate workflow spans appear:** The OpenAI Agents SDK emits a workflow trace and a task span with the default `Agent workflow` name. They represent separate instrumentation levels.
* **The last spans are missing:** Call `provider.shutdown()` when a short-lived process exits so `BatchSpanProcessor` can flush buffered spans.
## Next steps
* Explore runs in [Agent Observability](/user-guide/agent-observability)
* Configure an [OpenTelemetry Collector](/ingest-data/logging-agents/otel-collector)
* Instrument another [OpenAI application](/ingest-data/ai-agents/openai)
# Restate (/ingest-data/ai-agents/restate)
[Restate](https://restate.dev/) is a durable execution engine that handles retries, state, and orchestration for workflows and services. Restate exports **traces** natively via OTLP. Service **logs** are emitted through the OpenTelemetry Logs SDK in your application code. Both signals flow through a single OpenTelemetry Collector that authenticates against Parseable and routes each signal to its own stream.
## Architecture
```
Restate server ──OTLP HTTP──▶ OTel Collector :4318 ──▶ Parseable restate-traces
Node.js service ──OTLP HTTP──▶ OTel Collector :4318 ──▶ Parseable restate-logs
```
The OTel Collector is the single point that holds Parseable credentials. Neither the Restate server nor the application process needs auth configuration.
Restate exports **traces only**: server-side spans covering ingress, invocation lifecycle, and retry attempts. It does not export metrics or logs via OTLP. Application logs must be instrumented separately using the OpenTelemetry Logs SDK.
## Prerequisites
* Restate Server `1.7.0` or later
* Node.js `20+` and `@restatedev/restate-sdk`
* Docker (for the OTel Collector)
* A reachable Parseable instance
## Step 1: Run the OTel Collector
Create `otel-collector-config.yaml`:
```yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
otlp_http/traces:
endpoint: https://
headers:
Authorization: "Bearer "
X-P-Stream: "restate-traces"
otlp_http/logs:
endpoint: https://
headers:
Authorization: "Bearer "
X-P-Stream: "restate-logs"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp_http/traces]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp_http/logs]
```
Create `docker-compose.yml`:
```yaml
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
command: ["--config=/etc/otel/config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel/config.yaml:ro
ports:
- "4318:4318"
restart: unless-stopped
```
Start the collector:
```bash
docker compose up -d
```
## Step 2: Start Restate with tracing
The `[tracing]` TOML section is silently ignored by `restate-server`. Use the `RESTATE_TRACING_ENDPOINT` environment variable instead.
```bash
RESTATE_TRACING_ENDPOINT=otlp+http://localhost:4318/v1/traces \
restate-server
```
The `otlp+http://` scheme prefix forces HTTP/1 transport. Without it, Restate defaults to gRPC, which requires a separate port (`4317`) and uses the HTTP/2 framing that some Parseable deployments do not accept.
Save this as a start script (`start-restate.sh`):
```bash
#!/bin/sh
RESTATE_TRACING_ENDPOINT=otlp+http://localhost:4318/v1/traces \
restate-server
```
## Step 3: Instrument service logs
Install the OTel Logs SDK:
```bash
npm install @opentelemetry/sdk-logs \
@opentelemetry/exporter-logs-otlp-http \
@opentelemetry/api-logs \
@opentelemetry/resources \
@opentelemetry/semantic-conventions
```
Create `src/telemetry.ts`:
```ts
import { LoggerProvider, BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import { logs } from "@opentelemetry/api-logs";
const exporter = new OTLPLogExporter({
url: "http://localhost:4318/v1/logs",
});
const provider = new LoggerProvider({
resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: "restate-service" }),
processors: [new BatchLogRecordProcessor(exporter)],
});
logs.setGlobalLoggerProvider(provider);
export const logger = logs.getLogger("restate-service");
export async function shutdownTelemetry() {
await provider.shutdown();
}
```
Use the logger in your Restate handlers instead of `console.log`:
```ts
import * as restate from "@restatedev/restate-sdk";
import { logger } from "./telemetry.js";
import { SeverityNumber } from "@opentelemetry/api-logs";
const exampleWorkflow = restate.workflow({
name: "ExampleWorkflow",
handlers: {
run: async (ctx: restate.WorkflowContext, name: string): Promise => {
const greeting = await ctx.serviceClient(greeterSvc).greet(name);
await ctx.run("emit-completed", async () => {
logger.emit({
severityNumber: SeverityNumber.INFO,
severityText: "INFO",
body: `ExampleWorkflow done: ${greeting}`,
attributes: { workflow: "ExampleWorkflow", name },
});
});
return greeting;
},
},
});
```
Wrap `logger.emit()` inside `ctx.run()` to make log emission durable and replay-safe. Restate replays handler code on retry; calls outside `ctx.run()` execute on every replay and produce duplicate log records.
## Step 4: Register and run
Start the service:
```bash
npx tsx src/app.ts
```
Register the deployment:
```bash
restate deployments register http://localhost:9080
```
Invoke a workflow:
```bash
restate workflow submit ExampleWorkflow my-id '"World"'
```
## Verify
Check the collector is forwarding both signals:
```bash
docker compose logs otel-collector | grep "Preparing to make HTTP request"
```
You should see requests to both `/v1/traces` and `/v1/logs`.
Query Parseable:
```sql
-- Recent traces
SELECT p_timestamp, name, trace_id, span_id, duration_nano
FROM "restate-traces"
WHERE p_timestamp > now() - INTERVAL '5 minutes'
ORDER BY p_timestamp DESC
LIMIT 20;
```
```sql
-- Service logs
SELECT p_timestamp, body, severity_text
FROM "restate-logs"
WHERE p_timestamp > now() - INTERVAL '5 minutes'
ORDER BY p_timestamp DESC
LIMIT 20;
```
## What gets captured
### Traces (`restate-traces`)
These are **Restate server spans** exported via `RESTATE_TRACING_ENDPOINT`. The server emits one trace per invocation:
| Span | Description |
| -------------------- | ----------------------------------------------------------------- |
| `ingress` | Root span for the HTTP entry point where the request was accepted |
| `invocation` | Logical invocation; child of `ingress` |
| `invocation_attempt` | One span per execution attempt; retries produce new attempt spans |
| `invocation_end` | Marks completion (success, failure, or cancellation) |
`ctx.run()` blocks, `ctx.sleep()`, and outbound service calls appear as **events on the `invocation_attempt` span**, not as child spans. Attempt spans carry `restate.invocation.id`, `restate.service.name`, and `restate.handler.name` as attributes.
In Parseable, the trace stream gives you a timeline of Restate invocations and the span attributes attached to each attempt. This is where you can inspect duration, invocation metadata, and retry behavior for a workflow.
### Logs (`restate-logs`)
Logs are application-defined. The OTel SDK attaches `service.name` and any attributes passed to `logger.emit()`. A minimal schema to adopt:
| Field | Notes |
| --------------- | --------------------------------------------- |
| `body` | Human-readable message |
| `severity_text` | `INFO`, `WARN`, `ERROR` |
| `service.name` | Set in `resourceFromAttributes` |
| `workflow` | Add as attribute for cross-signal correlation |
In the logs stream, Parseable shows the emitted application records along with severity, service attributes, and workflow attributes. This makes it easier to connect what the service wrote with the Restate invocation you are investigating.
## Troubleshooting
* **No traces in collector**: The `[tracing]` TOML key is silently ignored. Confirm `RESTATE_TRACING_ENDPOINT` is set in the environment, not in the config file.
* **gRPC connection refused or FRAME\_SIZE\_ERROR**: Remove any `grpc://` or bare `http://` prefix. Use `otlp+http://localhost:4318/v1/traces` to force HTTP/1.
* **Duplicate log records on retry**: `logger.emit()` calls must be inside `ctx.run()`. Code outside `ctx.run()` re-executes on every Restate replay.
* **Collector deprecation warning `"otlphttp" alias is deprecated`**: Use `otlp_http/` as the exporter key (underscore, not camel-case). Both work; `otlp_http/` suppresses the warning.
# vLLM (/ingest-data/ai-agents/vllm)
Monitor vLLM inference workloads with OpenTelemetry and Parseable. Collect metrics, build dashboards, and analyze GPU costs for production LLM serving.
## Overview
[vLLM](https://github.com/vllm-project/vllm) is a fast and easy-to-use library for LLM inference and serving. Originally developed at UC Berkeley, vLLM has evolved into a community-driven project for high-performance model serving.
Integrate vLLM with Parseable to:
* **Monitor Inference Performance** - Track latency, throughput, and GPU utilization
* **Analyze Token Usage** - Measure input/output tokens and costs
* **Debug Issues** - Identify slow requests and errors
* **Optimize Resources** - Right-size GPU infrastructure based on metrics
## Architecture
```
┌─────────────┐ ┌──────────────┐ ┌────────────┐ ┌────────────┐
│ vLLM │───▶│ Metrics │───▶│ OTel │───▶│ Parseable │
│ Service │ │ Proxy │ │ Collector │ │ │
└─────────────┘ └──────────────┘ └────────────┘ └────────────┘
↓ ↓ ↓ ↓
Metrics Sanitization Collection Observability
```
## Prerequisites
* vLLM deployment with metrics endpoint accessible
* Docker or Podman with Compose
* Parseable instance (local or cloud)
## Quick Start
### 1. Clone the Repository
```bash
git clone https://github.com/opensourceops/vllm-inference-metrics.git
cd vllm-inference-metrics
```
### 2. Configure vLLM Endpoint
Edit `compose.yml` to point to your vLLM deployment:
```yaml
services:
proxy:
environment:
- VLLM_METRICS_URL=https://your-vllm-endpoint/metrics
```
For local vLLM deployments:
```yaml
environment:
- VLLM_METRICS_URL=http://localhost:8000/metrics
```
### 3. Start the Stack
Using Docker:
```bash
docker compose -f compose-otel.yml up -d
```
Using Podman:
```bash
podman compose -f compose-otel.yml up -d
```
### 4. Access Services
| Service | URL | Credentials |
| ---------------- | ------------------------ | ----------- |
| Parseable UI | `localhost:8080` | admin/admin |
| Metrics endpoint | `localhost:9090/metrics` | - |
| Health check | `localhost:9090/health` | - |
### 5. Verify Metrics Collection
```bash
# View proxy metrics
curl http://localhost:9090/metrics
# Check OTel Collector logs
docker compose logs -f otel-collector
# Query metrics in Parseable
curl -X POST http://localhost:8080/api/v1/query \
-H "Authorization: Basic YWRtaW46YWRtaW4=" \
-d '{"query": "SELECT * FROM vLLMmetrics LIMIT 10"}'
```
## Docker Compose Configuration
Complete `compose-otel.yml` example:
```yaml
services:
parseable:
image: quay.io/parseablehq/parseable:edge
command: ["parseable", "local-store"]
env_file: ./parseable.env
volumes:
- parseable-staging:/staging
ports: ["8080:8000"]
restart: unless-stopped
proxy:
image: python:3.11-alpine
volumes: ["./proxy.py:/app/proxy.py:ro"]
environment:
- VLLM_METRICS_URL=
command: >
sh -c "pip install --no-cache-dir flask requests && python /app/proxy.py"
ports: ["9090:9090"]
restart: unless-stopped
depends_on: [parseable]
otel-collector:
image: otel/opentelemetry-collector:latest
command: ["--config=/etc/otel-config.yaml"]
volumes:
- ./otel-config.yaml:/etc/otel-config.yaml:ro
ports:
- "4317:4317" # OTLP/gRPC
- "4318:4318" # OTLP/HTTP
restart: unless-stopped
depends_on:
proxy:
condition: service_healthy
parseable:
condition: service_started
volumes:
parseable-staging:
```
## Environment Variables
| Variable | Description | Default |
| ------------------ | --------------------------- | -------- |
| `VLLM_METRICS_URL` | vLLM metrics endpoint URL | Required |
| `P_USERNAME` | Parseable username | admin |
| `P_PASSWORD` | Parseable password | admin |
| `SCRAPE_INTERVAL` | Metrics collection interval | 2s |
## Key Metrics
| Metric Name | Type | Description |
| ------------------------------------ | --------- | ----------------------------- |
| `vllm_num_requests_running` | Gauge | Active inference requests |
| `vllm_num_requests_waiting` | Gauge | Queued requests |
| `vllm_gpu_cache_usage_perc` | Gauge | GPU KV-cache utilization |
| `vllm_num_preemptions_total` | Counter | Request preemptions |
| `vllm_prompt_tokens_total` | Counter | Total prompt tokens processed |
| `vllm_generation_tokens_total` | Counter | Total tokens generated |
| `vllm_request_latency_seconds` | Histogram | End-to-end request latency |
| `vllm_time_to_first_token_seconds` | Histogram | TTFT latency |
| `vllm_time_per_output_token_seconds` | Histogram | Inter-token latency |
## Example Queries
### Request Latency Analysis
```sql
SELECT
DATE_TRUNC('minute', p_timestamp) AS minute,
AVG(vllm_request_latency_seconds) AS avg_latency,
MAX(vllm_request_latency_seconds) AS max_latency
FROM vLLMmetrics
WHERE p_timestamp > NOW() - INTERVAL '1 hour'
GROUP BY minute
ORDER BY minute;
```
### GPU Cache Utilization
```sql
SELECT
p_timestamp,
vllm_gpu_cache_usage_perc
FROM vLLMmetrics
WHERE p_timestamp > NOW() - INTERVAL '30 minutes'
ORDER BY p_timestamp;
```
### Token Throughput
```sql
SELECT
DATE_TRUNC('hour', p_timestamp) AS hour,
SUM(vllm_prompt_tokens_total) AS input_tokens,
SUM(vllm_generation_tokens_total) AS output_tokens
FROM vLLMmetrics
GROUP BY hour
ORDER BY hour DESC
LIMIT 24;
```
## Cost Analysis
Track GPU costs with token metrics:
```sql
SELECT
DATE_TRUNC('day', p_timestamp) AS day,
COUNT(*) AS total_requests,
SUM(vllm_prompt_tokens_total + vllm_generation_tokens_total) AS total_tokens,
-- A100 PCIe at $1.64/hour example
ROUND(COUNT(*) * 0.001, 2) AS estimated_cost_usd
FROM vLLMmetrics
GROUP BY day
ORDER BY day DESC;
```
## Alerting
Set up alerts in Parseable for:
* **High Latency**: Alert when `vllm_request_latency_seconds` exceeds threshold
* **Queue Buildup**: Alert when `vllm_num_requests_waiting` grows
* **GPU Memory**: Alert when `vllm_gpu_cache_usage_perc` approaches 100%
## Resources
* [GitHub Repository](https://github.com/opensourceops/vllm-inference-metrics)
* [vLLM Documentation](https://docs.vllm.ai/)
* [Blog Post](https://www.parseable.com/blog/vllm-inference-metrics-otel)
# AWS CloudWatch (/ingest-data/cloud/aws-cloudwatch)
Collect and forward logs from AWS CloudWatch Log Groups to Parseable using the OpenTelemetry Collector.
## Overview
Integrate AWS CloudWatch with Parseable to:
* **Centralize AWS Logs** - Collect logs from Lambda, ECS, EC2, and other AWS services
* **Unified Observability** - Combine AWS logs with application logs
* **Cost Optimization** - Reduce CloudWatch costs by forwarding to Parseable
* **Advanced Analytics** - Use Parseable's SQL queries on AWS logs
## Prerequisites
* AWS account with CloudWatch Logs
* AWS credentials with CloudWatch read permissions
* OpenTelemetry Collector with `awscloudwatch` receiver
* Parseable instance running and accessible
### IAM Permissions
Create an IAM policy with the required permissions:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
}
]
}
```
## OpenTelemetry Collector Configuration
### Basic Configuration
Create an `otel-collector-config.yaml` file:
```yaml
receivers:
awscloudwatch:
region: us-east-1
logs:
poll_interval: 1m
groups:
autodiscover:
limit: 100
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "cloudwatch-logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [awscloudwatch]
exporters: [otlphttp/parseable]
```
### Filter Specific Log Groups
Collect logs from specific log groups only:
```yaml
receivers:
awscloudwatch:
region: us-east-1
logs:
poll_interval: 1m
groups:
named:
/aws/lambda/my-function:
/aws/ecs/my-cluster:
/aws/apigateway/my-api:
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "cloudwatch-logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [awscloudwatch]
exporters: [otlphttp/parseable]
```
### Autodiscover with Prefix Filter
Discover log groups matching a prefix:
```yaml
receivers:
awscloudwatch:
region: us-east-1
logs:
poll_interval: 1m
groups:
autodiscover:
limit: 50
prefix: /aws/lambda/
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "lambda-logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [awscloudwatch]
exporters: [otlphttp/parseable]
```
### Filter Log Streams
Filter specific log streams within log groups:
```yaml
receivers:
awscloudwatch:
region: us-east-1
logs:
poll_interval: 1m
groups:
named:
/aws/lambda/my-function:
names:
- "2024/01/15/[$LATEST]abc123"
prefixes:
- "2024/01/"
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "lambda-logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [awscloudwatch]
exporters: [otlphttp/parseable]
```
## Configuration Options
### Top Level Parameters
| Parameter | Required | Description |
| --------------- | -------- | ------------------------------ |
| `region` | Yes | AWS region (e.g., `us-east-1`) |
| `profile` | No | AWS profile name |
| `imds_endpoint` | No | Custom IMDS endpoint for EC2 |
| `logs` | No | Logs collection configuration |
### Logs Parameters
| Parameter | Default | Description |
| ------------------------ | ------- | --------------------------------- |
| `poll_interval` | `1m` | Time between log requests |
| `max_events_per_request` | `1000` | Max events per CloudWatch request |
| `groups` | All | Log group configuration |
### Group Parameters
| Parameter | Description |
| --------------------- | ------------------------------ |
| `autodiscover.limit` | Max log groups to discover |
| `autodiscover.prefix` | Log group name prefix filter |
| `named` | Specific log groups to collect |
## Running the Collector
### Docker with AWS Credentials
```bash
docker run -d \
--name otel-collector \
-v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
-v ~/.aws:/root/.aws:ro \
-e AWS_REGION=us-east-1 \
otel/opentelemetry-collector-contrib:latest
```
### Docker with Environment Variables
```bash
docker run -d \
--name otel-collector \
-v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
-e AWS_ACCESS_KEY_ID=your-access-key \
-e AWS_SECRET_ACCESS_KEY=your-secret-key \
-e AWS_REGION=us-east-1 \
otel/opentelemetry-collector-contrib:latest
```
### Kubernetes with IAM Roles for Service Accounts (IRSA)
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: otel-collector
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/OtelCollectorRole
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
spec:
template:
spec:
serviceAccountName: otel-collector
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:latest
volumeMounts:
- name: config
mountPath: /etc/otelcol/config.yaml
subPath: config.yaml
volumes:
- name: config
configMap:
name: otel-collector-config
```
## Querying CloudWatch Logs in Parseable
Once data is flowing, query your CloudWatch logs:
```sql
-- Get recent Lambda logs
SELECT p_timestamp, log_group, log_stream, message
FROM "cloudwatch-logs"
WHERE log_group LIKE '/aws/lambda/%'
ORDER BY p_timestamp DESC
LIMIT 100
-- Find Lambda errors
SELECT p_timestamp, log_group, message
FROM "cloudwatch-logs"
WHERE log_group LIKE '/aws/lambda/%'
AND (message LIKE '%ERROR%' OR message LIKE '%Exception%')
ORDER BY p_timestamp DESC
-- Count logs by log group
SELECT log_group, COUNT(*) as log_count
FROM "cloudwatch-logs"
WHERE p_timestamp > NOW() - INTERVAL '1 hour'
GROUP BY log_group
ORDER BY log_count DESC
```
## Troubleshooting
### Authentication Issues
1. Verify AWS credentials are configured correctly
2. Check IAM permissions include required CloudWatch actions
3. Verify the region matches your log groups
4. Check for credential expiration (if using temporary credentials)
### Missing Logs
1. Verify log groups exist and have recent logs
2. Check `poll_interval` is appropriate for your log volume
3. Verify autodiscover prefix matches your log group names
4. Check CloudWatch Logs retention settings
### High Latency
1. Reduce `poll_interval` for more frequent collection
2. Increase `max_events_per_request` for higher throughput
3. Filter to specific log groups to reduce API calls
## Next Steps
* Set up [alerts](/user-guide/alerting) for AWS log patterns
* Create [dashboards](/user-guide/dashboards) for AWS monitoring
* Configure [AWS S3](/self-hosted/storage-targets/aws-s3) for log storage
# AWS Data Firehose (/ingest-data/cloud/aws-data-firehose)
### Introduction
Amazon Data Firehose is a fully managed service for delivering real-time streaming data to various destination.
* Firehose dataset: The underlying entity of Amazon Data Firehose. You use Amazon Data Firehose by creating a Firehose dataset and then sending data to it.
* Record: The data of interest that your data producer sends to a Firehose dataset. A record can be as large as 1,000 KB.
* Data producer: Producers send records to Firehose streams. For example, a web server that sends log data to a Firehose dataset is a data producer. You can also configure your Firehose dataset to automatically read data from an existing Kinesis data dataset, and load it into destinations.
* Buffer size and buffer interval: Amazon Data Firehose buffers incoming streaming data to a certain size or for a certain period of time before delivering it to destinations. Buffer Size is in MBs and Buffer Interval is in seconds.
### Overview
You can send data to Parseable using Amazon Data Firehose dataset, and Parseable will automatically ingest the data and make it available for you to query and analyze.
Once logs are unified in Parseable, you can easily monitor and analyze the logs in real-time, and get insights into the performance and usage of your APIs.
### Pre-requisites
* Parseable deployed and running in your environment. Refer the [installation guide](/self-hosted/installation) for more information.
* AWS account with access to Amazon Data Firehose service.
### Create a Firehose dataset
First step is to create a Firehose dataset to ingest data from various sources and send it to Parseable. In the AWS Management Console, navigate to the Amazon Data Firehose service and click on `Create Firehose dataset`.
In the next screen, you can choose from various sources like Kinesis data dataset, Amazon MSK (Kafka), and Direct PUT (i.e. send data directly to Firehose dataset from an application).
### Destination settings
In the next step, you can configure the destination settings. Choose HTTP Endpoint as the destination. After that we'll need to provide the endpoint URL, Basic Auth token, and the Parseable dataset where you want to send the data.
For example to send events to the demo Parseable instance at `https://demo.parseable.com`:
* Endpoint URL will be `https://demo.parseable.com/api/v1/ingest`.
* Under Parameters, add the following:
* Authorization header with the value `Basic YWRtaW46YWRtaW4=`.
* Content-Type header with the value `application/json`.
* X-P-Stream header with the value `kinesisdata`.
### Backup settings
To ensure that no data is lost, you can configure the backup settings. You can choose to deliver the data to an S3 bucket in case the data can't be delivered to Parseable.
That's it. You have successfully created a Firehose dataset to ingest data from various sources and send it to Parseable. Now you can start sending data to the Firehose dataset and you'll see the data in Parseable available for query and analysis.
# AWS Kinesis (/ingest-data/cloud/aws-kinesis)
Stream logs from AWS Kinesis Data Streams and Kinesis Data Firehose to Parseable.
## Overview
Integrate AWS Kinesis with Parseable to:
* **Real-time Streaming** - Ingest logs as they arrive
* **High Throughput** - Handle massive log volumes
* **AWS Integration** - Collect logs from AWS services
* **Buffering** - Use Firehose for batched delivery
## Prerequisites
* AWS account with Kinesis access
* Parseable instance accessible from AWS
* IAM permissions for Kinesis
## Method 1: Kinesis Data Firehose
Use Firehose for managed delivery to Parseable's HTTP endpoint.
### Create Delivery Stream
1. Go to **Kinesis** → **Data Firehose**
2. Click **Create delivery dataset**
3. Configure source (Direct PUT or Kinesis Data Stream)
4. Select **HTTP Endpoint** as destination
5. Configure endpoint:
| Field | Value |
| ----------------- | -------------------------------------- |
| HTTP endpoint URL | `https://your-parseable/api/v1/ingest` |
| Content encoding | GZIP (recommended) |
| Access key | Your Parseable credentials |
### Configure Buffering
```
Buffer size: 5 MB
Buffer interval: 60 seconds
```
### IAM Role
Firehose needs permissions to deliver to HTTP endpoint:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:PutRecord",
"firehose:PutRecordBatch"
],
"Resource": "arn:aws:firehose:*:*:deliverystream/parseable-*"
}
]
}
```
## Method 2: Lambda Consumer
Use Lambda to consume from Kinesis and forward to Parseable.
### Lambda Function
```javascript
// index.js
const https = require('https');
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;
const STREAM_NAME = process.env.STREAM_NAME || 'kinesis-logs';
exports.handler = async (event) => {
const logs = event.Records.map(record => {
const payload = Buffer.from(record.kinesis.data, 'base64').toString('utf-8');
try {
return JSON.parse(payload);
} catch {
return { message: payload, timestamp: new Date().toISOString() };
}
});
const url = new URL(PARSEABLE_URL);
const options = {
hostname: url.hostname,
port: url.port || 443,
path: '/api/v1/ingest',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Basic ${PARSEABLE_AUTH}`,
'X-P-Stream': STREAM_NAME
}
};
return new Promise((resolve, reject) => {
const req = https.request(options, (res) => {
if (res.statusCode >= 200 && res.statusCode < 300) {
resolve({ statusCode: 200, body: 'Success' });
} else {
reject(new Error(`HTTP ${res.statusCode}`));
}
});
req.on('error', reject);
req.write(JSON.stringify(logs));
req.end();
});
};
```
### Lambda Configuration
```yaml
# serverless.yml
functions:
kinesisConsumer:
handler: index.handler
events:
- dataset:
type: kinesis
arn: arn:aws:kinesis:region:account:dataset/your-dataset
batchSize: 100
startingPosition: LATEST
environment:
PARSEABLE_URL: https://your-parseable.com
PARSEABLE_AUTH: ${ssm:/parseable/auth}
STREAM_NAME: kinesis-logs
```
## Method 3: OpenTelemetry Collector
Use OTel Collector with Kinesis receiver.
### Configuration
```yaml
receivers:
awskinesis:
stream_name: your-kinesis-dataset
region: us-east-1
encoding: json
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "kinesis-logs"
service:
pipelines:
logs:
receivers: [awskinesis]
exporters: [otlphttp/parseable]
```
## Best Practices
1. **Use Firehose** - For managed, reliable delivery
2. **Enable Compression** - Reduce data transfer costs
3. **Configure Retries** - Handle transient failures
4. **Monitor Metrics** - Track delivery success rates
5. **Set Buffer Sizes** - Balance latency vs efficiency
## Troubleshooting
### Delivery Failures
1. Check Parseable endpoint is accessible
2. Verify authentication credentials
3. Check CloudWatch logs for errors
4. Verify IAM permissions
### Data Loss
1. Enable S3 backup in Firehose
2. Configure dead letter queue
3. Monitor iterator age
## Next Steps
* Set up [AWS CloudWatch](/ingest-data/cloud/aws-cloudwatch) for CloudWatch logs
* Configure [alerts](/user-guide/alerting) for streaming metrics
* Create [dashboards](/user-guide/dashboards) for monitoring
# AWS Lambda (/ingest-data/cloud/aws-lambda)
Parseable AWS Lambda extension is a Lambda extension that allows you to send logs from your Lambda functions to your Parseable instance.
## Usage
To use the parseable-lambda-extension with a lambda function, it must be configured as a layer. There are two variants of the extension available: one for x86\_64 architecture and one for arm64 architecture.
You can add the extension as a layer with the AWS CLI tool:
```bash
$ aws lambda update-code-configuration \
--function-name MyAwesomeFunction
--layers ""
```
The extension's layer version ARN follows the pattern below.
```bash
# Layer Version ARN Pattern
arn:aws:lambda::724973952305:layer:parseable-lambda-extension--:1
```
* AWS\_REGION - This must match the region of the Lambda function to which you are adding the extension.
* ARCH - `x86_64` or `arm64`.
* VERSION - The version of the extension you want to use. Current version is v1.0. For current latest release `v1.0`, use the value `v1-0`.
## Configuration
The extension is configurable via environment variables set for your lambda function.
* `PARSEABLE_LOG_URL` - Parseable endpoint URL. It should be set to `https:///api/v1/ingest`. Change `` to your Parseable instance URL. (required)
* `PARSEABLE_USERNAME` - Username set for your Parseable instance. (required)
* `PARSEABLE_PASSWORD` - Password set for your Parseable instance. (required)
* `PARSEABLE_LOG_STREAM` - Parseable dataset name where you want to ingest logs. (default: `Lambda Function Name`).
Refer [Parseable installation documentation](/self-hosted/installation/) for more details.
## Container image lambda
In case if you deploy your lambda as container image, to inject extension as part of your function just copy it to your image:
```bash
FROM parseable/aws-lambda-extension:latest AS parseable-extension
FROM public.ecr.aws/lambda/python:3.8
# Layer code
WORKDIR /opt
COPY --from=parseable-extension /opt/ .
# Function code
WORKDIR /var/task
COPY app.py .
CMD app.lambda_handler
```
# Azure API Service (/ingest-data/cloud/azure-api-service)
## API Management Service Logs
Azure API Management (APIM) is a hybrid, multi cloud management platform for APIs across all environments. It provides a scalable, multi-cloud API management platform for securing, publishing, and analyzing APIs.
Azure API Management helps customers:
* Abstract backend architecture diversity and complexity from API consumers
* Securely expose services hosted on and outside of Azure as APIs
* Protect, accelerate, and observe APIs
* Enable API discovery and consumption by internal and external users
Azure API Management is made up of an API gateway, a management plane, and a developer portal. These components are Azure-hosted and fully managed by default.
## Overview
You can send Azure API Management Service logs to Parseable via Azure Event Hub & Azure Logic Apps. Once logs are unified in Parseable, you can easily monitor and analyze the logs in real-time, and get insights into the performance and usage of your APIs.
Most importantly, you have full ownership and control over this data, so you can use it to improve your API management and security, without cost or privacy concerns.
## Pre-requisites
Before you begin, you will need the following:
* Parseable deployed and running in your environment. Refer the [installation guide](/self-hosted/installation/) for more information.
* Active Azure subscription with APIM service created. Refer the [Azure API Management documentation](https://learn.microsoft.com/en-us/azure/api-management/authentication-authorization-overview) for more information.
* Azure Event Hub created. Refer the [Create an event hub using Azure portal documentation](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-enable-through-portal) for more information.
## Send Azure APIM Service Logs to Event Hub
* Navigate to the Azure API Management Service in the Azure portal.
* Click on the `Diagnostics settings` under the `Monitoring` section.
* Configure the APIM service to send logs to the Event Hub. Refer the image below for more information.
## Use Azure Logic Apps to send events to Parseable
* Navigate to the Azure Logic Apps in the Azure portal.
* Click on the `+ Add` button to create a new Logic App.
* Configure the Logic App to trigger when an event is received in the Event Hub. Refer the image below for more information. We have used the `When events are available in Event Hub` trigger to start the Logic App.
With this, you should now be able to see the APIM service logs in Parseable. You can now monitor and analyze the logs in real-time, and get insights into the performance and usage of your APIs. You have a detailed view of the API management and security, without cost or privacy concerns.
# Azure Event Hubs (/ingest-data/cloud/azure-event-hubs)
Stream logs from Azure Event Hubs to Parseable for centralized observability.
## Overview
Integrate Azure Event Hubs with Parseable to:
* **Stream Azure Logs** - Collect logs from Azure services
* **High Throughput** - Handle millions of events per second
* **Real-time Processing** - Process logs as they arrive
* **Azure Integration** - Native Azure ecosystem support
## Prerequisites
* Azure subscription with Event Hubs
* Event Hubs namespace and hub created
* Parseable instance accessible from Azure
* Connection string or managed identity
## Method 1: Azure Functions
Use Azure Functions to consume events and forward to Parseable.
### Function Code
```javascript
// index.js
const axios = require('axios');
module.exports = async function (context, eventHubMessages) {
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;
const logs = eventHubMessages.map(msg => {
if (typeof msg === 'string') {
try {
return JSON.parse(msg);
} catch {
return { message: msg, timestamp: new Date().toISOString() };
}
}
return msg;
});
try {
await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, logs, {
headers: {
'Authorization': `Basic ${PARSEABLE_AUTH}`,
'X-P-Stream': 'azure-events',
'Content-Type': 'application/json'
}
});
context.log(`Sent ${logs.length} events to Parseable`);
} catch (error) {
context.log.error('Error sending to Parseable:', error);
throw error;
}
};
```
### Function Configuration
```json
// function.json
{
"bindings": [
{
"type": "eventHubTrigger",
"name": "eventHubMessages",
"direction": "in",
"eventHubName": "your-event-hub",
"connection": "EventHubConnection",
"cardinality": "many",
"consumerGroup": "$Default"
}
]
}
```
### Application Settings
```
PARSEABLE_URL=https://your-parseable.com
PARSEABLE_AUTH=base64-encoded-credentials
EventHubConnection=Endpoint=sb://...
```
## Method 2: OpenTelemetry Collector
Use OTel Collector with Azure Event Hubs receiver.
### Configuration
```yaml
receivers:
azureeventhub:
connection: ${EVENTHUB_CONNECTION}
format: json
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "azure-events"
service:
pipelines:
logs:
receivers: [azureeventhub]
exporters: [otlphttp/parseable]
```
## Method 3: Fluent Bit
Use Fluent Bit with Azure Event Hubs input.
### Configuration
```yaml
service:
flush: 5
log_level: info
pipeline:
inputs:
- name: azure_event_hub
connection_string: ${EVENTHUB_CONNECTION}
hub_name: your-event-hub
consumer_group: $Default
outputs:
- name: http
match: '*'
host: parseable
port: 8000
uri: /api/v1/ingest
format: json
header: Authorization Basic YWRtaW46YWRtaW4=
header: X-P-Stream azure-events
```
## Diagnostic Settings
Export Azure resource logs to Event Hubs:
1. Go to your Azure resource
2. **Monitoring** → **Diagnostic settings**
3. Click **Add diagnostic setting**
4. Select log categories
5. Choose **Stream to an event hub**
6. Select your Event Hubs namespace and hub
## Best Practices
1. **Use Consumer Groups** - Separate consumers for different purposes
2. **Configure Partitions** - Scale throughput with partitions
3. **Enable Checkpointing** - Track processing progress
4. **Monitor Lag** - Watch for processing delays
5. **Handle Retries** - Implement retry logic
## Troubleshooting
### Connection Issues
1. Verify connection string is correct
2. Check network connectivity
3. Verify firewall rules allow access
4. Check managed identity permissions
### Missing Events
1. Verify consumer group configuration
2. Check checkpoint storage
3. Monitor Event Hubs metrics
## Next Steps
* Set up [Azure Blob Storage](/self-hosted/storage-targets/azure-blob-storage) for log storage
* Configure [alerts](/user-guide/alerting) for Azure metrics
* Create [dashboards](/user-guide/dashboards) for monitoring
# Azure Service Bus (/ingest-data/cloud/azure-service-bus)
### Service Bus Messages
Azure Service Bus is a fully managed enterprise integration message broker. Service Bus is most commonly used to decouple applications and services from each other, and is a reliable and secure platform for asynchronous data and state transfer.
Service Bus is used to decouple applications and services from each other, providing the following benefits:
* Load-balancing work across competing workers
* Safely routing and transferring data and control across service and application boundaries
* Coordinating transactional work that requires a high-degree of reliability
### Overview
You can send messages published to Azure Service Bus Queue to Parseable [Azure Logic Apps](https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-overview) service.
We'll use the Azure Logic Apps trigger "When a message is received in a queue" to send message to Parseable.
### Prerequisites
Before you begin, you will need the following:
* Parseable deployed and running in your environment. Refer the [installation guide](/self-hosted/installation/) for more information.
* Active Azure subscription with Service Bus namespace and queue created. Refer the [Azure Service Bus documentation](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview) for more information.
### Use Azure Logic Apps to send Service Bus messages to Parseable
* Navigate to the Azure Logic Apps in the Azure portal.
* Click on the `+ Add` button to create a new Logic App.
* Configure the Logic App to trigger when an message is available in the Message queue. Refer the image below for more information. We have used the `When a message is received in a queue` trigger to start the Logic App.
# Fly.io (/ingest-data/cloud/fly-io)
Ship logs and platform metrics from your [Fly.io](https://fly.io) apps to Parseable. Both pipelines are dedicated Fly apps deployed alongside (not inside) your workload apps; your existing `fly.toml` files are not modified.
## Overview
| Pipeline | Source | Shipper | Parseable route | Availability |
| -------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ---------------------------- | ------------------------------------- |
| Logs | Fly's NATS log stream (`nats://[fdaa::3]:4223`) | [`fly-log-shipper`](https://github.com/superfly/fly-log-shipper) with Parseable Vector sink | `POST /api/v1/logstream/...` | All Parseable editions |
| Metrics | Fly's managed Prometheus (`api.fly.io/prometheus//federate`) | [`parseable-fly-metrics`](https://github.com/parseablehq/parseable-fly-io-metrics) (Go agent) | `POST /v1/prometheus/write` | **Parseable Enterprise / Prism only** |
One agent per Fly org covers every app in that org - no per-app config.
***
## Logs
### Architecture
```
your-app-1 ─┐
your-app-2 ─┤── Fly NATS logs stream ──▶ fly-log-shipper ──▶ Parseable
your-app-N ─┘ (nats://[fdaa::3]:4223) (Fly app) /api/v1/logstream/
```
### Deploy
`fly-log-shipper` is Fly's official log forwarder, with a Parseable sink available in the upstream repo (see [PR #76](https://github.com/superfly/fly-log-shipper/pull/76)).
```bash
git clone https://github.com/superfly/fly-log-shipper
cd fly-log-shipper
# create the shipper app in your Fly org
flyctl apps create -fly-log-shipper --org
# auth + Parseable sink secrets
flyctl secrets set \
ACCESS_TOKEN=$(flyctl auth token) \
ORG= \
PARSEABLE_URL=https:// \
PARSEABLE_USERNAME= \
PARSEABLE_PASSWORD= \
PARSEABLE_STREAM=fly_logs
flyctl deploy --ha=false
```
Logs from every app in the org now flow into the `fly_logs` stream. Refer to the [fly-log-shipper README](https://github.com/superfly/fly-log-shipper#sinks) for the full list of sink env vars.
***
## Metrics
The Fly metrics pipeline writes via Parseable's `POST /v1/prometheus/write` route, which is available in **Parseable Enterprise** and **Prism** (distributed) deployments. The open-source Parseable server does not expose this route.
### Architecture
```
your-app-1 ─┐
your-app-2 ─┤── Fly managed Prometheus ──▶ parseable-fly-metrics ──▶ Parseable
your-app-N ─┘ api.fly.io/prometheus//federate (Fly app) /v1/prometheus/write
(ingestor node)
```
### Deploy
```bash
git clone https://github.com/parseablehq/parseable-fly-metrics
cd parseable-fly-metrics
# edit fly.toml - set `app = ""` and `primary_region`
flyctl apps create --org
flyctl secrets set \
FLY_ORG= \
FLY_METRICS_TOKEN=$(flyctl auth token) \
PARSEABLE_URL=https://:8010 \
PARSEABLE_USERNAME= \
PARSEABLE_PASSWORD=
flyctl deploy --ha=false
```
Expected log line per scrape interval:
```
scrape ok: families=61 series=134 samples=134 duration=460ms
```
`PARSEABLE_URL` must point at a Prism **ingestor** node, not the query node. Run a single agent machine per org; multiple instances would double-scrape.
### Configuration
| Variable | Required | Default | Notes |
| ---------------------- | -------- | ------------- | ------------------------------------------------------------------------------ |
| `FLY_ORG` | yes | - | Fly org slug. Find via `flyctl orgs list`. |
| `FLY_METRICS_TOKEN` | yes | - | Output of `flyctl auth token` or `flyctl tokens create readonly -o `. |
| `PARSEABLE_URL` | yes | - | Base URL of the Parseable ingestor. |
| `PARSEABLE_USERNAME` | yes | - | HTTP basic auth username. |
| `PARSEABLE_PASSWORD` | yes | - | HTTP basic auth password. |
| `PARSEABLE_STREAM` | no | `fly_metrics` | Target stream name. Parseable creates the stream automatically on first write. |
| `SCRAPE_INTERVAL_SECS` | no | `30` | Seconds between scrapes. |
***
## Verify
Query the Parseable query node:
```bash
curl -u : -X POST \
-H 'Content-Type: application/json' \
-d '{"query":"select metric_name, app, region, data_point_value from fly_metrics order by p_timestamp desc limit 5","startTime":"1h","endTime":"now"}' \
https:///api/v1/query
```
Sample row from the `fly_metrics` stream:
```json
{
"app": "hellofly",
"region": "sin",
"instance": "9080d161a52928",
"metric_name": "fly_instance_memory_buffers",
"metric_type": "gauge",
"data_point_value": 778240,
"time_unix_nano": "2026-05-18T06:45:38.637",
"scope_name": "prometheus_remote_write",
"p_format": "otel-metrics"
}
```
For logs, query the `fly_logs` stream the same way.
***
## Notes
* **Auth scheme.** Fly's Prometheus endpoint requires `Authorization: FlyV1 ` for tokens issued by recent `flyctl` versions. The agent handles this automatically; the `Bearer` scheme returns HTTP 401.
* **Idle apps.** Apps with `auto_stop_machines = "stop"` stop emitting metrics until they are woken. Fly's `/federate` returns HTTP 200 with an empty body in that state.
* **Self-observability blind spot.** Both agents (logs and metrics) are themselves Fly apps in the same org, so they appear in their own data. If an agent crashes, its own outage is invisible from Parseable; set a stream-silence alert as a watchdog.
# Google Cloud Pub/Sub (/ingest-data/cloud/gcp-pubsub)
Stream logs from Google Cloud Pub/Sub to Parseable using the OpenTelemetry Collector.
## Overview
Integrate Google Cloud Pub/Sub with Parseable to:
* **Stream GCP Logs** - Collect logs from Cloud Logging via Pub/Sub
* **Real-time Ingestion** - Process logs as they arrive
* **Unified Observability** - Combine GCP logs with other sources
* **Scalable Collection** - Handle high-volume log streams
## Prerequisites
* Google Cloud project with Pub/Sub enabled
* Pub/Sub subscription for log messages
* Service account with Pub/Sub permissions
* OpenTelemetry Collector with `googlecloudpubsub` receiver
* Parseable instance running and accessible
### IAM Permissions
Create a service account with the required permissions:
```bash
# Create service account
gcloud iam service-accounts create otel-collector \
--display-name="OpenTelemetry Collector"
# Grant Pub/Sub Subscriber role
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
--member="serviceAccount:otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/pubsub.subscriber"
# Create and download key
gcloud iam service-accounts keys create key.json \
--iam-account=otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com
```
## Setting Up Cloud Logging Export
### Create a Pub/Sub Topic
```bash
gcloud pubsub topics create parseable-logs
```
### Create a Subscription
```bash
gcloud pubsub subscriptions create parseable-logs-sub \
--topic=parseable-logs \
--ack-deadline=60
```
### Create a Log Sink
Export Cloud Logging logs to Pub/Sub:
```bash
gcloud logging sinks create parseable-sink \
pubsub.googleapis.com/projects/YOUR_PROJECT_ID/topics/parseable-logs \
--log-filter='resource.type="gce_instance" OR resource.type="cloud_function"'
```
### Grant Sink Permissions
```bash
# Get the sink's writer identity
SINK_IDENTITY=$(gcloud logging sinks describe parseable-sink --format='value(writerIdentity)')
# Grant publish permission
gcloud pubsub topics add-iam-policy-binding parseable-logs \
--member="$SINK_IDENTITY" \
--role="roles/pubsub.publisher"
```
## OpenTelemetry Collector Configuration
### Basic Configuration
Create an `otel-collector-config.yaml` file:
```yaml
receivers:
googlecloudpubsub:
project: your-gcp-project-id
subscription: projects/your-gcp-project-id/subscriptions/parseable-logs-sub
encoding: cloud_logging
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "gcp-logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [googlecloudpubsub]
exporters: [otlphttp/parseable]
```
### With Processing
Add processors to transform logs:
```yaml
receivers:
googlecloudpubsub:
project: your-gcp-project-id
subscription: projects/your-gcp-project-id/subscriptions/parseable-logs-sub
encoding: cloud_logging
processors:
batch:
timeout: 10s
send_batch_size: 1000
attributes:
actions:
- key: cloud.provider
value: gcp
action: insert
exporters:
otlphttp/parseable:
endpoint: "http://parseable:8000"
headers:
Authorization: "Bearer "
X-P-Stream: "gcp-logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [googlecloudpubsub]
processors: [batch, attributes]
exporters: [otlphttp/parseable]
```
## Configuration Options
| Parameter | Description |
| -------------- | ---------------------------------------------------------- |
| `project` | GCP project ID |
| `subscription` | Full subscription path |
| `encoding` | Message encoding (`cloud_logging`, `raw_text`, `raw_json`) |
## Running the Collector
### Docker with Service Account Key
```bash
docker run -d \
--name otel-collector \
-v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
-v $(pwd)/key.json:/etc/gcp/key.json \
-e GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/key.json \
otel/opentelemetry-collector-contrib:latest
```
### Kubernetes with Workload Identity
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: otel-collector
annotations:
iam.gke.io/gcp-service-account: otel-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
spec:
template:
spec:
serviceAccountName: otel-collector
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:latest
volumeMounts:
- name: config
mountPath: /etc/otelcol/config.yaml
subPath: config.yaml
volumes:
- name: config
configMap:
name: otel-collector-config
```
## Querying GCP Logs in Parseable
Once data is flowing, query your GCP logs:
```sql
-- Get recent GCP logs
SELECT p_timestamp, resource_type, severity, text_payload
FROM "gcp-logs"
ORDER BY p_timestamp DESC
LIMIT 100
-- Find Cloud Function errors
SELECT p_timestamp, resource_labels, text_payload
FROM "gcp-logs"
WHERE resource_type = 'cloud_function'
AND severity IN ('ERROR', 'CRITICAL')
ORDER BY p_timestamp DESC
-- Count logs by resource type
SELECT resource_type, COUNT(*) as log_count
FROM "gcp-logs"
WHERE p_timestamp > NOW() - INTERVAL '1 hour'
GROUP BY resource_type
ORDER BY log_count DESC
```
## Troubleshooting
### Authentication Issues
1. Verify service account key is valid
2. Check IAM permissions include Pub/Sub Subscriber role
3. Verify GOOGLE\_APPLICATION\_CREDENTIALS is set
4. Check Workload Identity binding if using GKE
### Missing Messages
1. Verify the subscription exists and is active
2. Check the log sink filter matches your logs
3. Verify the sink has publish permission on the topic
4. Check for message acknowledgment issues
### High Latency
1. Increase batch size for higher throughput
2. Reduce ack deadline if messages are timing out
3. Scale the collector horizontally
## Next Steps
* Set up [alerts](/user-guide/alerting) for GCP log patterns
* Create [dashboards](/user-guide/dashboards) for GCP monitoring
* Configure [AWS CloudWatch](/ingest-data/cloud/aws-cloudwatch) for multi-cloud
# Amazon ECS (/ingest-data/containers/amazon-ecs)
Collect and forward logs from Amazon ECS (Elastic Container Service) to Parseable.
## Overview
Integrate Amazon ECS with Parseable to:
* **Container Logs** - Collect logs from ECS tasks
* **Centralized Observability** - Unified view of container logs
* **AWS Integration** - Native AWS log routing
* **Fargate Support** - Works with both EC2 and Fargate
## Prerequisites
* AWS account with ECS
* ECS cluster running
* Parseable instance accessible from AWS
* IAM permissions for log routing
## Method 1: FireLens (Fluent Bit)
Use AWS FireLens with Fluent Bit sidecar for log routing.
### Task Definition
```json
{
"family": "my-app",
"containerDefinitions": [
{
"name": "log-router",
"image": "amazon/aws-for-fluent-bit:latest",
"essential": true,
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/fluent-bit/configs/parse-json.conf"
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/firelens",
"awslogs-region": "us-east-1",
"awslogs-dataset-prefix": "firelens"
}
},
"memoryReservation": 50
},
{
"name": "app",
"image": "my-app:latest",
"essential": true,
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "http",
"Host": "parseable.example.com",
"Port": "8000",
"URI": "/api/v1/ingest",
"Format": "json",
"Header": "Authorization Basic YWRtaW46YWRtaW4=",
"Header": "X-P-Stream ecs-logs"
}
}
}
]
}
```
### Custom Fluent Bit Config
Store in S3 or include in custom image:
```ini
[OUTPUT]
Name http
Match *
Host parseable.example.com
Port 8000
URI /api/v1/ingest
Format json
Header Authorization Basic YWRtaW46YWRtaW4=
Header X-P-Stream ecs-logs
tls On
tls.verify Off
```
## Method 2: CloudWatch to Parseable
Route logs through CloudWatch, then forward to Parseable.
### Task Definition with awslogs
```json
{
"containerDefinitions": [
{
"name": "app",
"image": "my-app:latest",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app",
"awslogs-region": "us-east-1",
"awslogs-dataset-prefix": "app"
}
}
}
]
}
```
Then use [AWS CloudWatch integration](/ingest-data/cloud/aws-cloudwatch) to forward to Parseable.
## Method 3: Sidecar Container
Deploy a log collector sidecar alongside your application.
### Task Definition with Sidecar
```json
{
"family": "my-app-with-collector",
"containerDefinitions": [
{
"name": "app",
"image": "my-app:latest",
"essential": true,
"mountPoints": [
{
"sourceVolume": "logs",
"containerPath": "/var/log/app"
}
]
},
{
"name": "log-collector",
"image": "fluent/fluent-bit:latest",
"essential": false,
"mountPoints": [
{
"sourceVolume": "logs",
"containerPath": "/var/log/app",
"readOnly": true
}
],
"environment": [
{"name": "PARSEABLE_URL", "value": "http://parseable:8000"},
{"name": "PARSEABLE_AUTH", "value": "YWRtaW46YWRtaW4="}
]
}
],
"volumes": [
{
"name": "logs",
"host": {}
}
]
}
```
## IAM Permissions
Required IAM permissions for FireLens:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::your-config-bucket/*"
}
]
}
```
## Best Practices
1. **Use FireLens** - AWS-native log routing
2. **Add Metadata** - Include task ID, cluster name
3. **Configure Retries** - Handle transient failures
4. **Monitor Log Router** - Watch sidecar health
5. **Use Secrets** - Store credentials in Secrets Manager
## Troubleshooting
### Logs Not Appearing
1. Check FireLens container logs
2. Verify Parseable endpoint is accessible
3. Check IAM permissions
4. Verify log configuration syntax
### High Memory Usage
1. Configure buffer limits
2. Reduce batch sizes
3. Check for log volume spikes
## Next Steps
* Configure [Amazon EKS](/ingest-data/containers/amazon-eks) for Kubernetes
* Set up [alerts](/user-guide/alerting) for container metrics
* Create [dashboards](/user-guide/dashboards) for monitoring
# Amazon EKS (/ingest-data/containers/amazon-eks)
Collect and forward logs from Amazon EKS (Elastic Kubernetes Service) to Parseable.
## Overview
Integrate Amazon EKS with Parseable to:
* **Kubernetes Logs** - Collect pod and container logs
* **AWS Integration** - Native AWS ecosystem support
* **Scalable Collection** - Handle large cluster deployments
* **Rich Metadata** - Include Kubernetes context
## Prerequisites
* Amazon EKS cluster
* kubectl configured
* Helm (recommended)
* Parseable instance accessible from EKS
## Method 1: Fluent Bit DaemonSet
Deploy Fluent Bit as a DaemonSet for log collection.
### Install with Helm
```bash
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm install fluent-bit fluent/fluent-bit \
--namespace logging \
--create-namespace \
--set config.outputs="[OUTPUT]\n Name http\n Match *\n Host parseable.example.com\n Port 8000\n URI /api/v1/ingest\n Format json\n Header Authorization Basic YWRtaW46YWRtaW4=\n Header X-P-Stream eks-logs"
```
### Custom Values
Create `fluent-bit-values.yaml`:
```yaml
config:
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
filters: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
outputs: |
[OUTPUT]
Name http
Match *
Host parseable.example.com
Port 8000
URI /api/v1/ingest
Format json
Header Authorization Basic YWRtaW46YWRtaW4=
Header X-P-Stream eks-logs
tls On
tls.verify Off
tolerations:
- operator: Exists
effect: NoSchedule
```
Install with custom values:
```bash
helm install fluent-bit fluent/fluent-bit \
--namespace logging \
--create-namespace \
-f fluent-bit-values.yaml
```
## Method 2: AWS for Fluent Bit
Use AWS's optimized Fluent Bit distribution.
### DaemonSet Manifest
```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
serviceAccountName: fluent-bit
containers:
- name: fluent-bit
image: amazon/aws-for-fluent-bit:latest
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config
mountPath: /fluent-bit/etc/
env:
- name: PARSEABLE_HOST
value: "parseable.example.com"
- name: PARSEABLE_AUTH
valueFrom:
secretKeyRef:
name: parseable-credentials
key: auth
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: config
configMap:
name: fluent-bit-config
tolerations:
- operator: Exists
effect: NoSchedule
```
### ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser cri
Tag kube.*
Refresh_Interval 5
Mem_Buf_Limit 50MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Merge_Log On
K8S-Logging.Parser On
[OUTPUT]
Name http
Match *
Host ${PARSEABLE_HOST}
Port 8000
URI /api/v1/ingest
Format json
Header Authorization Basic ${PARSEABLE_AUTH}
Header X-P-Stream eks-logs
tls On
parsers.conf: |
[PARSER]
Name cri
Format regex
Regex ^(?