Skip to main content

Fluentd

Overview​

Fluentd is a data collector for your applications and services. Fluentd can be configured to send logs to Parseable with HTTP output plugin and JSON output format.

This document explains how to set up Fluentd to ship logs to Parseable Docker Compose and Kubernetes. This should give you an idea on how to configure the output plugin for other scenarios.

For demo purpose, we used Fluentd's Memory Metrics Input plugin as the source of logs.

Docker Compose​

Please ensure Docker Compose installed on your machine. Then run the following commands to set up Parseable and Fluentd.

mkdir parseable
cd parseable
wget https://www.parseable.com/fluentd/fluentd.conf
wget https://www.parseable.com/fluentd/docker-compose.yaml
docker-compose up -d

You can now access the Parseable dashboard on http://localhost:8000. You should see a log stream called fluentbitdemo populated with log data generated by the Memory Metrics Input plugin.

Kubernetes​

  • Please ensure kubectl and helm installed and configured to access your Kubernetes cluster.
  • Parseable installed on your Kubernetes cluster. Refer the Parseable Kubernetes documentation.

Install Fluentd​

We use the official Fluentd Helm chart. But, we'll use a modified values.yaml file, that contains the configuration for Fluentd to send logs to Parseable.

wget https://www.parseable.com/fluentd/values.yaml
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluentd fluent/fluentd --values values.yaml -n f

Fluentd Configuration​

Let's take a deeper look at the Fluentd configuration in values.yaml. Here we use the kubernetes filter to enrich the logs with Kubernetes metadata. We then use the http output plugin to send logs to Parseable. Notice the Match section in the http output plugin. We use kube.* to match all logs from Kubernetes filter. With the header X-P-Stream fluentbitdemo, we tell Parseable to send the logs to the fluentbitdemo stream.

# Fluentd configuration
config:
  system: |
    <system>
      log_level debug
    </system>

  ## Input configuration
  inputs: |
    <source>
      @type monitor_agent
      tag memory
      emit_interval 5
    </source>

  ## Output configuration
  outputs: |
    <match *>
      @type http
      host parseable.parseable.svc.cluster.local    
      uri /api/v1/ingest
      port 80
      
      <headers>
        X-P-META-meta1 value1
        X-P-TAG-tag1 value1
        X-P-Stream fluentbitdemo
      </headers>
      
      <format>
        @type json
        time_key timestamp
        time_format iso8601
      </format>
      
      <buffer>
        @type memory
        flush_interval 5s
      </buffer>
      
      <auth>
        method basic
        username admin
        password admin
      </auth>
    </match>

Check logs in Parseable​

Port forward Parseable service to access the dashboard with:

kubectl port-forward svc/parseable 8000:80 -n parseable

You can now check the Parseable server fluentbitdemo stream to see the logs from this setup.

Navigate to the Parseable dashboard and verify that logs are being ingested correctly.