Kubernetes is a powerful tool for deploying and managing containers, but with many containers running in a cluster, it can be difficult to keep track of logs and troubleshoot issues. Fluent Bit is a log shipper that can be used to collect and aggregate logs from your containers and make them available for analysis and troubleshooting.

In this tutorial, we’ll demonstrate how to set up Fluent Bit in a Kubernetes cluster and use it to collect and aggregate logs from your containers.

Step 1: Deploy Fluent Bit to your Kubernetes cluster

The first step is to deploy Fluent Bit to your Kubernetes cluster. You can do this using a Fluent Bit DaemonSet, which will ensure that Fluent Bit is running on every node in your cluster. You can deploy Fluent Bit using the following YAML file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.11.1
        resources:
          limits:
            memory: 200Mi
          requests:
            memory: 100Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config

In this YAML file, we’re creating a DaemonSet that deploys Fluent Bit to every node in the cluster. We’re also mounting the host’s /var/log directory to the Fluent Bit container so that it can collect logs from all containers running on the host.

Step 2: Create a Fluent Bit configuration

Next, you’ll need to create a Fluent Bit configuration that tells Fluent Bit how to collect and process logs. You can do this using a ConfigMap in Kubernetes. You can create a ConfigMap using the following YAML file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    info

    [INPUT]
        Name tail
        Path /var/log/**.log
        Tag *
        DB /fluent-bit/data/db.dat
        Parser json
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [OUTPUT]
        Name elasticsearch
        Match *
        Host elasticsearch-logging
        Port 9200
        Logstash_Format On
        Retry_Limit False
        Type elasticsearch

In this configuration, we’re using the tail input plugin to collect logs from the host’s /var/log directory. We’re also using

IAM Permissions

Here is an example of an AWS Identity and Access Management (IAM) policy that allows EC2 instances to write logs to Amazon CloudWatch Logs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEC2ToPutLogEvents",
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/ec2/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalArn": [
                        "arn:aws:iam::*:role/ec2-instance-role"
                    ]
                }
            }
        },
        {
            "Sid": "AllowEC2ToCreateLogGroup",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalArn": [
                        "arn:aws:iam::*:role/ec2-instance-role"
                    ]
                }
            }
        }
    ]
}

In this policy, we have two statements:

The first statement allows EC2 instances to write logs to CloudWatch Logs using the logs:PutLogEvents action. The resource is specified as the log groups within the /aws/ec2 namespace, and the condition restricts the principal to the EC2 instance role.

The second statement allows EC2 instances to create new log groups in CloudWatch Logs using the logs:CreateLogGroup action. The resource is specified as all log groups in the account, and the condition restricts the principal to the EC2 instance role.

You can attach this policy to an IAM role that is associated with your EC2 instances, allowing them to write logs to CloudWatch Logs.

Looking around kubernetes cluster configuration

In a Kubernetes cluster, the Fluent Bit configuration can be stored in a ConfigMap or a Secret. The ConfigMap or Secret is then mounted as a volume in the Fluent Bit pod and used as the configuration file.

To find the Fluent Bit configuration in a Kubernetes cluster, you can use the following steps:

Find the Fluent Bit deployment or daemonset in your cluster using the kubectl get command:

kubectl get deployments,daemonsets -n NAMESPACE -o json | grep fluent-bit

Replace NAMESPACE with the name of the namespace where the Fluent Bit deployment or daemonset is running.

Get the details of the Fluent Bit deployment or daemonset using the kubectl describe command:

kubectl describe deployment/DAEMONSET_NAME -n NAMESPACE

Replace DAEMONSET_NAME with the name of the Fluent Bit deployment or daemonset you found in step 1.

Look for the ConfigMap or Secret that is being mounted as a volume in the Fluent Bit pod. In the output of the kubectl describe command, look for a section that starts with Volumes: and find the volume that is being used as the Fluent Bit configuration.

Get the contents of the ConfigMap or Secret using the kubectl get command:

kubectl get configmap/SECRET_NAME -n NAMESPACE -o json

Replace SECRET_NAME with the name of the ConfigMap or Secret you found in step 3.

This should give you the Fluent Bit configuration that is being used in the Fluent Bit pod in your Kubernetes cluster.