Deploying CrowdSec to ban them all.

Introduction 🔗

Usually, I don't write step-by-step guide on how to deploy things, but in this case, I will. In these times, it's almost impossible to open an HTTP port without getting spammed and attacked by botnets and other malicious actors.

One solution is the close the port and forward the traffic to a better, managed WAF (Web Application Firewall) by using Cloudflare Tunnel, delegating the security to Cloudflare.

However, adopting such a solution introduces a significant dependency on a centralized "lord" of the internet, compromising full infrastructure independence.

While absolute independence is debatable, given the necessity of root DNS for domain registration and Certificate Authorities for SSL, relying on a platform like Cloudflare presents a unique single point of failure. Unlike the decentralized nature of DNS and certificate management, Cloudflare's centralization means its outages directly translate to your downtime. In fact, a major outage at Cloudflare last week demonstrated exactly this vulnerability.

This article explores an alternative path: preserving your infrastructure's autonomy while effectively securing your network from attacks. We will demonstrate how to achieve robust protection using CrowdSec, maintaining control and independence without using centralized services.

Why would I publish a port on the home network ? 🔗

While the best practice is to use cloud providers to expose your service and avoid leaking your IP address, leveraging a home network connection offers distinct, often overlooked, advantages for certain projects:

• You're likely to get less attacked by bots, since ISP assigned IP addresses are typically less targeted by bots compared to IPs assigned by cloud providers.
• Your home IP address is less likely to be pre-emptively blocked or flagged due to the unrelated, malicious activities often hosted on shared cloud infrastructure.
• For users with high-quality fiber connections, hosting directly can offer surprisingly low latency, potentially rivaling or even surpassing the performance of services routed through distant cloud nodes or additional tunneling layers.
• Using a robust reverse proxy directly behind the public HTTP port ensures that most common web attacks are filtered before they ever reach your backend applications.

The major drawback is the exposure of your physical location, as your ISP-assigned IP address can be linked to your geographic data. Therefore, this approach is not suitable for professional or high-stakes services where anonymity and guaranteed uptime are critical.

However, for personal projects, small community sites, or low-stakes experiments, hosting on a home network offers a cost-effective, high-performing alternative that retains independence from large-scale cloud ecosystems.

How to protect against attacks and spammers ? 🔗

A traditional solution against malicious traffic and brute-force attacks involves analyzing system logs to identify and block offenders. Tools like Fail2ban have long been the standard for this, parsing logs for failed login attempts and temporarily banning the source IP address.

However, Fail2ban suffers from a critical limitation: the reaction time. Since bans are applied only after the logs are written, read, and processed, there is an inherent delay. The malicious actor has already made several attempts before the block takes effect.

CrowdSec solves this issue in multiple manners:

• CrowdSec uses and builds a crowdsourced blocklist to block known attackers, allowing you to proactively block threats before they ever interact with your systems.
• Like Fail2ban, CrowdSec parses logs in real time, but it simplifies deployment by offering an extensive library of community-validated configurations (or "scenarios") designed to recognize and handle common attack patterns.
• Beyond simple spam or brute-force attempts, CrowdSec uses its log analysis to identify complex attack scenarios. This includes recognizing the initial stages of large-scale threats like Distributed Denial of Service (DDoS) attempts or application-specific attacks such as SQL injection.
• For defense against attacks aimed at your web applications, CrowdSec offers a WAF component. This intercepts and inspects HTTP requests before they reach your reverse proxy or backend services, providing a critical layer of preventative security.

How to deploy CrowdSec 🔗

This article is about deploying a distributed setup of CrowdSec on Kubernetes and protect against Web Application attacks. It can also handle SSH and many other types of attacks, but this won't be the focus of this article.

Architecture 🔗

Here are the components that play a role in this setup:

• The persistence: Which is either SQLite on disk or a PostgreSQL instance (or any other database supported by CrowdSec). This is used to persist the state of CrowdSec Local API.
• The CrowdSec Local API:
  • Stores Alerts and Decisions to the database.
  • Is used to manage decisions, alerts, blocklists, and more.
• The CrowdSec Agent (log processor): It's a daemon used to read logs, whether it's from files or from a log storage like Loki, Elasticsearch, etc.
• The CrowdSec AppSec: It's a daemon that looks like the CrowdSec Agent, with the difference of having an open port for the reverse proxy to send traffic to it and inspect it.
• The reverse proxy (remediation component): Which will forward traffic to the AppSec component of CrowdSec, which will then inspect the traffic and block any malicious traffic. The reverse proxy logs will also be used to identify additional patterns of attacks.
• The CrowdSec Central API: This is used for metrics on crowdsec.net and contributing to the CrowdSec Community blocklist.

CrowdSec pricing plan 🔗

The CrowdSec Central API is actually not free and has multiple setup available:

• Free and no crowdsourcing: which is not worth it, and you should prefer fail2ban instead.
• Free and crowdsourcing: which allows you to contribute to the CrowdSec community blocklist. CrowdSec also offers you 3 free blocklists maintained by third parties.
• Paid with or without crowdsourcing: which can be interesting if you want to use the CrowdSec community blocklist, but want to avoid leaking data to a third party like CrowdSec. The paid plan also removes the metrics limitation. There is many advantages in paying CrowdSec in which I won't cover here.

This article will cover an alternative solution to CrowdSec metrics limitation, so don't jump too soon on the Paid plan.

Deploying CrowdSec on Kubernetes 🔗

1. Register on CrowdSec.net and fetch the registration token 🔗

Go to app.crowdsec.net and register an account. Then, fetch the registration token the Security Engines page.

2. Deploy an Ingress Controller on Kubernetes 🔗

This article will cover the Traefik ingress controller, but any other ingress controller will do as well. If you plan to use the NGINX Ingress Controller, you will need to replace it with the fork maintained by CrowdSec: crowdsec/ingress-nginx which re-adds Lua support.

It's almost confirmed that the NGINX Ingress Controller will be deprecated for InGate, so the fact they dropped support for Lua practically means that you should either migrate to a better ingress controller or use CrowdSec's fork.

Anyway, you should deploy an Ingress Controller on your Kubernetes cluster if this isn't done yet. The Ingress Controller is the main reverse proxy and should be exposed to the external network through MetalLB, or ServiceLB. You can also use a NodePort service, and do the port forwarding manually.

Note that if you've deployed Traefik, you'll need to send the logs to a volume and print it to the standard output so that Kubernetes can catch it.

For example, if you've used Helm to deploy Traefik:

 1deployment:
 2  enabled: true
 3
 4  additionalVolumes:
 5    - name: traefik-logs
 6      emptyDir: {}
 7
 8  additionalContainers:
 9    - name: tail-accesslogs
10      image: busybox
11      imagePullPolicy: IfNotPresent
12      args: ['/bin/sh', '-c', 'tail -n+1 -F /var/log/traefik/access.log']
13      volumeMounts:
14        - name: traefik-logs
15          mountPath: /var/log/traefik
16
17additionalVolumeMounts:
18  - name: traefik-logs
19    mountPath: /var/log/traefik
20
21logs:
22  general:
23    level: INFO
24  access:
25    enabled: true
26    format: json
27    addInternals: false
28    filePath: /var/log/traefik/access.log
29    bufferingSize: 100
30    fields:
31      general:
32        defaultmode: drop
33        names:
34          ClientHost: keep
35          ClientAddr: keep
36          ClientUsername: keep
37          StartUTC: keep
38          RequestHost: keep
39          RequestAddr: keep
40          RequestMethod: keep
41          RequestPath: keep
42          RequestProtocol: keep
43          DownstreamStatus: keep
44          DownstreamContentSize: keep
45          RequestCount: keep
46          ServiceAddr: keep
47          RouterName: keep
48          ServiceName: keep
49          ServiceURL: keep
50          Duration: keep
51      headers:
52        defaultmode: drop
53        names:
54          Authorization: redact
55          Referer: keep
56          User-Agent: keep
57          Origin: keep
58          Content-Type: keep
59          Range: keep
60          Cookie: redact
61          Set-Cookie: redact
62          Content-Security-Policy: drop
63          Permissions-Policy: drop

3. Deploy CrowdSec on Kubernetes 🔗

While CrowdSec offers a Helm Chart, these are badly maintained, and their quality is not great (many variables that does nothing, many bugs, etc.).

This is why, we'll deploy and configure the deployment manually with Kustomize. Since deployments made by Kustomize can be very opinionated, feel free to adjust the deployment as you see fit.

For now, we'll consider the following:

• The Base will store generic and common configurations for all environments.
• The Overlays will store environment-specific configurations, like secrets, and PVCs.

Base - Kustomization 🔗

Let's start with the base:

 1labels:
 2  - pairs:
 3      app.kubernetes.io/name: crowdsec
 4      app.kubernetes.io/part-of: crowdsec
 5      app.kubernetes.io/managed-by: Kustomize
 6    includeSelectors: true
 7    includeTemplates: true
 8
 9resources:
10  - agent/daemonset.yaml
11  - agent/service.yaml # Optional: is only used if you want to expose the metrics.
12  - agent/certificate.yaml
13  - appsec/deployment.yaml
14  - appsec/service.yaml
15  - appsec/certificate.yaml
16  - lapi/deployment.yaml
17  - lapi/service.yaml
18  - lapi/certificate.yaml
19
20namespace: crowdsec
21
22configMapGenerator:
23  - name: crowdsec-custom-start
24    options:
25      labels:
26        app.kubernetes.io/component: lapi
27    files:
28      - docker_start.sh=lapi/files/docker_start.sh
29  - name: crowdsec-lapi-config
30    options:
31      labels:
32        app.kubernetes.io/component: lapi
33    files:
34      - profiles.yaml=lapi/files/profiles.yaml
35  - name: crowdsec-lapi-notifications
36    options:
37      labels:
38        app.kubernetes.io/component: lapi
39    files:
40      - notifications.yaml=lapi/files/notifications.yaml
41  - name: crowdsec-agent-config
42    options:
43      labels:
44        app.kubernetes.io/component: agent
45    files:
46      - acquis.yaml=agent/files/acquis.yaml
47  - name: crowdsec-appsec-config
48    options:
49      labels:
50        app.kubernetes.io/component: appsec
51    files:
52      - acquis.yaml=appsec/files/acquis.yaml
53  - name: crowdsec-appsec-haproxy-config
54    options:
55      labels:
56        app.kubernetes.io/component: appsec
57    files:
58      - haproxy.cfg=appsec/files/haproxy.cfg

We'll create each of these resources.

Base - Local API 🔗

Let's configure the deployment of the CrowdSec Local API, which is responsible for storing alerts, decisions and managing communication with the central API.

1. Write the docker_start.sh script. You'll need to fetch it in the Helm Charts GitHub repository:

   base/lapi/files/docker_start.sh

   1#!/bin/bash
   2
   3# shellcheck disable=SC2292      # allow [ test ] syntax
   4# shellcheck disable=SC2310      # allow "if function..." syntax with -e
   5
   6set -e
   7shopt -s inherit_errexit
   8
   9...
   

2. Write the profiles.yaml file, which configure the behavior of CrowdSec when receiving an alert. Here's an example:

   base/lapi/files/profiles.yaml

    1name: default_ip_remediation
    2filters:
    3  - Alert.Remediation == true && Alert.GetScope() == "Ip"
    4decisions:
    5  - type: ban
    6    duration: 72h
    7duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 72)
    8notifications:
    9  #- slack_default
   10  - file_default
   11on_success: break
   12---
   13name: default_range_remediation
   14filters:
   15  - Alert.Remediation == true && Alert.GetScope() == "Range"
   16decisions:
   17  - type: ban
   18    duration: 72h
   19duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 72)
   20notifications:
   21  #- slack_default
   22  - file_default
   23on_success: break
   24---
   25name: vpatch_remediation
   26filters:
   27  - Alert.GetScenario() contains "vpatch"
   28decisions:
   29  - type: ban
   30    duration: 72h
   31duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 72)
   32notifications:
   33  #- slack_default
   34  - file_default
   35on_success: break
   

3. Write the notifications.yaml file, which configure the notification channels. Usually, this is stored in a Secret, but since we'll be simply outputting it to a file, we'll use a ConfigMap as declared in the kustomization.yaml:

   base/lapi/files/notifications.yaml

    1type: file
    2name: file_default
    3log_level: info
    4format: |
    5  {{range . -}}
    6  { "time": "{{.StopAt}}", "program": "crowdsec", "alert": {{. | toJson }} }
    7  {{ end -}}
    8log_path: '/crowdsec-logs/crowdsec_alerts.json'
    9rotate:
   10  enabled: true # Change to false if you want to handle log rotate on system basis
   11  max_size: 10 # in MB
   12  max_files: 5 # Number of files to keep
   13  max_age: 5 # in days but may remove files before this if max_files is reached
   14  compress: true # Compress rotated files using gzip
   

4. Write the deployment.yaml:

   base/lapi/deployment.yaml

     1apiVersion: apps/v1
     2kind: Deployment
     3metadata:
     4  name: crowdsec-lapi
     5  labels:
     6    app.kubernetes.io/component: lapi
     7spec:
     8  replicas: 1
     9  selector:
    10    matchLabels:
    11      app.kubernetes.io/component: lapi
    12  strategy:
    13    type: Recreate # Only if using a SQLite database
    14  template:
    15    metadata:
    16      labels:
    17        app.kubernetes.io/component: lapi
    18    spec:
    19      automountServiceAccountToken: false
    20      terminationGracePeriodSeconds: 30
    21      containers:
    22        - name: crowdsec-lapi
    23          image: 'crowdsecurity/crowdsec:v1.7.3'
    24          imagePullPolicy: IfNotPresent
    25          command:
    26            - sh
    27            - '-c'
    28            - >-
    29              cp -nR /staging/etc/crowdsec/* /etc/crowdsec_data/ && bash /docker_start.sh
    30          env:
    31            - name: LOCAL_API_URL
    32              value: https://localhost:8080
    33            - name: DISABLE_AGENT
    34              value: 'true'
    35            - name: USE_TLS
    36              value: 'true'
    37            - name: LAPI_CERT_FILE
    38              value: /certs/server/tls.crt
    39            - name: LAPI_KEY_FILE
    40              value: /certs/server/tls.key
    41            - name: CACERT_FILE
    42              value: /certs/server/ca.crt
    43            - name: CLIENT_CERT_FILE
    44              value: /certs/client/tls.crt
    45            - name: CLIENT_KEY_FILE
    46              value: /certs/client/tls.key
    47            - name: INSECURE_SKIP_VERIFY
    48              value: 'false'
    49            - name: CUSTOM_HOSTNAME
    50              valueFrom:
    51                fieldRef:
    52                  fieldPath: metadata.name
    53            # CrowdSec uses Organisation Units (OUs) to validate certificates.
    54            - name: AGENTS_ALLOWED_OU
    55              value: agent-ou,appsec-ou
    56          resources:
    57            limits:
    58              memory: 293Mi
    59            requests:
    60              cpu: 10m
    61              memory: 293Mi
    62          livenessProbe:
    63            failureThreshold: 3
    64            periodSeconds: 10
    65            successThreshold: 1
    66            timeoutSeconds: 5
    67            httpGet:
    68              path: /health
    69              port: lapi
    70              scheme: HTTPS
    71          readinessProbe:
    72            failureThreshold: 3
    73            periodSeconds: 10
    74            successThreshold: 1
    75            timeoutSeconds: 5
    76            httpGet:
    77              path: /health
    78              port: lapi
    79              scheme: HTTPS
    80          startupProbe:
    81            failureThreshold: 30
    82            periodSeconds: 10
    83            successThreshold: 1
    84            timeoutSeconds: 5
    85            httpGet:
    86              path: /health
    87              port: lapi
    88              scheme: HTTPS
    89          securityContext:
    90            allowPrivilegeEscalation: false
    91            privileged: false
    92            readOnlyRootFilesystem: true
    93          ports:
    94            - name: lapi
    95              containerPort: 8080
    96              protocol: TCP
    97            - name: metrics
    98              containerPort: 6060
    99              protocol: TCP
   100          volumeMounts:
   101            - name: tls
   102              mountPath: /certs/server
   103              readOnly: true
   104            - name: client-tls
   105              mountPath: /certs/client
   106              readOnly: true
   107            - mountPath: /etc/crowdsec_data
   108              name: persistence # Only if using a SQLite database
   109              subPath: crowdsec-etc
   110            - name: config
   111              mountPath: /etc/crowdsec/profiles.yaml
   112              subPath: profiles.yaml
   113              readOnly: true
   114            - mountPath: /etc/crowdsec/config.yaml.local
   115              name: config
   116              subPath: config.yaml.local
   117              readOnly: true
   118            - mountPath: /var/lib/crowdsec/data
   119              name: persistence # Only if using a SQLite database
   120              subPath: crowdsec-data
   121            - name: persistence # Only if using a SQLite database
   122              mountPath: /etc/crowdsec
   123              subPath: crowdsec-etc
   124            - name: tmp
   125              mountPath: /tmp
   126              subPath: tmp
   127            - name: custom-start
   128              mountPath: /docker_start.sh
   129              subPath: docker_start.sh
   130            - mountPath: /etc/crowdsec/notifications
   131              name: notifications
   132        - name: tail-crowdsec-logs # We need this sidecar container to forward Crowdsec Alerts to Kubernetes logs.
   133          image: busybox
   134          imagePullPolicy: IfNotPresent
   135          command:
   136            ['tail', '-n+1', '-F', '/crowdsec-logs/crowdsec_alerts.json']
   137          volumeMounts:
   138            - mountPath: /crowdsec-logs
   139              name: crowdsec-logs
   140      volumes:
   141        - name: persistence # Only if using a SQLite database
   142        - name: notifications
   143          configMap:
   144            name: crowdsec-lapi-notifications
   145        - name: config
   146          configMap:
   147            name: crowdsec-lapi-config
   148        - name: tls
   149          secret:
   150            secretName: crowdsec-lapi-tls
   151        - name: client-tls
   152          secret:
   153            secretName: crowdsec-appsec-tls
   154        - name: tmp
   155          emptyDir: {}
   156        - name: custom-start
   157          configMap:
   158            name: crowdsec-custom-start
   159        - name: crowdsec-logs
   160          emptyDir: {}
   

   If you look closely, you'll notice that persistence is missing a configuration. This will be patched in a Kustomize overlay later.

   However, do note that the persistence volume might not be required if you plan to use PostgreSQL as database. You'll still need to mount an emptyDir at the /etc/crowdsec and /var/lib/crowdsec/data locations.

   NOTE

   The containerized CrowdSec can use multiple environment variables to configure itself. This also applies to the Agent and Appsec components.

   • CONFIG_FILE: The path to the configuration file. Defaults to /etc/crowdsec/config.yaml.
   • NO_HUB_UPGRADE=true: Disables the automatic upgrade of the CrowdSec Hub.
   • DEBUG=true: Enables debug mode for the start script.
   • COLLECTIONS/DISABLE_COLLECTIONS: Comma-separated list of collections to enable/disable.
   • PARSERS/DISABLE_PARSERS: Comma-separated list of parsers to enable/disable.
   • SCENARIOS/DISABLE_SCENARIOS: Comma-separated list of scenarios to enable/disable.
   • POSTOVERFLOWS/DISABLE_POSTOVERFLOWS: Comma-separated list of postoverflows to enable/disable.
   • CONTEXTS/DISABLE_CONTEXTS: Comma-separated list of contexts to enable/disable.
   • APPSEC_CONFIGS/DISABLE_APPSEC_CONFIGS: Comma-separated list of AppSec configurations to enable/disable.
   • APPSEC_RULES/DISABLE_APPSEC_RULES: Comma-separated list of AppSec rules to enable/disable.
   • METRICS_PORT: The port to expose the metrics on.
   • DISABLE_LOCAL_API=true: Disables the Local API role.
   • DISABLE_AGENT=true: Disables the Agent role.

   Local API specific:

   • DISABLE_ONLINE_API=true: Do not connect to the Central API (crowdsec.net).
   • ENROLL_KEY: Enroll the Local API to the Central API.
   • USE_WAL=true: Set .db_config.use_wal to the set value. Only applies to SQLite databases.
   • LAPI_CERT_FILE: The path to the Local API certificate file. Defaults to /certs/server/tls.crt.
   • LAPI_KEY_FILE: The path to the Local API key file. Defaults to /certs/server/tls.key.
   • AGENTS_ALLOWED_OU: The allowed Organizational Units for certificates given by the Agent/AppSec components during TLS client authentication. Defaults to agent-ou.
   • BOUNCER_ALLOWED_OU: The allowed Organizational Units for certificates given by the Bouncer component during TLS client authentication. Defaults to bouncer-ou.
   • BOUNCER_KEY_<ID> (ID is case-insensitive): Shared secret between the Local API and the Bouncer component.
   • ENABLE_CONSOLE_MANAGEMENT: Use the Central API to manage the Local API (Premium feature).
   • CAPI_WHITELISTS_PATH: Path to the CAPI whitelists file. Avoid IPs and CIDRs getting registered in the community blocklist. Deprecated by Allowlists.
   • DSN: The database connection string.
   • TYPE: The database type.
   • TEST_MODE=true: Enable the test mode.
   • LEVEL_[TRACE|DEBUG|INFO|WARN|ERROR]=true: Set the logging level of a specific log level.

5. Write the service.yaml:

   base/lapi/service.yaml

    1apiVersion: v1
    2kind: Service
    3metadata:
    4  name: crowdsec-lapi
    5  labels:
    6    app.kubernetes.io/component: lapi
    7spec:
    8  type: ClusterIP
    9  ports:
   10    - port: 6060
   11      targetPort: metrics
   12      protocol: TCP
   13      name: metrics
   14    - port: 8080
   15      targetPort: lapi
   16      protocol: TCP
   17      name: lapi
   18  selector:
   19    app.kubernetes.io/component: lapi
   

6. Write the certificate.yaml. We'll cert-manager to generate the certificate for us.

   base/lapi/certificate.yaml

    1apiVersion: cert-manager.io/v1
    2kind: Certificate
    3metadata:
    4  name: crowdsec-lapi
    5spec:
    6  secretName: crowdsec-lapi-tls
    7  duration: 2160h
    8  renewBefore: 720h
    9  subject:
   10    organizationalUnits: [lapi-ou]
   11  commonName: crowdsec-lapi
   12  dnsNames:
   13    - crowdsec-lapi.crowdsec # Make sure to match the namespace
   14    - crowdsec-lapi.crowdsec.svc.cluster.local # Make sure to match the namespace
   15    - localhost
   16  issuerRef:
   17    name: private-cluster-issuer
   18    kind: ClusterIssuer
   

   private-cluster-issuer is a ClusterIssuer that generates private certificates for the CrowdSec LAPI. It has a self-signed root CA and a private key that looks like this:

    1# THIS IS PURELY AN EXAMPLE, DO NOT DEPLOY IT
    2apiVersion: cert-manager.io/v1
    3kind: ClusterIssuer
    4metadata:
    5  name: selfsigned-cluster-issuer
    6  namespace: cert-manager
    7spec:
    8  selfSigned: {}
    9---
   10apiVersion: cert-manager.io/v1
   11kind: Certificate
   12metadata:
   13  name: my-root-ca
   14  namespace: cert-manager
   15spec:
   16  isCA: true
   17  duration: 43800h # 5 year
   18  renewBefore: 720h # 30 days before expiry
   19  secretName: my-root-ca
   20  isCA: true
   21  privateKey:
   22    algorithm: RSA
   23    encoding: PKCS1
   24    size: 2048
   25  subject:
   26    organizations: [Me]
   27  commonName: My Root CA
   28  issuerRef:
   29    name: selfsigned-cluster-issuer
   30    kind: ClusterIssuer
   31---
   32apiVersion: cert-manager.io/v1
   33kind: ClusterIssuer
   34metadata:
   35  name: private-cluster-issuer
   36spec:
   37  ca:
   38    secretName: my-root-ca
   

That's it for the LAPI. We'll be applying environment specific patches later. If you plan to monitor Crowdsec with Prometheus, you can add a ServiceMonitor resource targeting the metrics port.

Base - Agent 🔗

We'll now configure the deployment the CrowdSec Agent. The CrowdSec agent is a log processor similar to Promtail and Filebeat. It should be deployed as a DaemonSet to fetch the container logs on each host of the Kubernetes cluster. Knowing that the Ingress Controller will output its logs to /var/log/containers, we can do the following:

1. First write the acquisition configuration at base/agent/files/acquis.yaml. (This can be moved to a Kustomize overlay if the configuration differs between environments.)

   base/agent/files/acquis.yaml

   1filenames:
   2  - /var/log/containers/ingress-traefik-*_<REPLACE_ME: namespace>_*access*.log
   3labels: # These labels will trigger parsers
   4  program: traefik
   5  type: containerd
   6poll_without_inotify: true
   7source: file
   

   If you use NGINX, replace traefik with nginx instead. Make sure the log filename matches the Ingress Controller container logs on the Kubernetes node. We'll be mounting Kubernetes logs at /var/log/containers in the container.

2. Write the daemonset.yaml:

   base/agent/daemonset.yaml

     1apiVersion: apps/v1
     2kind: DaemonSet
     3metadata:
     4  name: crowdsec-agent
     5  labels:
     6    app.kubernetes.io/component: agent
     7spec:
     8  selector:
     9    matchLabels:
    10      app.kubernetes.io/component: agent
    11  template:
    12    metadata:
    13      labels:
    14        app.kubernetes.io/component: agent
    15    spec:
    16      automountServiceAccountToken: false
    17      terminationGracePeriodSeconds: 30
    18      initContainers:
    19        - name: wait-for-lapi
    20          image: busybox
    21          imagePullPolicy: IfNotPresent
    22          command:
    23            [
    24              'sh',
    25              '-c',
    26              'until nc "$LAPI_HOST" "$LAPI_PORT" -z; do echo waiting for lapi to start; sleep 5; done',
    27            ]
    28          resources:
    29            limits:
    30              memory: 50Mi
    31            requests:
    32              cpu: 1m
    33              memory: 10Mi
    34          securityContext:
    35            allowPrivilegeEscalation: false
    36            privileged: false
    37            readOnlyRootFilesystem: true
    38            runAsUser: 1000
    39            runAsGroup: 2000
    40            runAsNonRoot: true
    41            capabilities:
    42              drop: [ALL]
    43          env:
    44            - name: MY_NAMESPACE
    45              valueFrom:
    46                fieldRef:
    47                  fieldPath: metadata.namespace
    48            - name: LAPI_HOST
    49              value: 'crowdsec-lapi.$(MY_NAMESPACE)'
    50            - name: LAPI_PORT
    51              value: '8080'
    52      containers:
    53        - name: crowdsec-agent
    54          image: 'crowdsecurity/crowdsec:v1.7.3'
    55          imagePullPolicy: IfNotPresent
    56          env:
    57            - name: MY_NAMESPACE
    58              valueFrom:
    59                fieldRef:
    60                  fieldPath: metadata.namespace
    61            - name: DISABLE_LOCAL_API
    62              value: 'true'
    63            - name: DISABLE_ONLINE_API
    64              value: 'true'
    65            - name: CROWDSEC_BYPASS_DB_VOLUME_CHECK
    66              value: 'true'
    67            - name: USE_TLS
    68              value: 'true'
    69            - name: CLIENT_CERT_FILE
    70              value: /certs/tls.crt
    71            - name: CLIENT_KEY_FILE
    72              value: /certs/tls.key
    73            - name: CACERT_FILE
    74              value: /certs/ca.crt
    75            - name: LOCAL_API_URL
    76              value: 'https://crowdsec-lapi.$(MY_NAMESPACE):8080'
    77            - name: COLLECTIONS # This is used to download a bunch of scenarios, parsers, enrichers, etc...
    78              value: crowdsecurity/traefik crowdsecurity/linux # This should cover basic HTTP scenarios and enrich logs with GeoIP data. This also covers SSH.
    79            - name: UNREGISTER_ON_EXIT
    80              value: 'true'
    81          resources:
    82            limits:
    83              memory: 189Mi
    84            requests:
    85              cpu: 10m
    86              memory: 189Mi
    87          ports:
    88            - name: metrics
    89              containerPort: 6060
    90              protocol: TCP
    91          livenessProbe:
    92            failureThreshold: 3
    93            httpGet:
    94              path: /metrics
    95              port: metrics
    96              scheme: HTTP
    97            periodSeconds: 10
    98            successThreshold: 1
    99            timeoutSeconds: 5
   100          readinessProbe:
   101            failureThreshold: 3
   102            httpGet:
   103              path: /metrics
   104              port: metrics
   105              scheme: HTTP
   106            periodSeconds: 10
   107            successThreshold: 1
   108            timeoutSeconds: 5
   109          startupProbe:
   110            failureThreshold: 30
   111            httpGet:
   112              path: /metrics
   113              port: metrics
   114              scheme: HTTP
   115            periodSeconds: 10
   116            successThreshold: 1
   117            timeoutSeconds: 5
   118          securityContext:
   119            allowPrivilegeEscalation: false
   120            privileged: false
   121            readOnlyRootFilesystem: true
   122            capabilities:
   123              drop: [ALL]
   124          volumeMounts:
   125            - name: varlog
   126              mountPath: /var/log
   127              readOnly: true
   128            - name: config
   129              mountPath: /etc/crowdsec/acquis.yaml
   130              subPath: acquis.yaml
   131              readOnly: true
   132            - name: tls
   133              mountPath: /certs
   134              readOnly: true
   135            - mountPath: /var/lib/crowdsec/data
   136              name: tmp
   137              subPath: crowdsec-data
   138            - mountPath: /etc/crowdsec
   139              name: tmp
   140              subPath: crowdsec-etc
   141            - name: tmp
   142              mountPath: /tmp
   143              subPath: tmp
   144      volumes:
   145        - name: varlog
   146          hostPath:
   147            path: /var/log
   148            type: Directory
   149        - name: config
   150          configMap:
   151            name: crowdsec-agent-config
   152        - name: tls
   153          secret:
   154            secretName: crowdsec-agent-tls
   155        - name: tmp
   156          emptyDir: {}
   

   NOTE

   The containerized CrowdSec can use multiple environment variables to configure itself. This also applies to the Agent and Appsec components.

   • CONFIG_FILE: The path to the configuration file. Defaults to /etc/crowdsec/config.yaml.
   • NO_HUB_UPGRADE=true: Disables the automatic upgrade of the CrowdSec Hub.
   • DEBUG=true: Enables debug mode for the start script.
   • COLLECTIONS/DISABLE_COLLECTIONS: Comma-separated list of collections to enable/disable.
   • PARSERS/DISABLE_PARSERS: Comma-separated list of parsers to enable/disable.
   • SCENARIOS/DISABLE_SCENARIOS: Comma-separated list of scenarios to enable/disable.
   • POSTOVERFLOWS/DISABLE_POSTOVERFLOWS: Comma-separated list of postoverflows to enable/disable.
   • CONTEXTS/DISABLE_CONTEXTS: Comma-separated list of contexts to enable/disable.
   • APPSEC_CONFIGS/DISABLE_APPSEC_CONFIGS: Comma-separated list of AppSec configurations to enable/disable.
   • APPSEC_RULES/DISABLE_APPSEC_RULES: Comma-separated list of AppSec rules to enable/disable.
   • METRICS_PORT: The port to expose the metrics on.
   • DISABLE_LOCAL_API=true: Disables the Local API role.
   • DISABLE_AGENT=true: Disables the Agent role.

   AppSec/Agent specific:

   • CUSTOM_HOSTNAME: The login and machine ID of the Agent when adding it to the Local API. Defaults to localhost.
   • USE_TLS=true: Enables TLS to connect to the Local API.
   • CACERT_FILE: Specifies the path to the CA certificate file, to check against the Local API certificate.
   • CLIENT_CERT_FILE: For client authentication to the Local API, the path to the Client certificate file.
   • CLIENT_KEY_FILE: For client authentication to the Local API, the path to the Client key file.
   • AGENT_USERNAME: Similar to CUSTOM_HOSTNAME, but used by password-based authentication to the Local API.
   • AGENT_PASSWORD: For password-based authentication to the Local API.
   • INSECURE_SKIP_VERIFY=true: Disables the verification of the Local API certificate.

   You can see we've mounted /var/log from the host using hostPath. The container is required to run as root to be able to read the logs. You should customize COLLECTIONS to match your needs, but, usually, crowdsecurity/traefik and crowdsecurity/linux should be enough.

3. (Optional) Write the service.yaml:

   base/agent/service.yaml

    1apiVersion: v1
    2kind: Service
    3metadata:
    4  name: crowdsec-agent
    5  labels:
    6    app.kubernetes.io/component: agent
    7spec:
    8  type: ClusterIP
    9  ports:
   10    - port: 6060
   11      targetPort: metrics
   12      protocol: TCP
   13      name: metrics
   14  selector:
   15    app.kubernetes.io/component: agent
   

4. Write the certificate.yaml.

   base/agent/certificate.yaml

    1apiVersion: cert-manager.io/v1
    2kind: Certificate
    3metadata:
    4  name: crowdsec-agent
    5spec:
    6  secretName: crowdsec-agent-tls
    7  duration: 2160h
    8  renewBefore: 720h
    9  subject:
   10    organizationalUnits: [agent-ou]
   11  commonName: CrowdSec agent
   12  dnsNames:
   13    - crowdsec-agent.crowdsec
   14    - crowdsec-agent.crowdsec.svc.cluster.local
   15    - localhost
   16  issuerRef:
   17    name: private-cluster-issuer
   18    kind: ClusterIssuer
   

That's it for the Agent. There won't be any overlay for the agents as there is nothing environment specific.

Base - AppSec 🔗

As said in previous parts, the deployment of the AppSec component is very similar to the Agent. One difference is that it's not a DaemonSet, but a Deployment since it will be receiving traffic and not fetch logs for each node. Based on the in-bound or out-of-bound traffic, AppSec is able to detect attack scenarios.

1. Write the acquis.yaml:

   base/appsec/files/acquis.yaml

    1# In old guides, you may see "appsec_config". You should use "appsec_configs" instead.
    2# appsec_config: crowdsecurity/appsec-default
    3appsec_configs:
    4  - crowdsecurity/appsec-default # This includes base-config, vpatch and generic rules.
    5  - crowdsecurity/crs # This includes OWASP CRS, which is used in most WAF, including Cloudflare.
    6labels:
    7  type: appsec
    8listen_addr: 0.0.0.0:7422
    9path: /
   10source: appsec
   

   crowdsecurity/crs can trigger many false positives, so I recommend that you should write your own AppSec configuration. You can follow this guide to know more.

2. Write the haproxy.cfg:

   base/appsec/files/haproxy.cfg

    1defaults
    2  mode http
    3  balance roundrobin
    4  log global
    5  timeout connect 5s
    6  timeout client 50s
    7  timeout server 50s
    8
    9crt-store web
   10  crt-base /certs/
   11  key-base /certs/
   12  load crt "tls.crt" key "tls.key"
   13
   14frontend front
   15  bind :7423 ssl crt "@web/tls.crt"
   16  capture request header Host len 64
   17  option forwardfor
   18  default_backend back
   19
   20# Inherits mode and balance
   21backend back
   22  mode http
   23  http-request set-header Host %[req.hdr(Host)]
   24  server s1 127.0.0.1:7422 check
   

3. Write the deployment.yaml:

   base/appsec/deployment.yaml

     1apiVersion: apps/v1
     2kind: Deployment
     3metadata:
     4  name: crowdsec-appsec
     5  labels:
     6    app.kubernetes.io/component: appsec
     7spec:
     8  replicas: 1
     9  strategy:
    10    type: RollingUpdate
    11    rollingUpdate:
    12      maxUnavailable: 1
    13  selector:
    14    matchLabels:
    15      app.kubernetes.io/component: appsec
    16  template:
    17    metadata:
    18      labels:
    19        app.kubernetes.io/component: appsec
    20    spec:
    21      automountServiceAccountToken: false
    22      terminationGracePeriodSeconds: 30
    23      securityContext:
    24        fsGroup: 2000
    25        runAsUser: 1000
    26        runAsGroup: 2000
    27        runAsNonRoot: true
    28      initContainers:
    29        - name: wait-for-lapi
    30          image: busybox
    31          imagePullPolicy: IfNotPresent
    32          command:
    33            [
    34              'sh',
    35              '-c',
    36              'until nc "$LAPI_HOST" "$LAPI_PORT" -z; do echo waiting for lapi to start; sleep 5; done',
    37            ]
    38          resources:
    39            limits:
    40              memory: 50Mi
    41            requests:
    42              cpu: 1m
    43              memory: 10Mi
    44          securityContext:
    45            allowPrivilegeEscalation: false
    46            privileged: false
    47            readOnlyRootFilesystem: true
    48            runAsUser: 1000
    49            runAsGroup: 2000
    50            runAsNonRoot: true
    51            capabilities:
    52              drop: [ALL]
    53          env:
    54            - name: MY_NAMESPACE
    55              valueFrom:
    56                fieldRef:
    57                  fieldPath: metadata.namespace
    58            - name: LAPI_HOST
    59              value: 'crowdsec-lapi.$(MY_NAMESPACE)'
    60            - name: LAPI_PORT
    61              value: '8080'
    62        - name: init-perms
    63          image: crowdsecurity/crowdsec:v1.7.3
    64          imagePullPolicy: IfNotPresent
    65          securityContext:
    66            runAsUser: 0
    67            runAsGroup: 0
    68            runAsNonRoot: false
    69          command:
    70            - sh
    71            - -c
    72            - |-
    73              cp -Rf /staging/var/lib/crowdsec/data/* /out
    74              chown -R 1000:2000 /out
    75          volumeMounts:
    76            - name: crowdsec-default-data
    77              mountPath: /out
    78      containers:
    79        - name: crowdsec-appsec
    80          image: 'crowdsecurity/crowdsec:v1.7.3'
    81          imagePullPolicy: IfNotPresent
    82          env:
    83            - name: MY_NAMESPACE
    84              valueFrom:
    85                fieldRef:
    86                  fieldPath: metadata.namespace
    87            - name: DISABLE_LOCAL_API
    88              value: 'true'
    89            - name: DISABLE_ONLINE_API
    90              value: 'true'
    91            - name: CROWDSEC_BYPASS_DB_VOLUME_CHECK
    92              value: 'true'
    93            - name: LOCAL_API_URL
    94              value: 'https://crowdsec-lapi.$(MY_NAMESPACE):8080'
    95            - name: USE_TLS
    96              value: 'true'
    97            - name: CLIENT_CERT_FILE
    98              value: /certs/tls.crt
    99            - name: CLIENT_KEY_FILE
   100              value: /certs/tls.key
   101            - name: CACERT_FILE
   102              value: /certs/ca.crt
   103            - name: COLLECTIONS # Download collections, which contains parsers, enrichers, etc...
   104              value: >-
   105                crowdsecurity/traefik crowdsecurity/linux
   106                crowdsecurity/appsec-generic-rules crowdsecurity/appsec-virtual-patching
   107                crowdsecurity/appsec-crs
   108            - name: APPSEC_CONFIGS # Download AppSec configurations
   109              value: crowdsecurity/appsec-default
   110            - name: UNREGISTER_ON_EXIT
   111              value: 'true'
   112          resources:
   113            limits:
   114              memory: 250Mi
   115            requests:
   116              cpu: 10m
   117              memory: 250Mi
   118          ports:
   119            - name: appsec
   120              containerPort: 7422
   121              protocol: TCP
   122            - name: metrics
   123              containerPort: 6060
   124              protocol: TCP
   125          livenessProbe:
   126            failureThreshold: 3
   127            httpGet:
   128              path: /metrics
   129              port: metrics
   130              scheme: HTTP
   131            periodSeconds: 10
   132            successThreshold: 1
   133            timeoutSeconds: 5
   134          readinessProbe:
   135            failureThreshold: 3
   136            httpGet:
   137              path: /metrics
   138              port: metrics
   139              scheme: HTTP
   140            periodSeconds: 10
   141            successThreshold: 1
   142            timeoutSeconds: 5
   143          startupProbe:
   144            failureThreshold: 30
   145            httpGet:
   146              path: /metrics
   147              port: metrics
   148              scheme: HTTP
   149            periodSeconds: 10
   150            successThreshold: 1
   151            timeoutSeconds: 5
   152          securityContext:
   153            allowPrivilegeEscalation: false
   154            privileged: false
   155            readOnlyRootFilesystem: true
   156            runAsUser: 1000
   157            runAsGroup: 2000
   158            runAsNonRoot: true
   159            capabilities:
   160              drop: [ALL]
   161          volumeMounts:
   162            - name: config
   163              mountPath: /etc/crowdsec/acquis.yaml
   164              subPath: acquis.yaml
   165              readOnly: true
   166            - name: tls
   167              mountPath: /certs
   168              readOnly: true
   169            - mountPath: /var/lib/crowdsec/data
   170              name: tmp
   171              subPath: crowdsec-data
   172            - mountPath: /etc/crowdsec
   173              name: tmp
   174              subPath: crowdsec-etc
   175            - name: tmp
   176              mountPath: /tmp
   177              subPath: tmp
   178            - name: crowdsec-default-data
   179              mountPath: /staging/var/lib/crowdsec/data
   180        - name: haproxy
   181          image: registry-1.docker.io/library/haproxy:3.2.8-alpine
   182          imagePullPolicy: IfNotPresent
   183          ports:
   184            - name: appsec-secure
   185              containerPort: 7423
   186          livenessProbe:
   187            tcpSocket:
   188              port: appsec-secure
   189            initialDelaySeconds: 30
   190            periodSeconds: 30
   191          resources:
   192            limits:
   193              memory: 250Mi
   194            requests:
   195              cpu: 10m
   196              memory: 128Mi
   197          volumeMounts:
   198            - name: haproxy-config
   199              mountPath: /usr/local/etc/haproxy/haproxy.cfg
   200              subPath: haproxy.cfg
   201              readOnly: true
   202            - name: tls
   203              mountPath: /certs
   204              readOnly: true
   205            - name: tmp
   206              mountPath: /tmp
   207              subPath: tmp
   208      volumes:
   209        - name: config
   210          configMap:
   211            name: crowdsec-appsec-config
   212        - name: tls
   213          secret:
   214            secretName: crowdsec-appsec-tls
   215        - name: tmp
   216          emptyDir: {}
   217        - name: crowdsec-default-data
   218          emptyDir: {}
   219        - name: haproxy-config
   220          configMap:
   221            name: crowdsec-appsec-haproxy-config
   

   This section requires some explanation. Crowdsec Appsec does not open an HTTPS port, the only way to secure it is by using a lightweight L7 reverse proxy like HAProxy.

   Moreover, compared to the Crowdsec Agent that is running as root to fetch the logs, the Crowdsec Appsec component is running as rootless, therefore, we need to fix the permissions inside the container by using a sidecar container (init-perms) and a volume (crowdsec-default-data).

   NOTE

   The containerized CrowdSec can use multiple environment variables to configure itself. This also applies to the Agent and Appsec components.

   • CONFIG_FILE: The path to the configuration file. Defaults to /etc/crowdsec/config.yaml.
   • NO_HUB_UPGRADE=true: Disables the automatic upgrade of the CrowdSec Hub.
   • DEBUG=true: Enables debug mode for the start script.
   • COLLECTIONS/DISABLE_COLLECTIONS: Comma-separated list of collections to enable/disable.
   • PARSERS/DISABLE_PARSERS: Comma-separated list of parsers to enable/disable.
   • SCENARIOS/DISABLE_SCENARIOS: Comma-separated list of scenarios to enable/disable.
   • POSTOVERFLOWS/DISABLE_POSTOVERFLOWS: Comma-separated list of postoverflows to enable/disable.
   • CONTEXTS/DISABLE_CONTEXTS: Comma-separated list of contexts to enable/disable.
   • APPSEC_CONFIGS/DISABLE_APPSEC_CONFIGS: Comma-separated list of AppSec configurations to enable/disable.
   • APPSEC_RULES/DISABLE_APPSEC_RULES: Comma-separated list of AppSec rules to enable/disable.
   • METRICS_PORT: The port to expose the metrics on.
   • DISABLE_LOCAL_API=true: Disables the Local API role.
   • DISABLE_AGENT=true: Disables the Agent role.

   AppSec/Agent specific:

   • CUSTOM_HOSTNAME: The login and machine ID of the Agent when adding it to the Local API. Defaults to localhost.
   • USE_TLS=true: Enables TLS to connect to the Local API.
   • CACERT_FILE: Specifies the path to the CA certificate file, to check against the Local API certificate.
   • CLIENT_CERT_FILE: For client authentication to the Local API, the path to the Client certificate file.
   • CLIENT_KEY_FILE: For client authentication to the Local API, the path to the Client key file.
   • AGENT_USERNAME: Similar to CUSTOM_HOSTNAME, but used by password-based authentication to the Local API.
   • AGENT_PASSWORD: For password-based authentication to the Local API.
   • INSECURE_SKIP_VERIFY=true: Disables the verification of the Local API certificate.

4. Write the service.yaml:

   base/appsec/service.yaml

    1apiVersion: v1
    2kind: Service
    3metadata:
    4  name: crowdsec-appsec
    5  labels:
    6    app.kubernetes.io/component: appsec
    7spec:
    8  type: ClusterIP
    9  ports:
   10    - port: 6060
   11      targetPort: metrics
   12      protocol: TCP
   13      name: metrics
   14    - port: 7422
   15      targetPort: appsec
   16      protocol: TCP
   17      name: appsec
   18    - port: 7423
   19      targetPort: appsec-secure
   20      protocol: TCP
   21      name: appsec-secure
   22  selector:
   23    app.kubernetes.io/component: appsec
   

5. Write the certificate.yaml:

   base/appsec/certificate.yaml

    1apiVersion: cert-manager.io/v1
    2kind: Certificate
    3metadata:
    4  name: crowdsec-appsec
    5spec:
    6  secretName: crowdsec-appsec-tls
    7  duration: 2160h
    8  renewBefore: 720h
    9  subject:
   10    organizationalUnits: [appsec-ou]
   11  commonName: CrowdSec Appsec
   12  dnsNames:
   13    - crowdsec-appsec.crowdsec
   14    - crowdsec-appsec.crowdsec.svc.cluster.local
   15    - localhost
   16  issuerRef:
   17    name: private-cluster-issuer
   18    kind: ClusterIssuer