Falco Runtime Threat Detection and Alert Triage

Objective

Falco is a cloud-native runtime security tool that uses eBPF or kernel modules to monitor system calls. It detects anomalous behavior in running containers that admission controls cannot catch — an attacker who has already gained code execution. This exercise deploys Falco, simulates three real-world attack scenarios, verifies alerts fire, and produces actionable triage runbooks.

Run attack simulations only in isolated test environments. Shell exec into containers and outbound connection attempts will trigger real security monitoring in production environments.

Prerequisites

Kubernetes cluster with node access (Falco requires kernel-level access)
Helm installed; cluster-admin permissions
Prometheus + Alertmanager for alert routing (optional but recommended)
A test pod running (nginx or busybox) for the exec simulation

Steps

Install Falco with Helm

Falco runs as a DaemonSet and can use either a kernel module or eBPF probe. Use eBPF for managed Kubernetes where you cannot install kernel modules.

# Add Falco Helm repo
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco with eBPF driver (safe for managed K8s)
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set falco.json_output=true \
  --set falco.log_stderr=true \
  --set falco.priority=debug \
  --wait

# Verify Falco DaemonSet is running on all nodes
kubectl get pods -n falco -o wide
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=20

Write custom Falco rules

Create a custom rules file targeting our three attack scenarios. Custom rules supplement (not replace) the default ruleset.

# custom-rules.yaml (store as a ConfigMap)
cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: falco-custom-rules
  namespace: falco
data:
  custom-rules.yaml: |
    # Rule 1: Shell spawned in container
    - rule: Shell Spawned in Container
      desc: A shell was spawned in a running container
      condition: >
        spawned_process and container and
        (proc.name in (bash, sh, zsh, ksh, fish) or
         proc.name endswith sh) and
        not container.image.repository in (allowed_shell_images)
      output: >
        Shell spawned (user=%user.name container=%container.name
        image=%container.image.repository:%container.image.tag
        pod=%k8s.pod.name ns=%k8s.ns.name cmd=%proc.cmdline
        parent=%proc.pname)
      priority: WARNING
      tags: [container, shell, mitre_execution]

    # Rule 2: Outbound connection to suspicious IP
    - rule: Outbound Connection to C2 IP
      desc: Container made outbound TCP connection to known C2 range
      condition: >
        outbound and container and
        fd.sip.name not in (allowed_outbound_hosts) and
        not fd.sip in (dns_server_ip) and
        fd.sport > 1024
      output: >
        Suspicious outbound connection (user=%user.name
        container=%container.name image=%container.image.repository
        pod=%k8s.pod.name conn=%fd.name)
      priority: CRITICAL
      tags: [network, mitre_command_and_control]

    # Rule 3: setuid/setgid binary execution (privilege escalation)
    - rule: SUID Binary Executed
      desc: A setuid binary was executed in a container
      condition: >
        spawned_process and container and
        (proc.is_suid_exe=true or proc.is_sgid_exe=true) and
        not proc.name in (su, sudo)
      output: >
        SUID binary executed (user=%user.name exe=%proc.exe
        container=%container.name pod=%k8s.pod.name
        image=%container.image.repository)
      priority: ERROR
      tags: [container, privilege_escalation, mitre_privilege_escalation]

    - list: allowed_shell_images
      items: []

    - list: allowed_outbound_hosts
      items: ["kubernetes.default.svc"]

    - macro: dns_server_ip
      condition: fd.sip startswith "10.96.0."
EOF

# Patch Falco to load custom rules
helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set customRules."custom-rules\.yaml"="$(kubectl get cm falco-custom-rules -n falco -o jsonpath='{.data.custom-rules\.yaml}')"

Deploy a target pod for attack simulation

# Deploy a vulnerable-looking nginx pod
kubectl run target-pod \
  --image=nginx:latest \
  --namespace=default \
  --port=80

kubectl wait pod/target-pod --for=condition=Ready --timeout=60s

# Open a terminal to watch Falco logs in real time
kubectl logs -n falco -l app.kubernetes.io/name=falco -f &
FALCO_LOG_PID=$!

Scenario 1: Shell exec in running container

The most common post-exploitation technique. An attacker who has compromised a web application might exec into the container to explore the environment.

# Simulate: attacker execs bash in a running container
ATTACK_START=$(date -u +%s)
kubectl exec -it target-pod -- /bin/bash -c "id; hostname; ls /etc/passwd"

# Check Falco logs for the alert (within 2-5 seconds)
sleep 3
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=20 | \
  grep -i "shell spawned"

# Expected Falco output (JSON):
# {"output":"Shell spawned (user=root container=target-pod
#   image=nginx:latest pod=target-pod ns=default
#   cmd=bash -c id; hostname; ls /etc/passwd parent=runc)",
#   "priority":"WARNING","rule":"Shell Spawned in Container",
#   "time":"2024-01-15T10:30:00.000000000Z"}
ALERT_TIME=$(date -u +%s)
echo "Alert latency: $((ALERT_TIME - ATTACK_START)) seconds"

Target SLA: Alert should fire within 5 seconds of the exec command. Falco processes system calls in near-real-time via eBPF ring buffer.

Scenario 2: Outbound connection to C2 IP

# Simulate: container making outbound connection to external IP
kubectl exec target-pod -- /bin/sh -c \
  "curl -s --connect-timeout 3 http://203.0.113.1 || true"
# 203.0.113.1 is TEST-NET (RFC 5737) — safe to use in exercises

# Check Falco for the network alert
sleep 3
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=10 | \
  grep -i "outbound\|c2\|suspicious"

Write triage runbooks

Each alert type needs a runbook so on-call engineers can respond consistently. Document these in your runbook repository.

## RUNBOOK: Shell Spawned in Container
## Alert: falco:Shell Spawned in Container  Priority: WARNING

## Step 1: Identify the container
kubectl get pod $POD_NAME -n $NAMESPACE -o yaml | \
  grep -A5 "image:"

## Step 2: Check if exec was from platform team (kubectl exec)
## vs unexpected process tree
kubectl get events -n $NAMESPACE --field-selector \
  involvedObject.name=$POD_NAME | grep exec

## Step 3: Examine running processes in the container
kubectl exec $POD_NAME -n $NAMESPACE -- ps aux

## Step 4: Check for data exfiltration indicators
kubectl exec $POD_NAME -n $NAMESPACE -- \
  netstat -an 2>/dev/null | grep ESTABLISHED

## Step 5: Isolate if confirmed malicious
kubectl label pod $POD_NAME quarantine=true -n $NAMESPACE
## Apply NetworkPolicy that blocks all egress from quarantine label

## Step 6: Capture forensic snapshot
kubectl exec $POD_NAME -n $NAMESPACE -- \
  tar czf /tmp/forensics.tar.gz /proc /etc /var/log 2>/dev/null
kubectl cp $NAMESPACE/$POD_NAME:/tmp/forensics.tar.gz ./forensics.tar.gz

## Escalation: If confirmed attacker, immediately notify security team
## SLA: Alert to triage: 5 min | Triage to decision: 15 min

Success Criteria

Falco DaemonSet running on all cluster nodes Custom rules loaded — verify with kubectl exec into Falco pod and check rule list Shell exec alert fires within 5 seconds of kubectl exec Outbound connection alert visible in Falco logs Alert output contains: container name, pod name, namespace, image, and command Triage runbook written for at least 2 of the 3 scenarios with specific kubectl commands