An easy-to-Understand Overview of Popular extended BPF Tools
An in-kernel virtual machine BPF, or Berkeley Packet Filter, is known for running programs passed from user space. It was first implemented on BSD and then.
Updated
On this page
Key takeaways
- An in-kernel virtual machine BPF, or Berkeley Packet Filter, is known for running programs passed from user space.
- Falcon analyzes kernel system calls and provides an in-depth visibility into cluster, container, and host activity.
- As it continues to prove beneficial in Linux programming, the popularity and demand for eBPF tools will continue to increase in the future.
An in-kernel virtual machine BPF, or Berkeley Packet Filter, is known for running programs passed from user space. It was first implemented on BSD and then Linux, followed by the classic BPF (cBPF). In recent times, the BPF infrastructure in Linux has undergone a major transition and has given birth to the extended BPF – or eBPF as it is popularly called. Some of the popular use-cases where extended BPF has brought about flexibility and high performance include tracing and monitoring events on the system, enforcing access control over groups, and processing packets at a very low level (XDP).
eBPF is a major technology, the origins of which can be traced back to the Linux kernel. eBPF can be used to run sandboxes programs in an operating system kernel and safety extend kernel capabilities without having to load kernel modules or change its source code.
To understand how beneficial extended BPF is, let’s get down to the roots of the OS. Ideally, the operating system has been a place to implement security, observability, and networking functionality, as the kernel possesses the ability to exert overall control on the entire system. However, it is also difficult for the OS kernel to evolve, given its extensive need for stability and security. In essence, there is lowered innovation at the operating system level as opposed to the functionality outside the OS. Here’s where eBPF comes in.
Extended BPF allows application developers to run sandboxes programs within the operating system in order to run eBPF programs for adding more capabilities to the operating system at runtime. The OS is then able to guarantee excellent execution and safety – nearly as if it had been naively compiled using a JIT (Just-In-Time) compiler and verification engine. In consequence, this has led to an upsurge in eBPF-based projects covering next-generation observability and networking.
eBPF Tools/Frameworks #
BCC
BCC is a collection of tools used to create kernel tracing and manipulation programs. It makes programs much simpler to write with front-end interfaces in Python and lua and kernel instrumentation in C. BCC helps simplify many tasks such as network traffic control and performance analysis. It also provides an API for other programs to use.
Since it conveniently provides a C programming environment for kernel BPF C code along with Python or lua for a front-end interface, you can just simply use embedded C for data extraction from traced events and use a Python script or a C++ program to display data as per your suitability. BCC enhances the potential of eBPF and makes it much more tangible. As a matter of fact, BCC’s repository comprises over 70+ ready-to-go BPF programs used for debugging and performance analysis.
How BCC works:
· BCC tools can work only with the availability of kernel sources, as most of them need to register callbacks on the kernel API to get their data. These callbacks are registered using the probe infrastructure.
· When you run a BCC tool, its current directory is switched into the kernel source directory even before the compilation.
· Then BCC compiles the C program defining the BCC tool’s logic. The C program can include kernel data structures and kernel headers for probes to function.
If you’re not using the probe infrastructure, BCC adds a common helpers.h include directive during the compilation of the eBPF C program, that is found in src/cc/export/helpers.h in the BCC source.
Libbpf
Libbpf is another set of tools that is used to build eBPF applications. With respect to analytic, networking, and security applications, it offers numerous benefits over BCC. It is usually used with BPF CO-RE (compile once, run everywhere) – which was basically designed to create binaries that can run on different kernel versions and to solve portability issues with BPF.
Libbpf mitigates several roadblocks and enables developers to focus on the task at hand. It produces simple binaries, which compile once and can run anywhere. It eliminates numerous dependencies and helps you efficiently execute your code. Libbpf also creates binaries that can use less memory.
How Libbpf works:
· Libbpf loads, checks, and relocates BPF programs, sorts out maps and hooks, helping developers to implement their programs sans the heavy work.
· Libbpf helps match offsets and types and updates them automatically, enabling a program to function on a target host without requiring additions such as Clang.
· The tool uses a vmheader file that has multiple kernel types; ergo, users need not rely on system-wide kernel headers. They will need to include vmlinux.h to switch from BCC to libbpf.
Libbpf has several advantages – for starters, it has no dependencies, making it easier to use on multiple machines. It also boasts of excellent resource usage, low memory usage, and limited impact on performance.
Bpftrace
Bpftrace is an open source tracer for Linux and helps analyze troubleshooting software and production performance problems. It uses LLVM as a backend source and compiles scripts to BPF-bytecode. It deploys BCC to interact with existing Linux tracing capabilities such as user-level dynamic tracing (probes), tracepoints, and kernel dynamic tracing (probes) as well as with the with the Linux BPF system.
How bpftrace works:
· While installing bpftrace you have to be running a Linux 4.9 kernel or higher.
· You can use one-liner scripts to check if the installation works. Or, just create a new bpftrace file and execute the code in it.
· In a bpftrace suntan, use probes to specify what events to instrument. If need be, you can use a filter to filter down the events based on a boolean expression. The action part is the mini program that runs.
· Use static instrumentation as opposed to the dynamic one for more stable programs.
Bpftrace works well for a specific kind of analysis – it decomposes metrics into distributions and creates new metrics for visibility into blind spots. It is thus best for short scripts and ad hoc investigations. The Netflix performance team for example, uses bpftrace for ad hoc analysis.
eBPF Cloud Native Tools #
Falcon
Falcon is a Cloud Native tool that can be used to run eBPF applications. A well-known runtime security tool, Falcon can generate functions to monitor and combat anomalous activity in nodes and containers or report it to the necessary channels.
Falcon analyzes kernel system calls and provides an in-depth visibility into cluster, container, and host activity. It can also add Kubernetes application context to its findings and help security, DevOps, and cloud teams understand concise functions.
Falcons technical architecture has been designed to meet the needs of modern environments. It has an underlying eBPF technology, therefore it barely has a performance impact, with nearly 50% of the CPU usage of its alternatives.
The three main components of Falcon:
· The userspace program – It handles signals, dissects information from a Falcon driver, and sends alerts.
· The configuration – It defines how Falcon runs, when to trigger alerts, and what rules to postulate.
· The driver – It is a software that must be installed to run Falcon. It adheres to the Falcon driver specification and sends system call information. Presently, it supports an eBPF probe built from the same modules, Kernel module built on libscap and libsinsp C++ libraries, and userspace instrumentation.
How Falcon works:
· Define any rule that the tool can monitor. Any anomalous activity will do.
· Configure Falcon and publish the alert to a log aggregator, queue, or topic.
· Further on, configure a Google Cloud Function or something similar, that the alert can trigger.
Inspector Gadget #
Developed by Kinfolk, Inspector Gadget is a tool that helps with tracing, debugging, and observing applications inside Kubernetes. It is a collection of gadgets that’ll help debug and inspect Kubernetes resources and applications. It helps to manage the packaging, deployment and execution of BCC-based and custom-built BPF programs in a Kubernetes cluster. It also maps low-level kernel primitives to high-level Kubernetes resources automatically, thereby making it easier to find relevant information.
Every node deploys Inspector Gadget as a privileged DaemonSet. The tool uses in-kernel BPF helper programs for event monitoring – events related to syscalls from userspace programs in a pod. The kernel helps run the BPF programs and collect log data. Inspector Gadget’s userspace utilities then fetch this data and display it.
How Inspector Gadget works:
· Deploy Inspector Gadgets agents in the cluster as privileged DaemonSets.
· Install the gadget kubectl plugin.
· Choose a gadget and use it for probing data.
Inspector Gadget has a traceloop tool that deploys standalone eBPF traceloop tool underneath. Once you use the privileged DaemonSet forward, all pods system calls will be traced and some of the recent calls will be stored for future consumption.
Cilium
Cilium is completely based on eBPF. It is a Kubernetes cni plugin and relies on eBPF to provide and transparently secure network connectivity and balance loads between application workloads. It implements distributed load balancing for application container traffic and can replace components such as tube-proxy. In eBPF, efficient hashtables are used to implement load balancing, allowing for extensive scale.
For east-west load balancing, Cilium conducts service-to-backend translation in the Linux kernel’s socket layer in a way that you can avoid per-packet NAT operations overhead in lower layers.
For north-south load balancing, Cilium’s eBPF implementation can be attached to XDP (eXpress Data Path). It is also optimized for maximum performance and supports direct server return (DSR).
How Cilium Works: #
· Cilium works best with a mufti-cluster setup, so you can choose it whenever you need a high throughput solution.
· Adapt your cluster provisioning strategy accordingly to deploy Cilium as the pod network solution.
· Cilium has a lot of options and configurations; choose what suits you the best. For instance, you can replace the tube-proxy with Cilium’s own Kubernetes proxying solution.
In order to ensure load balancing between servers and to deliver security, Cilium leverages eBPF by first running eBPF programs at the socket layer and then running them at the XDP/tc layer. At the socket layer, it handles service-to-service communication, i.e., East-West traffic. At the XDP/tc layer, it handles external traffic coming into the node, i.e., North-South communication.
Hubble
Hubble is actually a security observability tool. It leverages eBPF for visibility by building on top of Helium. To that end, all visibility becomes programmable and allows for a dynamic approach, minimizing overhead and delivering a detailed insight. Hubble is one of those tools that has been specifically created to make best use of the latest eBPF capabilities.
How Hubble works:
· If one is already using Cilium, they only need to deploy Hubble’s relay and UI.
· After that, use it to view its service map, flows, and policies.
· If you are not using Cilium and you have another CNI plugin, deploy Cilium on top of another CNI.
Cilium, to control the flow of packets, sends BPF programs to the kernel, thereby enforcing policies and load balancing. The events generated from this flow are extracted by Hubble to create a collection of flow data and metrics to be consumed.
All of Hubble’s data comes from the events map in the eBPF datapath, therefore, all of its features are eBPF related. One of its major capabilities is a Service Dependence and Communication Map that offers the visualization of a service connections map. As the number of microservices in your architecture increases, it gets rather complicated to understand internal communication and how data flow in the setup is. Hubble makes this easier, especially with a lower overhead.
eBPF is a very powerful mechanism in Linux applications and helps simplify code execution in the Linux kernel space. It has already been used to create programs for tracing, networking, security, debugging, etc. It is a revolutionary technology as it helps programmers execute custom bytecode inside the kernel apace itself without changing the kernel or loading kernel modules. As it continues to prove beneficial in Linux programming, the popularity and demand for eBPF tools will continue to increase in the future.
Related reading