How to Add eBPF Observability to Your Project
When it comes to integrating security, networking, and observability features, working within the Linux kernel is perfect. Historically, developers have.
Updated
On this page
Key takeaways
- When it comes to integrating security, networking, and observability features, working within the Linux kernel is perfect.
- 2. Incorporate a tool into your product Now consider including execsnoop into your product.
- Install the bcc or bpftrace tools, include them into your observability offering, and periodically pull package updates.
When it comes to integrating security, networking, and observability features, working within the Linux kernel is perfect. It is not, however, without its difficulties. Historically, developers have encountered complex architectures and difficult-to- debug abstraction layers, altered kernel source code or added modules. Both of these issues are addressed by the extended BPF (eBPF).
The Extended Berkeley Packet Filter (eBPF) is a kernel feature introduced in Linux 4.x that enables programs to run without modifying the kernel source code or installing extra modules. Consider it a lightweight, sandbox virtual machine (VM) embedded within the Linux kernel, where programmers can execute BPF bytecode that makes use of specific kernel resources.
Using eBPF avoids modifying kernel source code and simplifies software’s ability to exploit existing layers. As a result, it is a very compelling technology that can revolutionize the way networking, observability profoundly, and security are supplied.
The Functioning of eBPF #
eBPF applications are event-driven and have a code path associated with them. The code path comprises particular triggers referred to as hooks that, when passed, run any associated eBPF applications. Network events, system calls, function entry, and kernel tracepoints are all instances of hooks. When the code is activated, it is initially compiled to BPF bytecode. Before the bytecode is executed, it is validated to verify that it does not generate a loop. This step protects the software from inadvertently or intentionally compromising the Linux kernel.
After being activated by a hook, a program performs assistance calls. These helper cells are routines that provide eBPF with several memory access functionalities. The kernel must declare helper calls in advance, yet the set of available functions increases. An eBPF was first designed to improve the observability and security of network packet filtering. However, it evolved into a means of making the implementation of user- supplied code safer, more convenient, and more performant over time.
The Benefits of eBPF #
eBPF is often used to track userspace processes, and its benefits are evident in this case. It is a safe and effective way for ensuring the following:
1. Acceleration and performanceeBPF enable the relocation of packet processing from the kernel to the userspace. On the other hand, it is a just-in-time (JIT) compiler. After the bytecode is compiled, instead of performing a fresh interpretation of the bytecode for each method, eBPF is executed.
2. Low level of intrusivenessWhen used as a debugger, eBPF does not require the application to be stopped to examine its status.
3. SecurityEffectively, programs are sandboxes, meaning the kernel source code stays protected and untouched. The verification phase guarantees that resources are not clogged by programs that execute indefinitely.
4. It is tracing in a unified manner eBPF provides a centralized, robust, and easily accessible platform for process tracing. This increases visibility and security.
5. Programmability By utilizing eBPF, an environment’s feature set may be increased without introducing new layers. Similarly, because code is executed directly in the kernel, data may be stored between eBPF events rather than dumped as other tracers do.
6. ExpressivenesseBPF is expressive capable of accomplishing tasks that are often reserved for high-level languages.
7. Convenience Writing code that hooks kernel functions takes far less effort than generating and maintaining kernel modules.
There is a set of rules that must be keep in mind while running the program:
1. Execute your initial toolInstall the bcc or bpftrace tools first. For instance, bcc on Ubuntu: While this program is simplistic, I’ve resolved several performance issues with it alone, including misconfigured systems where a shell script repeatedly launches failed processes and when some little application crashes and restarts every few minutes but is not observed.
2. Incorporate a tool into your product Now consider including execsnoop into your product. Agents are almost certainly already operating on all of your customers’ computers. Is there a way to execute a command and return the text output? Or do you execute a command and then aggregate the result somewhere else (S3, Hive, Druid, etc.)? Given the breadth of available solutions, it’s basically a matter of personal taste depending on your current system and customer situations.
When you add your first tool to your product, configure it to operate for a brief period of time, between 10 and 60 seconds. As execsnoop(8) currently lacks a duration option, you may wrap it using watch -s2 60 execsnoop-bpfcc. If you want to operate these tools 24 hours a day, investigate overheads beforehand to determine the cost. Events with a low frequency of occurrence, such as process execution, should be captured in an insignificant amount. You may also use the bpftrace versions instead of bcc. These often lack canned options (-v, -l, etc. ), but they do support json output.
3. Do not be concerned about dependenciesIndeed, I recommend installing bcc or bpftrace on your customers’ computers, as they now need llm. This can accumulate to tens of Bytes, which can become an issue in some resource-constrained applications (embedded). We’ve been working diligently to address this issue in the future. bcc has updated the tools (libbpf-tools) to utilize BTF and CO-RE (rather than Python), which will eventually allow you to install 100-Byte binary versions of the tools without any dependencies. bpftrace has a similar goal of creating a tiny, dependency-free binary that takes advantage of newer kernel capabilities.
This does require Linux 5.8 or later to function properly, and your clients may not have that for years. In the meanwhile, I’d propose ignoring the llm dependencies for the time being, since they will be resolved eventually.
Not all Linux distributions have enabled CONFIG DEBUG INFO BTF=y, which is required for BTF and CO-future. It has been enabled in a number of major distributions, including Ubuntu 20.10, Fedora 30, and RHEL 8.2. However, if you are aware that any of your customers are using an unusual configuration, please verify and advise them or the bistro vendor to change CONFIG DEBUG INFO BTF=y and CONFIG DEBUG INFO BTF MODULES=y to avoid future misery.
4. bcc versus bpftraceThe bcc utilities are the easiest to use, as they generally contain several command-line parameters. The bpftrace tools are easier to update and configure, and bpftrace offers a json output mode.
If you’re absolutely new to tracing, go with bcc. If you want to perform some hacking and modifying of the tools, go with bpftrace. In the end, they are both solid solutions.
CASE STUDIES #
- **Netflix:**Netflix is developing a new graphical user interface (GUI) based on the bpftrace versions of these tools. While the bpftrace binary is installed on all target computers, the bpftrace tools (text files) are maintained on a web server and distributed as needed. This implies we can always run the most up-to-date version of the tools by upgrading them in a single location.
This is presently implemented as part of our FlameCommander UI, which also manages cloud-based flame graphs. Our prior BPF GUI was included with Vector and utilized bcc, but has now been deprecated. We’re likely to open source the new one and write about it on the Netflix technology blog.
- FacebookFacebook is a sophisticated user of BPF, but the technical details of how they use the tools fleet-wide are not entirely available. Given their involvement with bcc and their development of the BTF and CO-RE technologies, I’m inclined to believe their solution is based on the bcc libbpf-tool versions.
Conclusion
In short, check to see whether your monitoring systems already have a BPF agent and, if not, develop one using the current bcc or bpftrace tools rather than rebuilding everything from scratch. This is thinking like a systems administrator who installs and maintains the software rather than a programmer who programs everything. Install the bcc or bpftrace tools, include them into your observability offering, and periodically pull package updates.
Related reading