CVE-2004-0491

A local attacker can exploit this by using the `SHM_LOCK` and `SHM_UNLOCK` operations on System V shared memory segments to manipulate the `mm->locked_vm` counter without proper rlimit enforcement. The original check only required `CAP_IPC_LOCK`, not the per-user rlimit, so an unprivileged process can unlock pages belonging to another process, causing `locked_vm` to underflow or remain inflated [ref_id=1]. This lets the attacker mlock more memory than the `RLIMIT_MEMLOCK` limit would normally allow, exhausting system memory.

The linux-2.4.21-mlock.patch in Red Hat Enterprise Linux 3 contains an accounting flaw in `mm/mlock.c`, `mm/mmap.c`, `mm/mremap.c`, and `mm/shmem.c`. The vulnerability arises because `mm->locked_vm` is not correctly tracked when one process unlocks pages that belong to another process (e.g. via `SHM_UNLOCK` on a shared memory segment), allowing the mlock page count to become inconsistent.

The patch reworks the accounting in `shmem_lock()` to properly increment and decrement `mm->locked_vm` when `SHM_LOCK`/`SHM_UNLOCK` is performed, and ensures the rlimit check (`RLIMIT_MEMLOCK`) is applied before allowing the lock [ref_id=1]. It also introduces `can_do_mlock()` which returns true if the caller has either `CAP_IPC_LOCK` or a non-zero memlock rlimit, and adds rlimit checks (with `capable()` fallback) to `mlock.c`, `mmap.c`, and `mremap.c`. The default `RLIMIT_MEMLOCK` on many architectures is changed from `LONG_MAX` to `PAGE_SIZE` so that unprivileged users have a finite limit enforced from the start.