CVE-2017-12806

An attacker can trigger this vulnerability by providing a specially crafted image file to the ImageMagick `convert` utility. The `convert` utility processes the image, and the `format8BIM` function attempts to allocate a large amount of memory based on the malformed input. This excessive allocation exhausts available system memory, causing a denial of service. The vulnerability is triggered when processing an image file that specifies an unusually large dimension or data size within the 8BIM format [ref_id=1].

The vulnerability resides within the `format8BIM` function located in the `coders/meta.c` file. This function is responsible for parsing and processing image data in the 8BIM format. The issue arises on line 2184 of this file, where memory allocation occurs without sufficient bounds checking, leading to the memory exhaustion when processing a crafted input file [ref_id=1].

The patch, as indicated by the commit message in the reference, addresses the memory exhaustion by introducing checks within the `format8BIM` function. Specifically, it limits the maximum size of the image data that can be allocated, preventing the unbounded memory growth that led to the denial of service. This ensures that even with malformed input, the application will not attempt to allocate an amount of memory that exceeds system capabilities [ref_id=1].

``` root@ubuntu:/home/hjy/Desktop# convert oom-format8BIM /dev/null ==8638==ERROR: AddressSanitizer failed to allocate 0x72e8d000 (1927860224) bytes of LargeMmapAllocator: 12 ==8638==Process memory map follows: ... ==8638==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix.cc:66 "(("unable to mmap" && 0)) != (0)" (0x0, 0x0) ``` [ref_id=1]