Libtiff: tiffrasterscanlinesize64 produce too-big size and could cause oom

An attacker crafts a TIFF file with specially chosen values for `td->td_bitspersample` and `td->td_imagewidth` (offsets 0xC2-0xC3 and 0xAA-0xAB respectively) so that `TIFFRasterScanlineSize64()` computes an extremely large scanline size [ref_id=1]. When an application (such as the PoC program or `tiffcp`) calls `TIFFRasterScanlineSize()` and then allocates memory for that size, the allocation exhausts available memory, causing a denial of service [ref_id=1]. The crafted input can be smaller than 379 KB, making the attack easy to deliver remotely.

The bug resides in `TIFFRasterScanlineSize64()` in libtiff. This function computes a scanline size by multiplying user-controlled fields `td->td_bitspersample` and `td->td_imagewidth` (and optionally `td->td_samplesperpixel`) without any upper-bound check [ref_id=1]. The caller `TIFFRasterScanlineSize()` returns this unchecked value, which is then used for memory allocation [ref_id=1].

The advisory does not include a patch; it reports the issue at commit 4d0329a4539550f2396772b8c4c60c5fecdda7db [ref_id=1]. The recommended remediation is to add an upper-bound check in `TIFFRasterScanlineSize64()` (or its caller) so that the computed scanline size cannot exceed a sane maximum before being used for allocation [ref_id=1]. Without such a check, any code path that allocates based on the return value of `TIFFRasterScanlineSize()` is vulnerable to OOM.

1. Download the PoC input from https://github.com/PromptFuzz/crash_inputs/raw/main/libtiff/oom1/triger_input_41 [ref_id=1]. 2. Build libtiff with ASan. 3. Compile the PoC program: `clang++ -fsanitize=fuzzer,address -g -O0 -I/libtiff/include poc.cc -o poc.out libtiff.a -lz -ljpeg -llzma -ljbig` [ref_id=1]. 4. Run `./poc.out triger_input_41` to trigger the OOM [ref_id=1].