Integer overflow vulnerabilities are particularly insidious because they occur in seemingly safe code that performs arithmetic before allocating memory or copying data.

Unlike buffer overflows that directly write past boundaries, integer overflows corrupt the calculation that determines how much memory to use. This leads to undersized allocations followed by operations that assume the full size, resulting in heap corruption or information disclosure.

Example 1: Apple JBIG2 Image Processing (CVE-2020-9832)

The JBIG2 format is a highly compressed image format used in PDF documents. Apple's CoreGraphics framework, which handles image rendering on macOS and iOS, contained a critical integer overflow vulnerability in its JBIG2 decoder. JBIG2 vulnerabilities were exploited by NSO Group's Pegasus mobile device spyware.

The attack vector was particularly insidious. Apple's iMessage automatically renders images in received messages to generate previews. The ImageIO library, which CoreGraphics uses to process images, attempts to automatically detect the file format by examining the file's contents rather than trusting the file extension. An attacker could send a malicious JBIG2-encoded file named with a .gif extension via iMessage. The ImageIO library would ignore the extension, detect the JBIG2 format, and pass it to the vulnerable JBIG2 decoder—all without any user interaction. This allowed access to over 20 different image codecs simply by choosing an appropriate filename extension.

The vulnerability lies in how the decoder calculates memory allocation sizes for decompressed image data. JBIG2 images are composed of segments, each with width and height parameters supplied in the file format. The decoder must allocate a buffer large enough to hold the decompressed bitmap data.

Simplified vulnerable code pattern

struct jbig2_segment {
    uint32_t width;
    uint32_t height;
    uint32_t stride;   // bytes per row
    // ... other fields
};

void* decode_jbig2_segment(struct jbig2_segment *seg) {
    // Calculate total buffer size needed
    size_t buffer_size = seg->height * seg->stride;

    // Allocate buffer for decompressed data
    uint8_t *buffer = malloc(buffer_size);
    if (!buffer) return NULL;

    // Decompress data into buffer
    decompress_jbig2_data(buffer, seg);

    return buffer;
}

The vulnerability

The multiplication seg->height * seg->stride can overflow if both values are large. On a 32-bit system, if height is 0x10000 (65,536) and stride is 0x10000, the multiplication produces 0x100000000, which wraps to 0x00000000 when stored in a 32-bit size_t. On a 64-bit system, even larger values would be needed, but the principle remains the same.

The result: malloc(0) or malloc(small_value) allocates a tiny buffer, but decompress_jbig2_data writes height * stride bytes of actual data, causing a massive heap overflow.

Exploitation approach

Craft the malicious segment: Create a JBIG2 segment with carefully chosen dimensions:
- Set height = 0x10000 (65,536)
- Set stride = 0x10000 (65,536)
- Product: 0x10000 * 0x10000 = 0x100000000, wraps to 0 on 32-bit
Trigger undersized allocation: When the decoder processes this segment, it allocates only 4,095 bytes but the decompression routine expects to write over 4 billion bytes of data.
Corrupt heap metadata: The overflow overwrites:
- Adjacent heap chunks and their metadata
- Other allocated objects in the heap
- Potentially function pointers or vtables
Achieve code execution: By carefully structuring the heap layout and controlling the overflow data, an attacker can:
- Overwrite a function pointer with the address of shellcode
- Corrupt an Objective-C object's vtable pointer
- Manipulate allocator metadata to return a controlled pointer on the next allocation

Key challenges

Several factors complicate reliable exploitation. The heap layout must be carefully groomed to place sensitive data adjacent to the vulnerable buffer. The overflow data must be valid decompressed JBIG2 data, constraining what bytes can be written. Modern heap allocators include integrity checks that must be bypassed. Finally, ASLR requires information leaks to determine target addresses.

Defense

The fix checks for overflow before allocation. Modern code uses compiler built-ins to detect when multiplication would overflow:

size_t buffer_size;
if (__builtin_mul_overflow(seg->height, seg->stride, &buffer_size)) {
    return NULL;  // Overflow detected, reject segment
}
uint8_t *buffer = malloc(buffer_size);

Example 2: Microsoft SMBGhost (CVE-2020-0796)

SMBGhost is a critical vulnerability in the Windows 10 SMBv3 protocol implementation that allows unauthenticated remote code execution. The vulnerability exists in the kernel-mode driver srv2.sys, which handles Server Message Block version 3 protocol requests. An attacker can trigger this vulnerability by sending a specially crafted compressed message to a Windows 10 system with SMBv3 enabled.

The vulnerability lies in how the driver validates and decompresses SMB messages. SMBv3 supports compression to improve performance over slow links. Each compressed message includes a header specifying the original uncompressed size and the compressed size.

Simplified vulnerable code pattern

struct smb_compression_header {
    uint32_t protocol_id;
    uint32_t original_size;
    uint16_t compressed_algo;
    uint16_t flags;
    uint32_t offset;
};

int decompress_smb_message(struct smb_compression_header *hdr, 
                          uint8_t *compressed_data) {
    // Validate that offset + original_size doesn't exceed limits
    uint32_t total_size = hdr->offset + hdr->original_size;

    if (total_size > MAX_MESSAGE_SIZE) {
        return -1;  // Size check
    }

    // Allocate buffer for decompressed data
    uint8_t *buffer = ExAllocatePoolWithTag(
        NonPagedPool, 
        total_size,
        'bmS2'
    );

    if (!buffer) return -1;

    // Decompress into buffer
    decompress_data(buffer + hdr->offset, 
                   compressed_data, 
                   hdr->original_size);

    return 0;
}

The vulnerability

The addition hdr->offset + hdr->original_size can overflow. If an attacker sets:

offset = 0xFFFF0000
original_size = 0x00020000

The sum is 0x100010000, which wraps to 0x00010000 (65,536) when truncated to 32 bits. This passes the size check (0x00010000 < MAX_MESSAGE_SIZE), so the driver allocates only 65,536 bytes.

However, the decompression writes data at buffer + offset, which is buffer + 0xFFFF0000. This offset is far beyond the allocated buffer, writing into adjacent kernel memory.

Exploitation approach

Trigger the overflow: Send an SMB compressed message with:
- offset = 0xFFFF0000
- original_size = 0x00010000
- Compressed data that decompresses to controlled content
Target kernel structures: The overflow occurs in kernel memory (NonPagedPool), where critical structures are stored:
- Pool headers containing metadata for memory management
- Kernel objects such as process structures
- Function pointers in driver dispatch tables
Corrupt pool metadata: By overwriting adjacent pool headers, an attacker can:
- Cause the pool allocator to return an arbitrary address on the next allocation
- Overwrite kernel function pointers
- Corrupt security tokens or process objects
Achieve SYSTEM privileges: Once kernel memory is corrupted:
- Overwrite a process token to gain SYSTEM privileges
- Overwrite a driver dispatch function to execute shellcode
- Modify kernel security checks to disable protections

Why this is dangerous

Several factors make SMBGhost particularly severe:

The vulnerability is in the Windows kernel, giving complete system control
It's remotely exploitable without authentication
SMBv3 is enabled by default on Windows 10 and Server 2019
The vulnerability affects the listening service, not requiring user interaction

Key challenges

Kernel exploits must bypass Kernel Address Space Layout Randomization (KASLR)
Pool memory layout is dynamic and must be groomed carefully
Kernel crashes (Blue Screen of Death) occur if exploitation fails
Modern Windows has additional protections like Control Flow Guard (CFG)

Defense

Microsoft's patch validates the calculation before use. The fix uses overflow detection to reject invalid messages:

uint32_t total_size;
if (__builtin_add_overflow(hdr->offset, hdr->original_size, &total_size)) {
    return -1;  // Overflow detected
}
if (total_size > MAX_MESSAGE_SIZE) {
    return -1;
}

Alternatively, the code can check the operands before addition to ensure the result will fit:

if (hdr->offset > MAX_MESSAGE_SIZE - hdr->original_size) {
    return -1;  // Would overflow
}

Example 3: Generic Image Decoder Pattern

Many image format vulnerabilities follow a similar pattern because image files specify dimensions in their headers, and decoders must allocate buffers based on those dimensions. This pattern appears across PNG, JPEG, TIFF, GIF, and other formats. Understanding this common vulnerability pattern helps explain why image processing libraries have been a persistent source of security issues.

Common vulnerable pattern

Image decoders typically read dimension information from the file header and calculate the buffer size needed to hold the decompressed pixel data. A typical implementation looks like this:

struct image_header {
    uint32_t width;
    uint32_t height;
    uint8_t  bits_per_pixel;
    uint8_t  channels;  // RGB = 3, RGBA = 4
};

void* decode_image(struct image_header *hdr, uint8_t *compressed_data) {
    // Calculate bytes per pixel
    uint32_t bytes_per_pixel = (hdr->bits_per_pixel * hdr->channels) / 8;

    // Calculate row stride (bytes per row)
    uint32_t stride = hdr->width * bytes_per_pixel;

    // Calculate total buffer size
    size_t buffer_size = hdr->height * stride;

    // Allocate and decode
    uint8_t *buffer = malloc(buffer_size);
    decode_pixels(buffer, compressed_data, buffer_size);

    return buffer;
}

Multiple overflow points

This code contains three separate overflow vulnerabilities.

bytes_per_pixel calculation: If bits_per_pixel and channels are large enough, their multiplication can overflow. Even if this calculation doesn't overflow, the subsequent multiplication by width almost certainly will when all values are near their maximum.
stride calculation: Large width values cause overflow
buffer_size calculation: Large height and stride values cause overflow

Real-world examples

This pattern has appeared in numerous libraries over the years.

libpng (CVE-2004-0597): Overflow in row buffer allocation
libjpeg (CVE-2013-6629): Overflow in Huffman table size calculation
libtiff (CVE-2016-3945): Multiple overflows in strip and tile size calculations
giflib (CVE-2015-7555): Overflow in color map size calculation

Each of these vulnerabilities follows the same basic pattern of multiplying untrusted dimensions without overflow checking.

General exploitation approach

Exploiting these vulnerabilities follows a consistent methodology.

Find which calculation overflows with reasonable input values
Craft image dimensions to trigger the overflow
Ensure the resulting allocation is small enough to overflow into adjacent memory
Control the subsequent write operation to corrupt specific targets
Trigger the corrupted pointer or data to execute attacker-controlled code

Why these vulnerabilities persist

Several factors explain why these vulnerabilities continue to appear.

Image formats are complex with many calculation points
Decoders are often written in C for performance
Dimensions are untrusted input from potentially malicious files
Developers often miss the overflow possibility in nested multiplications
Legacy codebases may predate modern overflow-safe arithmetic

Common Patterns and Prevention

Vulnerable calculation patterns

Several calculation patterns appear repeatedly in vulnerable code. The first pattern involves nested multiplications, where multiple dimensions are multiplied together to calculate a buffer size. The second pattern adds values after multiplication, combining a calculated size with header or metadata sizes. The third pattern calculates stride values (bytes per row) before multiplying by the number of rows, creating multiple opportunities for overflow.

// Pattern 1: Nested multiplications
size = width * height * bytes_per_pixel;

// Pattern 2: Addition after multiplication  
total = (width * height) + header_size;

// Pattern 3: Stride calculations
stride = width * channels * bits_per_sample / 8;
size = height * stride;

Safe alternatives

Modern compilers provide built-in functions that detect overflow conditions. These functions perform the arithmetic operation and return a boolean indicating whether overflow occurred. Using these functions makes overflow detection explicit and reliable.

// Use builtin overflow detection
size_t temp, size;
if (__builtin_mul_overflow(width, height, &temp) ||
    __builtin_mul_overflow(temp, bytes_per_pixel, &size)) {
    return NULL;  // Overflow
}

// Or check before operation
if (width > SIZE_MAX / height) return NULL;
size = width * height;
if (size > SIZE_MAX / bytes_per_pixel) return NULL;
size *= bytes_per_pixel;

Defense in depth

No single defensive technique eliminates all integer overflow vulnerabilities. A comprehensive approach combines multiple layers of protection, each addressing different aspects of the problem.

Input validation: Reject unreasonably large dimensions early. Before performing any calculations, check that width, height, and other parameters are within reasonable bounds for the application. This catches obviously malicious values before they can cause problems.
Overflow-safe arithmetic: Use compiler builtins or manual checks for every calculation that could overflow. This is the primary defense and should be applied consistently throughout the codebase.
Bounds checking: Verify all array accesses even after safe allocation. Even if the allocation size is correct, verify that indexes stay within bounds during processing. This catches logic errors that might remain even with correct arithmetic.
Fuzzing: Test with malformed inputs and sanitizers enabled. Automated testing with tools like AddressSanitizer and UndefinedBehaviorSanitizer exposes arithmetic errors during development, long before they reach production.
Memory tagging: Use hardware features like ARM's Memory Tagging Extension (MTE) to detect out-of-bounds writes at runtime. This provides a last line of defense when other protections fail.

Integer overflow vulnerabilities demonstrate that memory safety requires careful attention to arithmetic, not just pointer operations. Every calculation that influences allocation size or copy length is a potential vulnerability if it processes untrusted input.

Memory Vulnerabilities and Exploitation

Example 1: Apple JBIG2 Image Processing (CVE-2020-9832)

Simplified vulnerable code pattern

The vulnerability

Exploitation approach

Key challenges

Defense

Example 2: Microsoft SMBGhost (CVE-2020-0796)

Simplified vulnerable code pattern

The vulnerability

Exploitation approach

Why this is dangerous

Key challenges

Defense

Example 3: Generic Image Decoder Pattern

Common vulnerable pattern

Multiple overflow points

Real-world examples

General exploitation approach

Why these vulnerabilities persist

Common Patterns and Prevention

Vulnerable calculation patterns

Safe alternatives

Defense in depth