VYPR
Low severityNVD Advisory· Published Feb 17, 2024· Updated Nov 4, 2025

CVE-2024-20925

CVE-2024-20925

Description

Vulnerability in the Oracle Java SE, Oracle GraalVM Enterprise Edition product of Oracle Java SE (component: JavaFX). Supported versions that are affected are Oracle Java SE: 8u391; Oracle GraalVM Enterprise Edition: 20.3.12 and 21.3.8. Difficult to exploit vulnerability allows unauthenticated attacker with network access via multiple protocols to compromise Oracle Java SE, Oracle GraalVM Enterprise Edition. Successful attacks require human interaction from a person other than the attacker. Successful attacks of this vulnerability can result in unauthorized update, insert or delete access to some of Oracle Java SE, Oracle GraalVM Enterprise Edition accessible data. Note: This vulnerability applies to Java deployments, typically in clients running sandboxed Java Web Start applications or sandboxed Java applets, that load and run untrusted code (e.g., code that comes from the internet) and rely on the Java sandbox for security. This vulnerability does not apply to Java deployments, typically in servers, that load and run only trusted code (e.g., code installed by an administrator). CVSS 3.1 Base Score 3.1 (Integrity impacts). CVSS Vector: (CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:N/I:L/A:N).

AI Insight

LLM-synthesized narrative grounded in this CVE's description and references.

A difficult-to-exploit JavaFX vulnerability allows unauthenticated remote attackers to corrupt some Java SE/GraalVM data via media framing issues.

Vulnerability

Overview CVE-2024-20925 is a vulnerability in the JavaFX component of Oracle Java SE (version 8u391) and Oracle GraalVM Enterprise Edition (versions 20.3.12 and 21.3.8) [1]. The root cause is an integer handling issue in the media framing code of JavaFX, specifically observed in the VideoFrame class where signed integers were used for width and height, potentially leading to integer overflow or underflow [2][3]. Additionally, the Pixels class lacked proper overflow checks when computing buffer sizes, which could be exploited to bypass allocation size checks [4].

Attack

Vector and Exploitation An unauthenticated attacker with network access can trigger the vulnerability by sending crafted media data over multiple protocols, but the attack is difficult to exploit and requires user interaction [1]. The exploitation path involves convincing a user to load malicious media content in a sandboxed Java Web Start application or applet [1]. The vulnerability does not affect server-side deployments that run only trusted code [1].

Impact

The successful exploitation allows an unauthorized attacker to update, insert, or delete some data accessible by the affected Java component [1]. The CVSS 3.1 base score is 3.1 (Low), with only Integrity impact (no Confidentiality or Availability impact) [1].

Mitigation

Oracle has released patches for this vulnerability as part of the January 2024 Critical Patch Update. The commits fixing the issue are available in the OpenJDK repositories, including the jfx21u, jfx17u, and jfx main branches [2][3][4]. Affected users should apply the latest updates to Java SE and GraalVM Enterprise Edition.

AI Insight generated on May 20, 2026. Synthesized from this CVE's description and the cited reference URLs; citations are validated against the source bundle.

Affected packages

Versions sourced from the GitHub Security Advisory.

PackageAffected versionsPatched versions
org.openjfx:javafx-mediaMaven
< 17.0.1017.0.10
org.openjfx:javafx-mediaMaven
>= 18, < 21.0.221.0.2

Affected products

37

Patches

3
18206453163d

backports of JDK-8313048, JDK-8313105, JDK-8313056 (#175)

https://github.com/openjdk/jfx17uJohan VosJan 16, 2024via ghsa
17 files changed · +612 363
  • modules/javafx.graphics/src/main/java/com/sun/glass/ui/gtk/GtkPixels.java+7 18 modified
    @@ -25,7 +25,6 @@
     package com.sun.glass.ui.gtk;
     
     import com.sun.glass.ui.Pixels;
    -import java.nio.Buffer;
     import java.nio.ByteBuffer;
     import java.nio.IntBuffer;
     
    @@ -52,31 +51,21 @@ protected void _fillDirectByteBuffer(ByteBuffer bb) {
             // Taken from MacPixels
             if (this.bytes != null) {
                 this.bytes.rewind();
    -            if (this.bytes.isDirect()) {
    -                _copyPixels(bb, this.bytes, getWidth()*getHeight());
    -            } else {
    -                bb.put(this.bytes);
    -            }
    +            bb.put(this.bytes);
                 this.bytes.rewind();
             } else {
                 this.ints.rewind();
    -            if (this.ints.isDirect()) {
    -                _copyPixels(bb, this.ints, getWidth()*getHeight());
    -            } else {
    -                for (int i=0; i<this.ints.capacity(); i++) {
    -                    int data = this.ints.get();
    -                    bb.put((byte)((data)&0xff));
    -                    bb.put((byte)((data>>8)&0xff));
    -                    bb.put((byte)((data>>16)&0xff));
    -                    bb.put((byte)((data>>24)&0xff));
    -                }
    +            for (int i=0; i<this.ints.capacity(); i++) {
    +                int data = this.ints.get();
    +                bb.put((byte)((data)&0xff));
    +                bb.put((byte)((data>>8)&0xff));
    +                bb.put((byte)((data>>16)&0xff));
    +                bb.put((byte)((data>>24)&0xff));
                 }
                 this.ints.rewind();
             }
         }
     
    -    protected native void _copyPixels(Buffer dst, Buffer src, int size);
    -
         @Override
         protected native void _attachInt(long ptr, int w, int h, IntBuffer ints, int[] array, int offset);
     
    
  • modules/javafx.graphics/src/main/java/com/sun/glass/ui/mac/MacPixels.java+8 17 modified
    @@ -24,7 +24,6 @@
      */
     package com.sun.glass.ui.mac;
     
    -import java.nio.Buffer;
     import java.nio.ByteBuffer;
     import java.nio.IntBuffer;
     
    @@ -69,29 +68,21 @@ protected MacPixels(int width, int height, IntBuffer data, float scalex, float s
         protected void _fillDirectByteBuffer(ByteBuffer bb) {
             if (this.bytes != null) {
                 this.bytes.rewind();
    -            if (this.bytes.isDirect() == true) {
    -                _copyPixels(bb, this.bytes, getWidth()*getHeight());
    -            } else {
    -                bb.put(this.bytes);
    -            }
    +            bb.put(this.bytes);
                 this.bytes.rewind();
             } else {
                 this.ints.rewind();
    -            if (this.ints.isDirect() == true) {
    -                _copyPixels(bb, this.ints, getWidth()*getHeight());
    -            } else {
    -                for (int i=0; i<this.ints.capacity(); i++) {
    -                    int data = this.ints.get();
    -                    bb.put((byte)((data>>0)&0xff));
    -                    bb.put((byte)((data>>8)&0xff));
    -                    bb.put((byte)((data>>16)&0xff));
    -                    bb.put((byte)((data>>24)&0xff));
    -                }
    +            for (int i=0; i<this.ints.capacity(); i++) {
    +                int data = this.ints.get();
    +                bb.put((byte)((data>>0)&0xff));
    +                bb.put((byte)((data>>8)&0xff));
    +                bb.put((byte)((data>>16)&0xff));
    +                bb.put((byte)((data>>24)&0xff));
                 }
                 this.ints.rewind();
             }
         }
    -    native protected void _copyPixels(Buffer src, Buffer dst, int size);
    +
         @Override native protected void _attachInt(long ptr, int w, int h, IntBuffer ints, int[] array, int offset);
         @Override native protected void _attachByte(long ptr, int w, int h, ByteBuffer bytes, byte[] array, int offset);
     
    
  • modules/javafx.graphics/src/main/java/com/sun/glass/ui/mac/MacView.java+3 3 modified
    @@ -99,9 +99,9 @@ static int getMultiClickMaxY_impl() {
                                     pixels.getWidth(), pixels.getHeight(), pixels.getScaleX(), pixels.getScaleY());
             }
         }
    -    native void _uploadPixelsDirect(long viewPtr, Buffer pixels, int width, int height, float scaleX, float scaleY);
    -    native void _uploadPixelsByteArray(long viewPtr, byte[] pixels, int offset, int width, int height, float scaleX, float scaleY);
    -    native void _uploadPixelsIntArray(long viewPtr, int[] pixels, int offset, int width, int height, float scaleX, float scaleY);
    +    private native void _uploadPixelsDirect(long viewPtr, Buffer pixels, int width, int height, float scaleX, float scaleY);
    +    private native void _uploadPixelsByteArray(long viewPtr, byte[] pixels, int offset, int width, int height, float scaleX, float scaleY);
    +    private native void _uploadPixelsIntArray(long viewPtr, int[] pixels, int offset, int width, int height, float scaleX, float scaleY);
     
         @Override
         protected void notifyResize(int width, int height) {
    
  • modules/javafx.graphics/src/main/java/com/sun/glass/ui/Pixels.java+19 24 modified
    @@ -85,25 +85,22 @@ public static int getNativeFormat() {
         private final float scaley;
     
         protected Pixels(final int width, final int height, final ByteBuffer pixels) {
    -        this.width = width;
    -        this.height = height;
    -        this.bytesPerComponent = 1;
    -        this.bytes = pixels.slice();
    -        if ((this.width <= 0) || (this.height <= 0) || ((this.width * this.height * 4) > this.bytes.capacity())) {
    -            throw new IllegalArgumentException("Too small byte buffer size "+this.width+"x"+this.height+" ["+(this.width*this.height*4)+"] > "+this.bytes.capacity());
    -        }
    -
    -        this.ints = null;
    -        this.scalex = 1.0f;
    -        this.scaley = 1.0f;
    +        this(width, height, pixels, 1.0f, 1.0f);
         }
     
         protected Pixels(final int width, final int height, final ByteBuffer pixels, float scalex, float scaley) {
             this.width = width;
             this.height = height;
             this.bytesPerComponent = 1;
             this.bytes = pixels.slice();
    -        if ((this.width <= 0) || (this.height <= 0) || ((this.width * this.height * 4) > this.bytes.capacity())) {
    +
    +        if (this.width <= 0 || this.height <= 0 ||
    +                this.width > ((Integer.MAX_VALUE / 4) / this.height)) {
    +
    +            throw new IllegalArgumentException("Invalid width*height");
    +        }
    +
    +        if ((this.width * this.height * 4) > this.bytes.capacity()) {
                 throw new IllegalArgumentException("Too small byte buffer size "+this.width+"x"+this.height+" ["+(this.width*this.height*4)+"] > "+this.bytes.capacity());
             }
     
    @@ -113,25 +110,22 @@ protected Pixels(final int width, final int height, final ByteBuffer pixels, flo
         }
     
         protected Pixels(final int width, final int height, IntBuffer pixels) {
    -        this.width = width;
    -        this.height = height;
    -        this.bytesPerComponent = 4;
    -        this.ints = pixels.slice();
    -        if ((this.width <= 0) || (this.height <= 0) || ((this.width * this.height) > this.ints.capacity())) {
    -            throw new IllegalArgumentException("Too small int buffer size "+this.width+"x"+this.height+" ["+(this.width*this.height)+"] > "+this.ints.capacity());
    -        }
    -
    -        this.bytes = null;
    -        this.scalex = 1.0f;
    -        this.scaley = 1.0f;
    +        this(width, height, pixels, 1.0f, 1.0f);
         }
     
         protected Pixels(final int width, final int height, IntBuffer pixels, float scalex, float scaley) {
             this.width = width;
             this.height = height;
             this.bytesPerComponent = 4;
             this.ints = pixels.slice();
    -        if ((this.width <= 0) || (this.height <= 0) || ((this.width * this.height) > this.ints.capacity())) {
    +
    +        if (this.width <= 0 || this.height <= 0 ||
    +                this.width > ((Integer.MAX_VALUE / 4) / this.height)) {
    +
    +            throw new IllegalArgumentException("Invalid width*height");
    +        }
    +
    +        if ((this.width * this.height) > this.ints.capacity()) {
                 throw new IllegalArgumentException("Too small int buffer size "+this.width+"x"+this.height+" ["+(this.width*this.height)+"] > "+this.ints.capacity());
             }
     
    @@ -238,6 +232,7 @@ public final void asByteBuffer(ByteBuffer bb) {
                 throw new RuntimeException("Too small buffer.");
             }
             _fillDirectByteBuffer(bb);
    +        bb.rewind();
         }
     
         // This method is called from the native code to reduce the number of JNI up-calls.
    
  • modules/javafx.graphics/src/main/native-font/directwrite.cpp+12 0 modified
    @@ -2167,10 +2167,14 @@ JNIEXPORT jbyteArray JNICALL OS_NATIVE(CreateAlphaTexture)
         /* In Only */
         if (arg2) lparg2 = getRECTFields(env, arg2, &_arg2);
         if (!lparg2) return NULL;
    +    if (lparg2->right <= lparg2->left) return NULL;
    +    if (lparg2->bottom <= lparg2->top) return NULL;
         DWRITE_TEXTURE_TYPE textureType = (DWRITE_TEXTURE_TYPE)arg1;
         UINT32 width = lparg2->right - lparg2->left;
         UINT32 height = lparg2->bottom - lparg2->top;
         UINT32 bpp = textureType == DWRITE_TEXTURE_CLEARTYPE_3x1 ? 3 : 1;
    +    if (height > UINT32_MAX / bpp) return NULL;
    +    if (height > 0 && width > UINT32_MAX / (height * bpp)) return NULL;
         UINT32 bufferSize = width * height * bpp;
         BYTE * buffer = new (std::nothrow) BYTE[bufferSize];
         HRESULT hr = ((IDWriteGlyphRunAnalysis *)arg0)->CreateAlphaTexture(textureType, lparg2, buffer, bufferSize);
    @@ -2233,6 +2237,10 @@ JNIEXPORT jint JNICALL OS_NATIVE(GetGlyphs)
         if (arg15) if ((lparg15 = env->GetShortArrayElements(arg15, NULL)) == NULL) goto fail;
         if (arg16) if ((lparg16 = env->GetShortArrayElements(arg16, NULL)) == NULL) goto fail;
         if (arg17) if ((lparg17 = env->GetIntArrayElements(arg17, NULL)) == NULL) goto fail;
    +    if (textStart < 0) goto fail;
    +    if (!arg1) goto fail;
    +    if (arg2 <= 0 || arg2 > env->GetArrayLength(arg1)) goto fail;
    +    if (textStart > env->GetArrayLength(arg1) - arg2) goto fail;
         const WCHAR* text = (const WCHAR*)(lparg1 + textStart);
     
         hr = ((IDWriteTextAnalyzer *)arg0)->GetGlyphs(text,
    @@ -2297,6 +2305,10 @@ JNIEXPORT jint JNICALL OS_NATIVE(GetGlyphPlacements)
         if (arg15) if ((lparg15 = env->GetIntArrayElements(arg15, NULL)) == NULL) goto fail;
         if (arg17) if ((lparg17 = env->GetFloatArrayElements(arg17, NULL)) == NULL) goto fail;
         if (arg18) if ((lparg18 = env->GetFloatArrayElements(arg18, NULL)) == NULL) goto fail;
    +    if (textStart < 0) goto fail;
    +    if (!arg1) goto fail;
    +    if (arg4 <= 0 || arg4 > env->GetArrayLength(arg1)) goto fail;
    +    if (textStart > env->GetArrayLength(arg1) - arg4) goto fail;
         const WCHAR* text = (const WCHAR*)(lparg1 + textStart);
     
         hr = ((IDWriteTextAnalyzer *)arg0)->GetGlyphPlacements(text,
    
  • modules/javafx.graphics/src/main/native-glass/gtk/GlassPixels.cpp+44 23 modified
    @@ -40,25 +40,6 @@ static void my_free(guchar *pixels, gpointer data) {
     
     extern "C" {
     
    -/*
    - * Class:     com_sun_glass_ui_gtk_GtkPixels
    - * Method:    _copyPixels
    - * Signature: (Ljava/nio/Buffer;Ljava/nio/Buffer;I)V
    - */
    -JNIEXPORT void JNICALL Java_com_sun_glass_ui_gtk_GtkPixels__1copyPixels
    -  (JNIEnv *env, jobject obj, jobject jDst, jobject jSrc, jint jSize)
    -{
    -    (void)obj;
    -
    -    //Taken from MacPixels (and fixed)
    -    void *src = env->GetDirectBufferAddress(jSrc);
    -    void *dst = env->GetDirectBufferAddress(jDst);
    -    if ((src != NULL) && (dst != NULL) && (jSize > 0))
    -    {
    -        memcpy(dst, src, jSize * 4);
    -    }
    -}
    -
     /*
      * Class:     com_sun_glass_ui_gtk_GtkPixels
      * Method:    _attachInt
    @@ -69,15 +50,35 @@ JNIEXPORT void JNICALL Java_com_sun_glass_ui_gtk_GtkPixels__1attachInt
     {
         (void)obj;
     
    +    if (!ptr) return;
    +    if (!array && !ints) return;
    +    if (offset < 0) return;
    +    if (w <= 0 || h <= 0) return;
    +
    +    if (w > (((INT_MAX - offset) / 4) / h))
    +    {
    +        return;
    +    }
    +
         jint *data;
         GdkPixbuf **pixbuf;
         guint8 *dataRGBA;
     
    +    jsize numElem;
    +    if (array == NULL) {
    +        numElem = env->GetDirectBufferCapacity(ints);
    +    } else {
    +        numElem = env->GetArrayLength(array);
    +    }
    +
    +    if ((w * h + offset) > numElem)
    +    {
    +        return;
    +    }
    +
         if (array == NULL) {
             data = (jint*) env->GetDirectBufferAddress(ints);
    -        assert((w*h*4 + offset * 4) == env->GetDirectBufferCapacity(ints));
         } else {
    -        assert((w*h + offset) == env->GetArrayLength(array));
             data = (jint*) env->GetPrimitiveArrayCritical(array, 0);
         }
     
    @@ -101,15 +102,35 @@ JNIEXPORT void JNICALL Java_com_sun_glass_ui_gtk_GtkPixels__1attachByte
     {
         (void)obj;
     
    +    if (!ptr) return;
    +    if (!array && !bytes) return;
    +    if (offset < 0) return;
    +    if (w <= 0 || h <= 0) return;
    +
    +    if (w > (((INT_MAX - offset) / 4) / h))
    +    {
    +        return;
    +    }
    +
         jbyte *data;
         GdkPixbuf **pixbuf;
         guint8 *dataRGBA;
     
    +    jsize numElem;
    +    if (array == NULL) {
    +        numElem = env->GetDirectBufferCapacity(bytes);
    +    } else {
    +        numElem = env->GetArrayLength(array);
    +    }
    +
    +    if ((w * h * 4 + offset) > numElem)
    +    {
    +        return;
    +    }
    +
         if (array == NULL) {
             data = (jbyte*) env->GetDirectBufferAddress(bytes);
    -        assert((w*h*4 + offset) == env->GetDirectBufferCapacity(bytes));
         } else {
    -        assert((w*h*4 + offset) == env->GetArrayLength(array));
             data = (jbyte*) env->GetPrimitiveArrayCritical(array, 0);
         }
     
    
  • modules/javafx.graphics/src/main/native-glass/gtk/GlassView.cpp+33 2 modified
    @@ -185,6 +185,9 @@ JNIEXPORT void JNICALL Java_com_sun_glass_ui_gtk_GtkView__1uploadPixelsDirect
     {
         (void)jView;
     
    +    if (!ptr) return;
    +    if (!buffer) return;
    +
         GlassView* view = JLONG_TO_GLASSVIEW(ptr);
         if (view->current_window) {
             void *data = env->GetDirectBufferAddress(buffer);
    @@ -203,10 +206,24 @@ JNIEXPORT void JNICALL Java_com_sun_glass_ui_gtk_GtkView__1uploadPixelsIntArray
     {
         (void)obj;
     
    +    if (!ptr) return;
    +    if (!array) return;
    +    if (offset < 0) return;
    +    if (width <= 0 || height <= 0) return;
    +
    +    if (width > ((INT_MAX - offset) / height))
    +    {
    +        return;
    +    }
    +
    +    if ((width * height + offset) > env->GetArrayLength(array))
    +    {
    +        return;
    +    }
    +
         GlassView* view = JLONG_TO_GLASSVIEW(ptr);
         if (view->current_window) {
             int *data = NULL;
    -        assert((width*height + offset) == env->GetArrayLength(array));
             data = (int*)env->GetPrimitiveArrayCritical(array, 0);
     
             view->current_window->paint(data + offset, width, height);
    @@ -225,11 +242,25 @@ JNIEXPORT void JNICALL Java_com_sun_glass_ui_gtk_GtkView__1uploadPixelsByteArray
     {
         (void)obj;
     
    +    if (!ptr) return;
    +    if (!array) return;
    +    if (offset < 0) return;
    +    if (width <= 0 || height <= 0) return;
    +
    +    if (width > (((INT_MAX - offset) / 4) / height))
    +    {
    +        return;
    +    }
    +
    +    if ((4 * width * height + offset) > env->GetArrayLength(array))
    +    {
    +        return;
    +    }
    +
         GlassView* view = JLONG_TO_GLASSVIEW(ptr);
         if (view->current_window) {
             unsigned char *data = NULL;
     
    -        assert((4*width*height + offset) == env->GetArrayLength(array));
             data = (unsigned char*)env->GetPrimitiveArrayCritical(array, 0);
     
             view->current_window->paint(data + offset, width, height);
    
  • modules/javafx.graphics/src/main/native-glass/mac/GlassPixels.m+75 42 modified
    @@ -56,40 +56,6 @@
         return com_sun_glass_ui_Pixels_Format_BYTE_BGRA_PRE;
     }
     
    -/*
    - * Class:     com_sun_glass_ui_mac_MacPixels
    - * Method:    _copyPixels
    - * Signature: (Ljava/nio/Buffer;Ljava/nio/Buffer;I)V
    - */
    -JNIEXPORT void JNICALL Java_com_sun_glass_ui_mac_MacPixels__1copyPixels
    -(JNIEnv *env, jobject jPixels, jobject jSrc, jobject jDst, jint jSize)
    -{
    -    LOG("Java_com_sun_glass_ui_mac_MacPixels__1copyPixels");
    -
    -    GLASS_ASSERT_MAIN_JAVA_THREAD(env);
    -
    -    void *src = (*env)->GetDirectBufferAddress(env, jSrc);
    -    void *dst = (*env)->GetDirectBufferAddress(env, jDst);
    -    if ((src != NULL) && (src != NULL) && (jSize > 0))
    -    {
    -        memcpy(src, dst, jSize);
    -    }
    -    GLASS_CHECK_EXCEPTION(env);
    -}
    -
    -/*
    - * Class:     com_sun_glass_ui_mac_MacPixels
    - * Method:    _attachInt
    - * Signature: (JIILjava/nio/IntBuffer;[II)V
    - */
    -JNIEXPORT void JNICALL Java_com_sun_glass_ui_mac_MacPixels__1attachInt
    -(JNIEnv *env, jobject jPixels, jlong jPtr, jint jWidth, jint jHeight, jobject jBuffer, jintArray jArray, jint jOffset)
    -{
    -    LOG("Java_com_sun_glass_ui_mac_MacPixels__1attachInt");
    -
    -    Java_com_sun_glass_ui_mac_MacPixels__1attachByte(env, jPixels, jPtr, jWidth, jHeight, jBuffer, jArray, 4*jOffset);
    -}
    -
     NSImage* getImage(u_int8_t* data, int jWidth, int jHeight, int jOffset) {
         NSImage* image = NULL;
         CGImageRef cgImage = NULL;
    @@ -119,16 +85,9 @@
         return image;
     }
     
    -/*
    - * Class:     com_sun_glass_ui_mac_MacPixels
    - * Method:    _attachByte
    - * Signature: (JIILjava/nio/ByteBuffer;[BI)V
    - */
    -JNIEXPORT void JNICALL Java_com_sun_glass_ui_mac_MacPixels__1attachByte
    +void attachCommon
     (JNIEnv *env, jobject jPixels, jlong jPtr, jint jWidth, jint jHeight, jobject jBuffer, jbyteArray jArray, jint jOffset)
     {
    -    LOG("Java_com_sun_glass_ui_mac_MacPixels__1attachByte");
    -
         GLASS_ASSERT_MAIN_JAVA_THREAD(env);
         {
             u_int8_t *data = NULL;
    @@ -152,3 +111,77 @@
         }
         GLASS_CHECK_EXCEPTION(env);
     }
    +
    +/*
    + * Class:     com_sun_glass_ui_mac_MacPixels
    + * Method:    _attachInt
    + * Signature: (JIILjava/nio/IntBuffer;[II)V
    + */
    +JNIEXPORT void JNICALL Java_com_sun_glass_ui_mac_MacPixels__1attachInt
    +(JNIEnv *env, jobject jPixels, jlong jPtr, jint jWidth, jint jHeight, jobject jBuffer, jintArray jArray, jint jOffset)
    +{
    +    LOG("Java_com_sun_glass_ui_mac_MacPixels__1attachInt");
    +
    +    if (!jPtr) return;
    +    if (!(jArray || jBuffer)) return;
    +    if (jOffset < 0) return;
    +    if (jWidth <= 0 || jHeight <= 0) return;
    +
    +    if (jOffset > (INT_MAX / 4)) {
    +        return;
    +    }
    +
    +    if (jWidth > (((INT_MAX - 4 * jOffset) / 4) / jHeight))
    +    {
    +        return;
    +    }
    +
    +    jsize numElem;
    +    if (jArray != NULL) {
    +        numElem = (*env)->GetArrayLength(env, jArray);
    +    } else {
    +        numElem = (*env)->GetDirectBufferCapacity(env, jBuffer);
    +    }
    +
    +    if ((jWidth * jHeight + jOffset) > numElem)
    +    {
    +        return;
    +    }
    +
    +    attachCommon(env, jPixels, jPtr, jWidth, jHeight, jBuffer, jArray, 4 * jOffset);
    +}
    +
    +/*
    + * Class:     com_sun_glass_ui_mac_MacPixels
    + * Method:    _attachByte
    + * Signature: (JIILjava/nio/ByteBuffer;[BI)V
    + */
    +JNIEXPORT void JNICALL Java_com_sun_glass_ui_mac_MacPixels__1attachByte
    +(JNIEnv *env, jobject jPixels, jlong jPtr, jint jWidth, jint jHeight, jobject jBuffer, jbyteArray jArray, jint jOffset)
    +{
    +    LOG("Java_com_sun_glass_ui_mac_MacPixels__1attachByte");
    +
    +    if (!jPtr) return;
    +    if (!(jArray || jBuffer)) return;
    +    if (jOffset < 0) return;
    +    if (jWidth <= 0 || jHeight <= 0) return;
    +
    +    if (jWidth > (((INT_MAX - jOffset) / 4) / jHeight))
    +    {
    +        return;
    +    }
    +
    +    jsize numElem;
    +    if (jArray != NULL) {
    +        numElem = (*env)->GetArrayLength(env, jArray);
    +    } else {
    +        numElem = (*env)->GetDirectBufferCapacity(env, jBuffer);
    +    }
    +
    +    if ((4 * jWidth * jHeight + jOffset) > numElem)
    +    {
    +        return;
    +    }
    +
    +    attachCommon(env, jPixels, jPtr, jWidth, jHeight, jBuffer, jArray, jOffset);
    +}
    
  • modules/javafx.graphics/src/main/native-glass/mac/GlassView.m+29 41 modified
    @@ -43,23 +43,6 @@
         #define LOG(MSG, ...) GLASS_LOG(MSG, ## __VA_ARGS__);
     #endif
     
    -//#define FORCE_NOISE
    -#ifdef FORCE_NOISE
    -static inline void *_GenerateNoise(int width, int height)
    -{
    -    static int *pixels = NULL;
    -    pixels = realloc(pixels, width*height*4);
    -
    -    int *src = pixels;
    -    for (int i=0; i<width*height; i++)
    -    {
    -        *src++ = random();
    -    }
    -
    -    return (void*)pixels;
    -}
    -#endif
    -
     static inline NSView<GlassView>* getGlassView(JNIEnv *env, jlong jPtr)
     {
         assert(jPtr != 0L);
    @@ -613,15 +596,12 @@
     {
         LOG("Java_com_sun_glass_ui_mac_MacView__1uploadPixelsDirect");
         if (!jPtr) return;
    +    if (!jBuffer) return;
     
         GLASS_ASSERT_MAIN_JAVA_THREAD(env);
         NSView<GlassView> *view = getGlassView(env, jPtr);
     
    -#ifndef FORCE_NOISE
         void *pixels = (*env)->GetDirectBufferAddress(env, jBuffer);
    -#else
    -    void *pixels = _GenerateNoise(jWidth, jHeight);
    -#endif
     
         // must be in the middle of begin/end
         if ((jWidth > 0) && (jHeight > 0))
    @@ -640,27 +620,31 @@
     {
         LOG("Java_com_sun_glass_ui_mac_MacView__1uploadPixelsByteArray");
         if (!jPtr) return;
    +    if (!jArray) return;
    +    if (jOffset < 0) return;
    +    if (jWidth <= 0 || jHeight <= 0) return;
    +
    +    if (jWidth > (((INT_MAX - jOffset) / 4) / jHeight))
    +    {
    +        return;
    +    }
     
         GLASS_ASSERT_MAIN_JAVA_THREAD(env);
     
    +    if ((4 * jWidth * jHeight + jOffset) > (*env)->GetArrayLength(env, jArray))
    +    {
    +        return;
    +    }
    +
         jboolean isCopy = JNI_FALSE;
         u_int8_t *data = (*env)->GetPrimitiveArrayCritical(env, jArray, &isCopy);
         {
    -        assert((4*jWidth*jHeight + jOffset) == (*env)->GetArrayLength(env, jArray));
    -
             NSView<GlassView> *view = getGlassView(env, jPtr);
     
    -#ifndef FORCE_NOISE
             void *pixels = (data+jOffset);
    -#else
    -        void *pixels = _GenerateNoise(jWidth, jHeight);
    -#endif
     
             // must be in the middle of begin/end
    -        if ((jWidth > 0) && (jHeight > 0))
    -        {
    -            [view pushPixels:pixels withWidth:(GLuint)jWidth withHeight:(GLuint)jHeight withScaleX:(GLfloat)jScaleX withScaleY:(GLfloat)jScaleY withEnv:env];
    -        }
    +        [view pushPixels:pixels withWidth:(GLuint)jWidth withHeight:(GLuint)jHeight withScaleX:(GLfloat)jScaleX withScaleY:(GLfloat)jScaleY withEnv:env];
         }
         (*env)->ReleasePrimitiveArrayCritical(env, jArray, data, JNI_ABORT);
     }
    @@ -675,27 +659,31 @@
     {
         LOG("Java_com_sun_glass_ui_mac_MacView__1uploadPixelsIntArray");
         if (!jPtr) return;
    +    if (!jArray) return;
    +    if (jOffset < 0) return;
    +    if (jWidth <= 0 || jHeight <= 0) return;
    +
    +    if (jWidth > ((INT_MAX - jOffset) / jHeight))
    +    {
    +        return;
    +    }
     
         GLASS_ASSERT_MAIN_JAVA_THREAD(env);
     
    +    if ((jWidth * jHeight + jOffset) > (*env)->GetArrayLength(env, jArray))
    +    {
    +        return;
    +    }
    +
         jboolean isCopy = JNI_FALSE;
         u_int32_t *data = (*env)->GetPrimitiveArrayCritical(env, jArray, &isCopy);
         {
    -        assert((jWidth*jHeight + jOffset) == (*env)->GetArrayLength(env, jArray));
    -
             NSView<GlassView> *view = getGlassView(env, jPtr);
     
    -#ifndef FORCE_NOISE
             void *pixels = (data+jOffset);
    -#else
    -        void *pixels = _GenerateNoise(jWidth, jHeight);
    -#endif
     
             // must be in the middle of begin/end
    -        if ((jWidth > 0) && (jHeight > 0))
    -        {
    -            [view pushPixels:pixels withWidth:(GLuint)jWidth withHeight:(GLuint)jHeight withScaleX:(GLfloat)jScaleX withScaleY:(GLfloat)jScaleY withEnv:env];
    -        }
    +        [view pushPixels:pixels withWidth:(GLuint)jWidth withHeight:(GLuint)jHeight withScaleX:(GLfloat)jScaleX withScaleY:(GLfloat)jScaleY withEnv:env];
         }
         (*env)->ReleasePrimitiveArrayCritical(env, jArray, data, JNI_ABORT);
     }
    
  • modules/javafx.graphics/src/main/native-glass/win/Pixels.cpp+20 3 modified
    @@ -248,9 +248,26 @@ JNIEXPORT void JNICALL Java_com_sun_glass_ui_win_WinPixels__1fillDirectByteBuffe
     {
         Pixels pixels(env, jPixels);
     
    -    memcpy(env->GetDirectBufferAddress(bb), pixels.GetBits(),
    -            pixels.GetWidth() * pixels.GetHeight() * 4);
    +    if (bb == NULL) {
    +        return;
    +    }
    +
    +    const int width = pixels.GetWidth();
    +    const int height = pixels.GetHeight();
    +    if (width <= 0 || height <= 0 || width > ((INT_MAX / 4) / height)) {
    +        return;
    +    }
    +    const int size = width * height * 4;
    +    const int bbCapacity = env->GetDirectBufferCapacity(bb);
    +    if (bbCapacity < size) {
    +        return;
    +    }
    +
    +    void *bbAddr = env->GetDirectBufferAddress(bb);
    +    if (bbAddr == NULL) {
    +        return;
    +    }
    +    memcpy(bbAddr, pixels.GetBits(), size);
     }
     
     } // extern "C"
    -
    
  • modules/javafx.media/src/main/native/jfxmedia/jni/NativeVideoBuffer.cpp+3 3 modified
    @@ -71,8 +71,8 @@ JNIEXPORT jobject JNICALL Java_com_sun_media_jfxmediaimpl_NativeVideoBuffer_nati
     {
         CVideoFrame *frame = (CVideoFrame*)jlong_to_ptr(nativeHandle);
         if (frame) {
    -        void *dataPtr = frame->GetDataForPlane((int)plane);
    -        jlong capacity = (jlong)frame->GetSizeForPlane((int)plane);
    +        void *dataPtr = frame->GetDataForPlane((unsigned int)plane);
    +        jlong capacity = (jlong)frame->GetSizeForPlane((unsigned int)plane);
             return env->NewDirectByteBuffer(dataPtr, capacity);
         }
         return NULL;
    @@ -203,7 +203,7 @@ JNIEXPORT jintArray JNICALL Java_com_sun_media_jfxmediaimpl_NativeVideoBuffer_na
             jintArray strides = env->NewIntArray(count);
             jint *strideArray = new jint[count];
     
    -        for (int ii=0; ii < count; ii++) {
    +        for (unsigned int ii=0; ii < count; ii++) {
                 strideArray[ii] = frame->GetStrideForPlane(ii);
             }
     
    
  • modules/javafx.media/src/main/native/jfxmedia/PipelineManagement/VideoFrame.cpp+103 32 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -23,50 +23,51 @@
      * questions.
      */
     
    +// For UINT_MAX, we cannot use GLib here, since it is shared code between
    +// GStreamer and AVFoundation.
    +#include <limits.h>
    +
     #include "VideoFrame.h"
     #include <Common/VSMemory.h>
     
     //*************************************************************************************************
     //********** class CVideoFrame
     //*************************************************************************************************
     CVideoFrame::CVideoFrame()
    -:   m_iWidth(0),
    -    m_iHeight(0),
    -    m_iEncodedWidth(0),
    -    m_iEncodedHeight(0),
    +:   m_uiWidth(0),
    +    m_uiHeight(0),
    +    m_uiEncodedWidth(0),
    +    m_uiEncodedHeight(0),
         m_typeFrame(UNKNOWN),
         m_bHasAlpha(false),
         m_dTime(0.0),
    -    m_FrameDirty(false),
    -    m_iPlaneCount(1)
    +    m_FrameDirty(false)
     {
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    +    Reset();
     }
     
     CVideoFrame::~CVideoFrame()
     {
     }
     
    -int CVideoFrame::GetWidth()
    +unsigned int CVideoFrame::GetWidth()
     {
    -    return m_iWidth;
    +    return m_uiWidth;
     }
     
    -int CVideoFrame::GetHeight()
    +unsigned int CVideoFrame::GetHeight()
     {
    -    return m_iHeight;
    +    return m_uiHeight;
     }
     
    -int CVideoFrame::GetEncodedWidth()
    +unsigned int CVideoFrame::GetEncodedWidth()
     {
    -    return m_iEncodedWidth;
    +    return m_uiEncodedWidth;
     }
     
    -int CVideoFrame::GetEncodedHeight()
    +unsigned int CVideoFrame::GetEncodedHeight()
     {
    -    return m_iEncodedHeight;
    +    return m_uiEncodedHeight;
     }
     
     CVideoFrame::FrameType CVideoFrame::GetType()
    @@ -84,31 +85,41 @@ double CVideoFrame::GetTime()
         return m_dTime;
     }
     
    -int CVideoFrame::GetPlaneCount()
    +unsigned int CVideoFrame::GetPlaneCount()
     {
    -    return m_iPlaneCount;
    +    return m_uiPlaneCount;
    +}
    +
    +void CVideoFrame::SetPlaneCount(unsigned int count)
    +{
    +    if (count <= MAX_PLANE_COUNT) {
    +        m_uiPlaneCount = count;
    +    } else {
    +        // Should never happen
    +        m_uiPlaneCount = MAX_PLANE_COUNT;
    +    }
     }
     
    -void* CVideoFrame::GetDataForPlane(int planeIndex)
    +void* CVideoFrame::GetDataForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    +    if (planeIndex < MAX_PLANE_COUNT) {
             return m_pvPlaneData[planeIndex];
         }
         return NULL;
     }
     
    -unsigned long CVideoFrame::GetSizeForPlane(int planeIndex)
    +unsigned long CVideoFrame::GetSizeForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    +    if (planeIndex < MAX_PLANE_COUNT) {
             return m_pulPlaneSize[planeIndex];
         }
         return 0;
     }
     
    -int CVideoFrame::GetStrideForPlane(int planeIndex)
    +unsigned int CVideoFrame::GetStrideForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    -        return m_piPlaneStrides[planeIndex];
    +    if (planeIndex < MAX_PLANE_COUNT) {
    +        return m_puiPlaneStrides[planeIndex];
         }
         return 0;
     }
    @@ -118,12 +129,22 @@ CVideoFrame *CVideoFrame::ConvertToFormat(FrameType type)
         return NULL;
     }
     
    -void CVideoFrame::SwapPlanes(int aa, int bb)
    +void CVideoFrame::Reset()
    +{
    +    m_uiPlaneCount = 0;
    +    for (int i = 0; i < MAX_PLANE_COUNT; i++) {
    +        m_puiPlaneStrides[i] = 0;
    +        m_pulPlaneSize[i] = 0;
    +        m_pvPlaneData[i] = NULL;
    +    }
    +}
    +
    +void CVideoFrame::SwapPlanes(unsigned int aa, unsigned int bb)
     {
    -    if (aa != bb && aa >= 0 && aa < m_iPlaneCount && bb >= 0 && bb < m_iPlaneCount) {
    -        int stride = m_piPlaneStrides[aa];
    -        m_piPlaneStrides[aa] = m_piPlaneStrides[bb];
    -        m_piPlaneStrides[bb] = stride;
    +    if (aa != bb && aa < m_uiPlaneCount && bb < m_uiPlaneCount) {
    +        unsigned int stride = m_puiPlaneStrides[aa];
    +        m_puiPlaneStrides[aa] = m_puiPlaneStrides[bb];
    +        m_puiPlaneStrides[bb] = stride;
     
             unsigned long size = m_pulPlaneSize[aa];
             m_pulPlaneSize[aa] = m_pulPlaneSize[bb];
    @@ -134,3 +155,53 @@ void CVideoFrame::SwapPlanes(int aa, int bb)
             m_pvPlaneData[bb] = vptr;
         }
     }
    +
    +unsigned long CVideoFrame::CalcSize(unsigned int a, unsigned int b, bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return 0;
    +    }
    +
    +    if (b > 0 && a <= (UINT_MAX / b)) {
    +        return (a * b);
    +    }
    +
    +    *(pbValid) = false;
    +    return 0;
    +}
    +
    +unsigned long CVideoFrame::AddSize(unsigned long a, unsigned long b, bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return 0;
    +    }
    +
    +    // unsigned long can be 32-bit or 64-bit, make sure it is no more then UINT_MAX
    +    if (a <= UINT_MAX && b <= UINT_MAX && a <= (UINT_MAX - b)) {
    +        return (a + b);
    +    }
    +
    +    *(pbValid) = false;
    +    return 0;
    +}
    +
    +void* CVideoFrame::CalcPlanePointer(intptr_t baseAddress, unsigned int offset,
    +                                    unsigned long planeSize, unsigned long baseSize,
    +                                    bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return NULL;
    +    }
    +
    +    // We will read planeSize bytes from baseAddress starting with offset, so
    +    // make sure we do not read pass baseSize.
    +    unsigned long endOfPlane = AddSize(offset, planeSize, pbValid);
    +    if (*(pbValid)) { // Make sure AddSize() did not failed.
    +        if (endOfPlane <= baseSize) {
    +            return (void*)(baseAddress + offset);
    +        }
    +    }
    +
    +    *(pbValid) = false;
    +    return NULL;
    +}
    
  • modules/javafx.media/src/main/native/jfxmedia/PipelineManagement/VideoFrame.h+41 18 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -27,6 +27,9 @@
     #define _VIDEO_FRAME_H_
     
     #include <stdlib.h>
    +#include <stdint.h>
    +
    +#define MAX_PLANE_COUNT 4
     
     /**
      * class CVideoFrame
    @@ -56,41 +59,61 @@ class CVideoFrame
     
         double              GetTime();
     
    -    int                 GetWidth();
    -    int                 GetHeight();
    -    int                 GetEncodedWidth();
    -    int                 GetEncodedHeight();
    +    unsigned int        GetWidth();
    +    unsigned int        GetHeight();
    +    unsigned int        GetEncodedWidth();
    +    unsigned int        GetEncodedHeight();
     
         FrameType           GetType();
         bool                HasAlpha();
     
    -    int                 GetPlaneCount();
    -    void*               GetDataForPlane(int planeIndex);
    -    unsigned long       GetSizeForPlane(int planeIndex);
    -    int                 GetStrideForPlane(int planeIndex);
    +    unsigned int        GetPlaneCount();
    +    void                SetPlaneCount(unsigned int count);
    +    void*               GetDataForPlane(unsigned int planeIndex);
    +    unsigned long       GetSizeForPlane(unsigned int planeIndex);
    +    unsigned int        GetStrideForPlane(unsigned int planeIndex);
     
         virtual CVideoFrame *ConvertToFormat(FrameType type);
     
         bool                GetFrameDirty() { return m_FrameDirty; }
         void                SetFrameDirty(bool dirty) { m_FrameDirty = dirty; }
     
     protected:
    -    int                 m_iWidth;
    -    int                 m_iHeight;
    -    int                 m_iEncodedWidth;
    -    int                 m_iEncodedHeight;
    +    unsigned int        m_uiWidth;
    +    unsigned int        m_uiHeight;
    +    unsigned int        m_uiEncodedWidth;
    +    unsigned int        m_uiEncodedHeight;
         FrameType           m_typeFrame;
         bool                m_bHasAlpha;
         double              m_dTime;
         bool                m_FrameDirty;
     
         // frame data buffers
    -    int                 m_iPlaneCount;
    -    void*               m_pvPlaneData[4];
    -    unsigned long       m_pulPlaneSize[4];
    -    int                 m_piPlaneStrides[4];
    +    void*               m_pvPlaneData[MAX_PLANE_COUNT];
    +    unsigned long       m_pulPlaneSize[MAX_PLANE_COUNT];
    +    unsigned int        m_puiPlaneStrides[MAX_PLANE_COUNT];
    +
    +    void Reset();
    +    void SwapPlanes(unsigned int aa, unsigned int bb);
    +
    +    // CalcSize(), AddSize(), CalcPlanePointer() requires bValid to be set to
    +    // true initially, if bValid is false these functions do nothing. It is
    +    // implemented this way, so all these functions can be chain called without
    +    // checking bValid after each call. bValid will be set to false only if
    +    // calculation failed and will never be set to true.
    +    // Multiplies a and b, bValid set to false if integer overflow detected.
    +    unsigned long CalcSize(unsigned int a, unsigned int b, bool *pbValid);
    +    // Adds a and b, bValid set to false if integer overflow detected.
    +    unsigned long AddSize(unsigned long a, unsigned long b, bool *pbValid);
    +    // Calculates plane pointer (baseAddress + offset) and checks that calculated
    +    // pointer within buffer. Returns NULL and sets bValid to false if calculated
    +    // pointer is invalid.
    +    void* CalcPlanePointer(intptr_t baseAddress, unsigned int offset,
    +                           unsigned long planeSize, unsigned long baseSize,
    +                           bool *pbValid);
     
    -    void SwapPlanes(int aa, int bb);
    +private:
    +    unsigned int        m_uiPlaneCount;
     };
     
     #endif  //_VIDEO_FRAME_H_
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/gstreamer/GstVideoFrame.cpp+144 94 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -56,18 +56,23 @@ static GstBuffer *alloc_aligned_buffer(guint size)
     
         // allocate a buffer large enough to accommodate 16 byte alignment
         alignedSize = size;
    -    size += 16;
    +    if (size <= (G_MAXUINT - 16)) {
    +        size += 16;
    +    } else {
    +        return NULL;
    +    }
    +
         newData = (guint8*)g_try_malloc(size);
         if (NULL == newData) {
             return NULL;
         }
     
         alignedData = (guint8*)(((intptr_t)newData + 15) & ~15);
     
    -    return gst_buffer_new_wrapped_full((GstMemoryFlags)0, alignedData, alignedSize, 0, 0, newData, free_aligned_buffer);
    +    return gst_buffer_new_wrapped_full((GstMemoryFlags)0, alignedData, alignedSize, 0, alignedSize, newData, free_aligned_buffer);
     }
     
    -GstCaps *create_RGB_caps(CVideoFrame::FrameType type, gint width, gint height, gint encodedWidth, gint encodedHeight, gint stride)
    +GstCaps *create_RGB_caps(CVideoFrame::FrameType type, guint width, guint height, guint encodedWidth, guint encodedHeight, guint stride)
     {
         gint red_mask, green_mask, blue_mask, alpha_mask;
         GstCaps *newCaps;
    @@ -160,7 +165,8 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
         const GstStructure* str = gst_caps_get_structure(newCaps, 0);
         const gchar* sFormatFourCC = gst_structure_get_string(str, "format");
     
    -    // default to success
    +    // We should always start with success. See CalcSize(), AddSize() and
    +    // CalcPlanePointer() on how this flag is being used.
         m_bIsValid = true;
     
         // FIXME: make format type strings conformant with constant types
    @@ -209,83 +215,79 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
             m_bIsValid = false;
         }
     
    -    if(!gst_structure_get_int(str, "width", &m_iWidth))
    +    if (!gst_structure_get_int(str, "width", (int*)&m_uiWidth))
         {
     #if JFXMEDIA_DEBUG
             g_warning("width could not be retrieved from GstBuffer\n");
     #endif
    -        m_iWidth = 0;
    +        m_uiWidth = 0;
             m_bIsValid = false;
         }
    -    if(!gst_structure_get_int(str, "height", &m_iHeight))
    +    if (!gst_structure_get_int(str, "height", (int*)&m_uiHeight))
         {
     #if JFXMEDIA_DEBUG
             g_warning("height could not be retrieved from GstBuffer\n");
     #endif
    -        m_iHeight = 0;
    +        m_uiHeight = 0;
             m_bIsValid = false;
         }
     
    -    if (!gst_structure_get_int(str, "encoded-width", &m_iEncodedWidth)) {
    -        m_iEncodedWidth = m_iWidth;
    +    if (!gst_structure_get_int(str, "encoded-width", (int*)&m_uiEncodedWidth)) {
    +        m_uiEncodedWidth = m_uiWidth;
         }
    -    if (!gst_structure_get_int(str, "encoded-height", &m_iEncodedHeight)) {
    -        m_iEncodedHeight = m_iHeight;
    +    if (!gst_structure_get_int(str, "encoded-height", (int*)&m_uiEncodedHeight)) {
    +        m_uiEncodedHeight = m_uiHeight;
         }
     
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    +    Reset();
     
    -    unsigned long expectedSize = 0;
         switch (m_typeFrame) {
             case YCbCr_420p: {
    -            int offset;
    -            m_iPlaneCount = 3;
    +            unsigned int offset = 0;
    +            SetPlaneCount(3);
     
    -            if (!gst_structure_get_int(str, "stride-y", &m_piPlaneStrides[0])) {
    -                m_piPlaneStrides[0] = m_iEncodedWidth;
    +            if (!gst_structure_get_int(str, "stride-y", (int*)&m_puiPlaneStrides[0])) {
    +                m_puiPlaneStrides[0] = m_uiEncodedWidth;
                 }
    -            if (!gst_structure_get_int(str, "stride-v", &m_piPlaneStrides[1])) {
    -                m_piPlaneStrides[1] = m_iEncodedWidth/2;
    +            if (!gst_structure_get_int(str, "stride-v", (int*)&m_puiPlaneStrides[1])) {
    +                m_puiPlaneStrides[1] = m_uiEncodedWidth/2;
                 }
    -            if (!gst_structure_get_int(str, "stride-u", &m_piPlaneStrides[2])) {
    -                m_piPlaneStrides[2] = m_piPlaneStrides[1];
    +            if (!gst_structure_get_int(str, "stride-u", (int*)&m_puiPlaneStrides[2])) {
    +                m_puiPlaneStrides[2] = m_puiPlaneStrides[1];
                 }
     
    -            offset = 0;
    -            gst_structure_get_int(str, "offset-y", &offset);
    -            m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    -            m_pvPlaneData[0] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[0];
    +            gst_structure_get_int(str, "offset-y", (int*)&offset);
    +            m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &m_bIsValid);
    +            m_pvPlaneData[0] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[0], m_ulBufferSize, &m_bIsValid);
     
                 //
                 // Chroma offsets assume YV12 ordering
                 //
                 offset += m_pulPlaneSize[0];
    -            gst_structure_get_int(str, "offset-v", &offset);
    -            m_pulPlaneSize[1] = m_piPlaneStrides[1] * (m_iEncodedHeight/2);
    -            m_pvPlaneData[1] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[1];
    +            gst_structure_get_int(str, "offset-v", (int*)&offset);
    +            m_pulPlaneSize[1] = CalcSize(m_puiPlaneStrides[1], (m_uiEncodedHeight/2), &m_bIsValid);
    +            m_pvPlaneData[1] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[1], m_ulBufferSize, &m_bIsValid);
     
                 offset += m_pulPlaneSize[1];
    -            gst_structure_get_int(str, "offset-u", &offset);
    -            m_pulPlaneSize[2] = m_piPlaneStrides[2] * (m_iEncodedHeight/2);
    -            m_pvPlaneData[2] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[2];
    +            gst_structure_get_int(str, "offset-u", (int*)&offset);
    +            m_pulPlaneSize[2] = CalcSize(m_puiPlaneStrides[2], (m_uiEncodedHeight/2), &m_bIsValid);
    +            m_pvPlaneData[2] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[2], m_ulBufferSize, &m_bIsValid);
     
                 // process alpha channel (before we potentially swap Cb/Cr)
                 if (m_bHasAlpha) {
    -                m_iPlaneCount++;
    -                if (!gst_structure_get_int(str, "stride-a", &m_piPlaneStrides[3])) {
    -                    m_piPlaneStrides[3] = m_piPlaneStrides[0];
    +                SetPlaneCount(GetPlaneCount() + 1);
    +                if (!gst_structure_get_int(str, "stride-a", (int*)&m_puiPlaneStrides[3])) {
    +                    m_puiPlaneStrides[3] = m_puiPlaneStrides[0];
                     }
     
                     offset += m_pulPlaneSize[2];
    -                gst_structure_get_int(str, "offset-a", &offset);
    -                m_pulPlaneSize[3] = m_piPlaneStrides[3] * m_iEncodedHeight;
    -                m_pvPlaneData[3] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -                expectedSize += m_pulPlaneSize[3];
    +                gst_structure_get_int(str, "offset-a", (int*)&offset);
    +                m_pulPlaneSize[3] = CalcSize(m_puiPlaneStrides[3], m_uiEncodedHeight, &m_bIsValid);
    +                m_pvPlaneData[3] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[3], m_ulBufferSize, &m_bIsValid);
                 }
     
                 //
    @@ -299,21 +301,19 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
             }
     
             default:
    -            m_iPlaneCount = 1;
    -            if (!gst_structure_get_int(str, "line_stride", &m_piPlaneStrides[0])) {
    +            SetPlaneCount(1);
    +            if (!gst_structure_get_int(str, "line_stride", (int*)&m_puiPlaneStrides[0])) {
                     if (m_typeFrame == YCbCr_422) {
    -                    m_piPlaneStrides[0] = m_iEncodedWidth * 2; // 16 bpp
    +                    m_puiPlaneStrides[0] = m_uiEncodedWidth * 2; // 16 bpp
                     } else {
    -                    m_piPlaneStrides[0] = m_iEncodedWidth * 4; // 32 bpp
    +                    m_puiPlaneStrides[0] = m_uiEncodedWidth * 4; // 32 bpp
                     }
                 }
    -            m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    -            m_pvPlaneData[0] = m_pvBufferBaseAddress;
    -            expectedSize += m_pulPlaneSize[0];
    +            m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &m_bIsValid);
    +            m_pvPlaneData[0] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, 0,
    +                                                m_pulPlaneSize[0], m_ulBufferSize, &m_bIsValid);
                 break;
         }
    -
    -    m_bIsValid = m_bIsValid && (expectedSize <= m_ulBufferSize);
     }
     
     bool CGstVideoFrame::IsValid()
    @@ -381,9 +381,10 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
         GstBuffer *destBuffer = NULL;
         GstCaps *destCaps = NULL;
         GstMapInfo info;
    -    gint stride = m_iEncodedWidth * 4;
    -    int u_index, v_index;
    -    int status;
    +    guint stride = 0;
    +    guint alloc_size = 0;
    +    unsigned int u_index, v_index = 0;
    +    int status = 0;
     
         if (m_bIsI420) {
             u_index = 1;
    @@ -393,8 +394,26 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
             v_index = 1;
         }
     
    -    stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    -    destBuffer = alloc_aligned_buffer(stride * m_iEncodedHeight);
    +    // Make sure we do not have an integer overflow
    +    if (m_uiEncodedWidth <= (G_MAXUINT / 4)) {
    +        stride = m_uiEncodedWidth * 4;
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (stride <= (G_MAXUINT - 16)) {
    +        stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (m_uiEncodedHeight > 0 && stride <= (G_MAXUINT / m_uiEncodedHeight)) {
    +        alloc_size = stride * m_uiEncodedHeight;
    +    } else {
    +        return NULL;
    +    }
    +
    +    destBuffer = alloc_aligned_buffer(alloc_size);
         if (!destBuffer) {
             return NULL;
         }
    @@ -415,49 +434,49 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
             if (m_bHasAlpha) {
                 status = ColorConvert_YCbCr420p_to_ARGB32(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
                             (const uint8_t*)m_pvPlaneData[3],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index], m_piPlaneStrides[3]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index], m_puiPlaneStrides[3]);
             } else {
                 status = ColorConvert_YCbCr420p_to_ARGB32_no_alpha(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index]);
             }
         } else {
             if (m_bHasAlpha) {
                 status = ColorConvert_YCbCr420p_to_BGRA32(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
                             (const uint8_t*)m_pvPlaneData[3],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index], m_piPlaneStrides[3]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index], m_puiPlaneStrides[3]);
             } else {
                 status = ColorConvert_YCbCr420p_to_BGRA32_no_alpha(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index]);
             }
         }
     
         gst_buffer_unmap(destBuffer, &info);
     
    -    destCaps = create_RGB_caps(destType, m_iWidth, m_iHeight, m_iEncodedWidth, m_iEncodedHeight, stride);
    +    destCaps = create_RGB_caps(destType, m_uiWidth, m_uiHeight, m_uiEncodedWidth, m_uiEncodedHeight, stride);
         if (!destCaps) {
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer);
    @@ -476,12 +495,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
     
         if (0 == status && destSample) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_sample_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
     
         return NULL;
    @@ -493,16 +516,35 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
         GstBuffer *destBuffer;
         GstCaps *destCaps;
         GstMapInfo info;
    -    gint stride = m_iEncodedWidth * 4;
    -    int status = 1;
    +    guint stride = 0;
    +    guint alloc_size = 0;
    +    int status = 0;
     
         // Not handling alpha ...
         if (m_bHasAlpha) {
             return NULL;
         }
     
    -    stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    -    destBuffer = alloc_aligned_buffer(stride * m_iEncodedHeight);
    +    // Make sure we do not have an integer overflow
    +    if (m_uiEncodedWidth <= (G_MAXUINT / 4)) {
    +        stride = m_uiEncodedWidth * 4;
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (stride <= (G_MAXUINT - 16)) {
    +        stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (m_uiEncodedHeight > 0 && stride <= (G_MAXUINT / m_uiEncodedHeight)) {
    +        alloc_size = stride * m_uiEncodedHeight;
    +    } else {
    +        return NULL;
    +    }
    +
    +    destBuffer = alloc_aligned_buffer(alloc_size);
         if (!destBuffer) {
             return NULL;
         }
    @@ -521,23 +563,23 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
         // now do the conversion
         if (destType == ARGB) {
             status = ColorConvert_YCbCr422p_to_ARGB32_no_alpha(info.data, stride,
    -                                                           m_iEncodedWidth, m_iEncodedHeight,
    +                                                           m_uiEncodedWidth, m_uiEncodedHeight,
                                                                (uint8_t*)m_pvPlaneData[0] + 1,
                                                                (uint8_t*)m_pvPlaneData[0] + 2,
                                                                (uint8_t*)m_pvPlaneData[0],
    -                                                           m_piPlaneStrides[0], m_piPlaneStrides[0]);
    +                                                           m_puiPlaneStrides[0], m_puiPlaneStrides[0]);
         } else {
             status = ColorConvert_YCbCr422p_to_BGRA32_no_alpha(info.data, stride,
    -                                                           m_iEncodedWidth, m_iEncodedHeight,
    +                                                           m_uiEncodedWidth, m_uiEncodedHeight,
                                                                (uint8_t*)m_pvPlaneData[0] + 1,
                                                                (uint8_t*)m_pvPlaneData[0] + 2,
                                                                (uint8_t*)m_pvPlaneData[0],
    -                                                           m_piPlaneStrides[0], m_piPlaneStrides[0]);
    +                                                           m_puiPlaneStrides[0], m_puiPlaneStrides[0]);
         }
     
         gst_buffer_unmap(destBuffer, &info);
     
    -    destCaps = create_RGB_caps(destType, m_iWidth, m_iHeight, m_iEncodedWidth, m_iEncodedHeight, stride);
    +    destCaps = create_RGB_caps(destType, m_uiWidth, m_uiHeight, m_uiEncodedWidth, m_uiEncodedHeight, stride);
         if (!destCaps) {
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer);
    @@ -556,12 +598,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
     
         if (0 == status && destBuffer) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
     
         return NULL;
    @@ -574,7 +620,7 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
         GstCaps *srcCaps, *dstCaps;
         GstMapInfo srcInfo, destInfo;
         GstStructure* str;
    -    gint xx, yy, size;
    +    guint xx, yy, size;
         guint32 *srcData, *dstData;
     
         size = gst_buffer_get_size(m_pBuffer);
    @@ -610,7 +656,7 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
                 break;
             default:
                 // shouldn't have gotten this far...
    -// INLINE - gst_buffer_unref()
    +            // INLINE - gst_buffer_unref()
                 gst_buffer_unref(destBuffer);
                 gst_caps_unref(dstCaps);
                 return NULL;
    @@ -646,18 +692,18 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
         // Now copy data from src to dest, byteswapping as we copy
         srcData = (guint32*)srcInfo.data;
         dstData = (guint32*)destInfo.data;
    -    if (!(m_piPlaneStrides[0] & 3)) {
    +    if (!(m_puiPlaneStrides[0] & 3)) {
             // four byte alignment on the entire buffer, we can just loop once
             for (xx = 0; xx < size; xx += 4) {
                 *dstData++ = swap_uint32(*srcData++); // NOTE: SSE could be used here instead
             }
         } else {
    -        for (yy = 0; yy < m_iHeight; yy++) {
    -            for (xx = 0; xx < m_iWidth; xx++) {
    +        for (yy = 0; yy < m_uiHeight; yy++) {
    +            for (xx = 0; xx < m_uiWidth; xx++) {
                     dstData[xx] = swap_uint32(srcData[xx]); // NOTE: SSE could be used here instead
                 }
    -            dstData += m_piPlaneStrides[0];
    -            srcData += m_piPlaneStrides[0];
    +            dstData += m_puiPlaneStrides[0];
    +            srcData += m_puiPlaneStrides[0];
             }
         }
     
    @@ -666,12 +712,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
     
         if (destBuffer) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
         return NULL;
     }
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/avf/AVFMediaPlayer.mm+5 1 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2014, 2021, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -648,6 +648,10 @@ - (void) sendPixelBuffer:(CVPixelBufferRef)buf frameTime:(double)frameTime hostT
             return;
         }
     
    +    if (frame == NULL) {
    +        return;
    +    }
    +
         if (previousWidth < 0 || previousHeight < 0
             || previousWidth != frame->GetWidth() || previousHeight != frame->GetHeight())
         {
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/CVVideoFrame.h+4 3 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -40,8 +40,9 @@ class CVVideoFrame : public CVideoFrame {
         virtual CVideoFrame *ConvertToFormat(FrameType type);
     
     private:
    -    CVPixelBufferRef pixelBuffer;
    -    uint64_t frameHostTime;
    +    bool m_bDisposePixelBuffer;
    +    CVPixelBufferRef m_pixelBuffer;
    +    uint64_t m_frameHostTime;
     
         void PrepareChunky();
         void PreparePlanar();
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/CVVideoFrame.mm+62 39 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -39,32 +39,43 @@
         return false;
     }
     
    -CVVideoFrame::CVVideoFrame(CVPixelBufferRef buf, double frameTime, uint64_t frameHostTime)
    +CVVideoFrame::CVVideoFrame(CVPixelBufferRef pixelBuffer, double frameTime, uint64_t frameHostTime)
     :
    -  pixelBuffer(buf),
    -  frameHostTime(frameHostTime)
    +  m_bDisposePixelBuffer(false),
    +  m_pixelBuffer(pixelBuffer)
     {
         // We can assume that buf is retained at least for the duration of this ctor
         // So postpone retaining it until we're absolutely sure we'll use it
     
         // fail fast
    -    OSType type = CVPixelBufferGetPixelFormatType(buf);
    +    OSType type = CVPixelBufferGetPixelFormatType(pixelBuffer);
         if (!IsFormatSupported(type)) {
             throw "CVVideoFrame: Invalid PixelFormat";
         }
     
         m_dTime = frameTime;
    -    m_iWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    -    m_iHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    +    m_frameHostTime = frameHostTime;
    +    m_uiWidth = (unsigned int)CVPixelBufferGetWidth(pixelBuffer);
    +    m_uiHeight = (unsigned int)CVPixelBufferGetHeight(pixelBuffer);
     
         size_t extLeft, extRight, extTop, extBottom;
         CVPixelBufferGetExtendedPixels(pixelBuffer, &extLeft, &extRight, &extTop, &extBottom);
    -    m_iEncodedWidth = m_iWidth + (int)extLeft + (int)extRight;
    -    m_iEncodedHeight = m_iHeight + (int)extBottom; // ignore top, since 0,0 is where base addr starts
     
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    +    if (m_uiWidth <= (UINT_MAX - (unsigned int)extLeft) &&
    +          (m_uiWidth + (unsigned int)extLeft) <= (UINT_MAX - (unsigned int)extRight)) {
    +        m_uiEncodedWidth = m_uiWidth + (unsigned int)extLeft + (unsigned int)extRight;
    +    } else {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
    +
    +    if (m_uiHeight <= (UINT_MAX - (unsigned int)extBottom)) {
    +        // ignore top, since 0,0 is where base addr starts
    +        m_uiEncodedHeight = m_uiHeight + (unsigned int)extBottom;
    +    } else {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
    +
    +    Reset();
         m_FrameDirty = false;
     
         if (type != gLastFormat) {
    @@ -95,7 +106,8 @@
         }
     
         // Now retain the pixelBuffer so it doesn't go away
    -    CVPixelBufferRetain(pixelBuffer);
    +    m_bDisposePixelBuffer = true;
    +    CVPixelBufferRetain(m_pixelBuffer);
     }
     
     CVVideoFrame::~CVVideoFrame()
    @@ -105,39 +117,49 @@
     
     void CVVideoFrame::PrepareChunky()
     {
    -    m_iPlaneCount = 1;
    -    m_iWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    -    m_iHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    +    SetPlaneCount(1);
    +    m_uiWidth = (unsigned int)CVPixelBufferGetWidth(m_pixelBuffer);
    +    m_uiHeight = (unsigned int)CVPixelBufferGetHeight(m_pixelBuffer);
         m_bHasAlpha = (m_typeFrame == BGRA_PRE);
     
         // We MUST lock the base address during the lifetime of this object
         // or else we could cause a crash
    -    CVReturn cr = CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +    CVReturn cr = CVPixelBufferLockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
         if (kCVReturnSuccess != cr) {
             throw "CVVideoFrame: Unable to lock PixelBuffer base address";
         }
     
    -    m_pvPlaneData[0] = CVPixelBufferGetBaseAddress(pixelBuffer);
    -    m_piPlaneStrides[0] = (int)CVPixelBufferGetBytesPerRow(pixelBuffer);
    -    m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    +    m_pvPlaneData[0] = CVPixelBufferGetBaseAddress(m_pixelBuffer);
    +    m_puiPlaneStrides[0] = (int)CVPixelBufferGetBytesPerRow(m_pixelBuffer);
    +
    +    bool bValid = true; // CalcSize() requires bValid to be true when called
    +    m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &bValid);
    +    if (!bValid) {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
     }
     
     void CVVideoFrame::PreparePlanar()
     {
    -    m_iPlaneCount = (int)CVPixelBufferGetPlaneCount(pixelBuffer);
    +    SetPlaneCount((unsigned int)CVPixelBufferGetPlaneCount(m_pixelBuffer));
         m_bHasAlpha = false;
     
         // We MUST lock the base address during the lifetime of this object
         // or else we could cause a crash
    -    CVReturn cr = CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +    CVReturn cr = CVPixelBufferLockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
         if (kCVReturnSuccess != cr) {
             throw "CVVideoFrame: Unable to lock PixelBuffer base address";
         }
     
    -    for (int index = 0; index < m_iPlaneCount; index++) {
    -        m_piPlaneStrides[index] = (int)CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, index);
    -        m_pulPlaneSize[index] = CVPixelBufferGetHeightOfPlane(pixelBuffer, index) * m_piPlaneStrides[index];
    -        m_pvPlaneData[index] = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, index);
    +    bool bValid = true; // CalcSize() requires bValid to be true when called
    +    for (int index = 0; index < GetPlaneCount(); index++) {
    +        m_puiPlaneStrides[index] = (unsigned int)CVPixelBufferGetBytesPerRowOfPlane(m_pixelBuffer, index);
    +        unsigned long ulHeightOfPlane = (unsigned long)CVPixelBufferGetHeightOfPlane(m_pixelBuffer, index);
    +        m_pulPlaneSize[index] = CalcSize(ulHeightOfPlane, m_puiPlaneStrides[index], &bValid);
    +        if (!bValid) {
    +            throw "CVVideoFrame: Invalid frame size";
    +        }
    +        m_pvPlaneData[index] = CVPixelBufferGetBaseAddressOfPlane(m_pixelBuffer, index);
         }
     
         if (m_typeFrame == YCbCr_420p) {
    @@ -148,34 +170,35 @@
     
     void CVVideoFrame::Dispose()
     {
    -    if (pixelBuffer) {
    -        CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    -        CVPixelBufferRelease(pixelBuffer);
    -        pixelBuffer = 0;
    +    if (m_bDisposePixelBuffer) {
    +        CVPixelBufferUnlockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +        CVPixelBufferRelease(m_pixelBuffer);
    +        m_pixelBuffer = NULL;
    +        m_bDisposePixelBuffer = false;
         }
     }
     
     CVideoFrame *CVVideoFrame::ConvertToFormat(FrameType type)
     {
    -    if(YCbCr_422 == m_typeFrame && BGRA_PRE == type) {
    +    if (YCbCr_422 == m_typeFrame && BGRA_PRE == type) {
             CVPixelBufferRef destPixelBuffer = NULL;
    -        if(kCVReturnSuccess == CVPixelBufferCreate(NULL, m_iEncodedWidth, m_iEncodedHeight,
    +        if (kCVReturnSuccess == CVPixelBufferCreate(NULL, m_uiEncodedWidth, m_uiEncodedHeight,
                                                        k32BGRAPixelFormat, NULL, &destPixelBuffer)) {
    -            if(kCVReturnSuccess == CVPixelBufferLockBaseAddress(destPixelBuffer, 0)) {
    +            if (kCVReturnSuccess == CVPixelBufferLockBaseAddress(destPixelBuffer, 0)) {
                     uint8_t* bgra = (uint8_t*)CVPixelBufferGetBaseAddress(destPixelBuffer);
                     int32_t bgraStride = (int32_t)CVPixelBufferGetBytesPerRow(destPixelBuffer);
                     uint8_t* srcData = (uint8_t*)m_pvPlaneData[0];
     
    -                if(0 == ColorConvert_YCbCr422p_to_BGRA32_no_alpha(bgra,
    +                if (0 == ColorConvert_YCbCr422p_to_BGRA32_no_alpha(bgra,
                                                                       bgraStride,
    -                                                                  m_iEncodedWidth,
    -                                                                  m_iEncodedHeight,
    +                                                                  m_uiEncodedWidth,
    +                                                                  m_uiEncodedHeight,
                                                                       srcData + 1,
                                                                       srcData + 2,
                                                                       srcData,
    -                                                                  m_piPlaneStrides[0],
    -                                                                  m_piPlaneStrides[0])) {
    -                    return new CVVideoFrame(destPixelBuffer, m_dTime, frameHostTime);
    +                                                                  m_puiPlaneStrides[0],
    +                                                                  m_puiPlaneStrides[0])) {
    +                    return new CVVideoFrame(destPixelBuffer, m_dTime, m_frameHostTime);
                     }
                 }
             }
    
0c00753da13e

8313105: Improved media framing

https://github.com/openjdk/jfx21uAlexander MatveevSep 7, 2023via ghsa
7 files changed · +362 190
  • modules/javafx.media/src/main/native/jfxmedia/jni/NativeVideoBuffer.cpp+3 3 modified
    @@ -71,8 +71,8 @@ JNIEXPORT jobject JNICALL Java_com_sun_media_jfxmediaimpl_NativeVideoBuffer_nati
     {
         CVideoFrame *frame = (CVideoFrame*)jlong_to_ptr(nativeHandle);
         if (frame) {
    -        void *dataPtr = frame->GetDataForPlane((int)plane);
    -        jlong capacity = (jlong)frame->GetSizeForPlane((int)plane);
    +        void *dataPtr = frame->GetDataForPlane((unsigned int)plane);
    +        jlong capacity = (jlong)frame->GetSizeForPlane((unsigned int)plane);
             jobject buffer = env->NewDirectByteBuffer(dataPtr, capacity);
             if (env->ExceptionCheck()) {
                 env->ExceptionClear();
    @@ -215,7 +215,7 @@ JNIEXPORT jintArray JNICALL Java_com_sun_media_jfxmediaimpl_NativeVideoBuffer_na
                 return NULL;
             }
     
    -        for (int ii=0; ii < count; ii++) {
    +        for (unsigned int ii=0; ii < count; ii++) {
                 strideArray[ii] = frame->GetStrideForPlane(ii);
             }
     
    
  • modules/javafx.media/src/main/native/jfxmedia/PipelineManagement/VideoFrame.cpp+103 32 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -23,50 +23,51 @@
      * questions.
      */
     
    +// For UINT_MAX, we cannot use GLib here, since it is shared code between
    +// GStreamer and AVFoundation.
    +#include <limits.h>
    +
     #include "VideoFrame.h"
     #include <Common/VSMemory.h>
     
     //*************************************************************************************************
     //********** class CVideoFrame
     //*************************************************************************************************
     CVideoFrame::CVideoFrame()
    -:   m_iWidth(0),
    -    m_iHeight(0),
    -    m_iEncodedWidth(0),
    -    m_iEncodedHeight(0),
    +:   m_uiWidth(0),
    +    m_uiHeight(0),
    +    m_uiEncodedWidth(0),
    +    m_uiEncodedHeight(0),
         m_typeFrame(UNKNOWN),
         m_bHasAlpha(false),
         m_dTime(0.0),
    -    m_FrameDirty(false),
    -    m_iPlaneCount(1)
    +    m_FrameDirty(false)
     {
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    +    Reset();
     }
     
     CVideoFrame::~CVideoFrame()
     {
     }
     
    -int CVideoFrame::GetWidth()
    +unsigned int CVideoFrame::GetWidth()
     {
    -    return m_iWidth;
    +    return m_uiWidth;
     }
     
    -int CVideoFrame::GetHeight()
    +unsigned int CVideoFrame::GetHeight()
     {
    -    return m_iHeight;
    +    return m_uiHeight;
     }
     
    -int CVideoFrame::GetEncodedWidth()
    +unsigned int CVideoFrame::GetEncodedWidth()
     {
    -    return m_iEncodedWidth;
    +    return m_uiEncodedWidth;
     }
     
    -int CVideoFrame::GetEncodedHeight()
    +unsigned int CVideoFrame::GetEncodedHeight()
     {
    -    return m_iEncodedHeight;
    +    return m_uiEncodedHeight;
     }
     
     CVideoFrame::FrameType CVideoFrame::GetType()
    @@ -84,31 +85,41 @@ double CVideoFrame::GetTime()
         return m_dTime;
     }
     
    -int CVideoFrame::GetPlaneCount()
    +unsigned int CVideoFrame::GetPlaneCount()
     {
    -    return m_iPlaneCount;
    +    return m_uiPlaneCount;
    +}
    +
    +void CVideoFrame::SetPlaneCount(unsigned int count)
    +{
    +    if (count <= MAX_PLANE_COUNT) {
    +        m_uiPlaneCount = count;
    +    } else {
    +        // Should never happen
    +        m_uiPlaneCount = MAX_PLANE_COUNT;
    +    }
     }
     
    -void* CVideoFrame::GetDataForPlane(int planeIndex)
    +void* CVideoFrame::GetDataForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    +    if (planeIndex < MAX_PLANE_COUNT) {
             return m_pvPlaneData[planeIndex];
         }
         return NULL;
     }
     
    -unsigned long CVideoFrame::GetSizeForPlane(int planeIndex)
    +unsigned long CVideoFrame::GetSizeForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    +    if (planeIndex < MAX_PLANE_COUNT) {
             return m_pulPlaneSize[planeIndex];
         }
         return 0;
     }
     
    -int CVideoFrame::GetStrideForPlane(int planeIndex)
    +unsigned int CVideoFrame::GetStrideForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    -        return m_piPlaneStrides[planeIndex];
    +    if (planeIndex < MAX_PLANE_COUNT) {
    +        return m_puiPlaneStrides[planeIndex];
         }
         return 0;
     }
    @@ -118,12 +129,22 @@ CVideoFrame *CVideoFrame::ConvertToFormat(FrameType type)
         return NULL;
     }
     
    -void CVideoFrame::SwapPlanes(int aa, int bb)
    +void CVideoFrame::Reset()
    +{
    +    m_uiPlaneCount = 0;
    +    for (int i = 0; i < MAX_PLANE_COUNT; i++) {
    +        m_puiPlaneStrides[i] = 0;
    +        m_pulPlaneSize[i] = 0;
    +        m_pvPlaneData[i] = NULL;
    +    }
    +}
    +
    +void CVideoFrame::SwapPlanes(unsigned int aa, unsigned int bb)
     {
    -    if (aa != bb && aa >= 0 && aa < m_iPlaneCount && bb >= 0 && bb < m_iPlaneCount) {
    -        int stride = m_piPlaneStrides[aa];
    -        m_piPlaneStrides[aa] = m_piPlaneStrides[bb];
    -        m_piPlaneStrides[bb] = stride;
    +    if (aa != bb && aa < m_uiPlaneCount && bb < m_uiPlaneCount) {
    +        unsigned int stride = m_puiPlaneStrides[aa];
    +        m_puiPlaneStrides[aa] = m_puiPlaneStrides[bb];
    +        m_puiPlaneStrides[bb] = stride;
     
             unsigned long size = m_pulPlaneSize[aa];
             m_pulPlaneSize[aa] = m_pulPlaneSize[bb];
    @@ -134,3 +155,53 @@ void CVideoFrame::SwapPlanes(int aa, int bb)
             m_pvPlaneData[bb] = vptr;
         }
     }
    +
    +unsigned long CVideoFrame::CalcSize(unsigned int a, unsigned int b, bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return 0;
    +    }
    +
    +    if (b > 0 && a <= (UINT_MAX / b)) {
    +        return (a * b);
    +    }
    +
    +    *(pbValid) = false;
    +    return 0;
    +}
    +
    +unsigned long CVideoFrame::AddSize(unsigned long a, unsigned long b, bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return 0;
    +    }
    +
    +    // unsigned long can be 32-bit or 64-bit, make sure it is no more then UINT_MAX
    +    if (a <= UINT_MAX && b <= UINT_MAX && a <= (UINT_MAX - b)) {
    +        return (a + b);
    +    }
    +
    +    *(pbValid) = false;
    +    return 0;
    +}
    +
    +void* CVideoFrame::CalcPlanePointer(intptr_t baseAddress, unsigned int offset,
    +                                    unsigned long planeSize, unsigned long baseSize,
    +                                    bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return NULL;
    +    }
    +
    +    // We will read planeSize bytes from baseAddress starting with offset, so
    +    // make sure we do not read pass baseSize.
    +    unsigned long endOfPlane = AddSize(offset, planeSize, pbValid);
    +    if (*(pbValid)) { // Make sure AddSize() did not failed.
    +        if (endOfPlane <= baseSize) {
    +            return (void*)(baseAddress + offset);
    +        }
    +    }
    +
    +    *(pbValid) = false;
    +    return NULL;
    +}
    
  • modules/javafx.media/src/main/native/jfxmedia/PipelineManagement/VideoFrame.h+41 18 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -27,6 +27,9 @@
     #define _VIDEO_FRAME_H_
     
     #include <stdlib.h>
    +#include <stdint.h>
    +
    +#define MAX_PLANE_COUNT 4
     
     /**
      * class CVideoFrame
    @@ -56,41 +59,61 @@ class CVideoFrame
     
         double              GetTime();
     
    -    int                 GetWidth();
    -    int                 GetHeight();
    -    int                 GetEncodedWidth();
    -    int                 GetEncodedHeight();
    +    unsigned int        GetWidth();
    +    unsigned int        GetHeight();
    +    unsigned int        GetEncodedWidth();
    +    unsigned int        GetEncodedHeight();
     
         FrameType           GetType();
         bool                HasAlpha();
     
    -    int                 GetPlaneCount();
    -    void*               GetDataForPlane(int planeIndex);
    -    unsigned long       GetSizeForPlane(int planeIndex);
    -    int                 GetStrideForPlane(int planeIndex);
    +    unsigned int        GetPlaneCount();
    +    void                SetPlaneCount(unsigned int count);
    +    void*               GetDataForPlane(unsigned int planeIndex);
    +    unsigned long       GetSizeForPlane(unsigned int planeIndex);
    +    unsigned int        GetStrideForPlane(unsigned int planeIndex);
     
         virtual CVideoFrame *ConvertToFormat(FrameType type);
     
         bool                GetFrameDirty() { return m_FrameDirty; }
         void                SetFrameDirty(bool dirty) { m_FrameDirty = dirty; }
     
     protected:
    -    int                 m_iWidth;
    -    int                 m_iHeight;
    -    int                 m_iEncodedWidth;
    -    int                 m_iEncodedHeight;
    +    unsigned int        m_uiWidth;
    +    unsigned int        m_uiHeight;
    +    unsigned int        m_uiEncodedWidth;
    +    unsigned int        m_uiEncodedHeight;
         FrameType           m_typeFrame;
         bool                m_bHasAlpha;
         double              m_dTime;
         bool                m_FrameDirty;
     
         // frame data buffers
    -    int                 m_iPlaneCount;
    -    void*               m_pvPlaneData[4];
    -    unsigned long       m_pulPlaneSize[4];
    -    int                 m_piPlaneStrides[4];
    +    void*               m_pvPlaneData[MAX_PLANE_COUNT];
    +    unsigned long       m_pulPlaneSize[MAX_PLANE_COUNT];
    +    unsigned int        m_puiPlaneStrides[MAX_PLANE_COUNT];
    +
    +    void Reset();
    +    void SwapPlanes(unsigned int aa, unsigned int bb);
    +
    +    // CalcSize(), AddSize(), CalcPlanePointer() requires bValid to be set to
    +    // true initially, if bValid is false these functions do nothing. It is
    +    // implemented this way, so all these functions can be chain called without
    +    // checking bValid after each call. bValid will be set to false only if
    +    // calculation failed and will never be set to true.
    +    // Multiplies a and b, bValid set to false if integer overflow detected.
    +    unsigned long CalcSize(unsigned int a, unsigned int b, bool *pbValid);
    +    // Adds a and b, bValid set to false if integer overflow detected.
    +    unsigned long AddSize(unsigned long a, unsigned long b, bool *pbValid);
    +    // Calculates plane pointer (baseAddress + offset) and checks that calculated
    +    // pointer within buffer. Returns NULL and sets bValid to false if calculated
    +    // pointer is invalid.
    +    void* CalcPlanePointer(intptr_t baseAddress, unsigned int offset,
    +                           unsigned long planeSize, unsigned long baseSize,
    +                           bool *pbValid);
     
    -    void SwapPlanes(int aa, int bb);
    +private:
    +    unsigned int        m_uiPlaneCount;
     };
     
     #endif  //_VIDEO_FRAME_H_
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/gstreamer/GstVideoFrame.cpp+144 94 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -56,18 +56,23 @@ static GstBuffer *alloc_aligned_buffer(guint size)
     
         // allocate a buffer large enough to accommodate 16 byte alignment
         alignedSize = size;
    -    size += 16;
    +    if (size <= (G_MAXUINT - 16)) {
    +        size += 16;
    +    } else {
    +        return NULL;
    +    }
    +
         newData = (guint8*)g_try_malloc(size);
         if (NULL == newData) {
             return NULL;
         }
     
         alignedData = (guint8*)(((intptr_t)newData + 15) & ~15);
     
    -    return gst_buffer_new_wrapped_full((GstMemoryFlags)0, alignedData, alignedSize, 0, 0, newData, free_aligned_buffer);
    +    return gst_buffer_new_wrapped_full((GstMemoryFlags)0, alignedData, alignedSize, 0, alignedSize, newData, free_aligned_buffer);
     }
     
    -GstCaps *create_RGB_caps(CVideoFrame::FrameType type, gint width, gint height, gint encodedWidth, gint encodedHeight, gint stride)
    +GstCaps *create_RGB_caps(CVideoFrame::FrameType type, guint width, guint height, guint encodedWidth, guint encodedHeight, guint stride)
     {
         gint red_mask, green_mask, blue_mask, alpha_mask;
         GstCaps *newCaps;
    @@ -160,7 +165,8 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
         const GstStructure* str = gst_caps_get_structure(newCaps, 0);
         const gchar* sFormatFourCC = gst_structure_get_string(str, "format");
     
    -    // default to success
    +    // We should always start with success. See CalcSize(), AddSize() and
    +    // CalcPlanePointer() on how this flag is being used.
         m_bIsValid = true;
     
         // FIXME: make format type strings conformant with constant types
    @@ -209,83 +215,79 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
             m_bIsValid = false;
         }
     
    -    if(!gst_structure_get_int(str, "width", &m_iWidth))
    +    if (!gst_structure_get_int(str, "width", (int*)&m_uiWidth))
         {
     #if JFXMEDIA_DEBUG
             g_warning("width could not be retrieved from GstBuffer\n");
     #endif
    -        m_iWidth = 0;
    +        m_uiWidth = 0;
             m_bIsValid = false;
         }
    -    if(!gst_structure_get_int(str, "height", &m_iHeight))
    +    if (!gst_structure_get_int(str, "height", (int*)&m_uiHeight))
         {
     #if JFXMEDIA_DEBUG
             g_warning("height could not be retrieved from GstBuffer\n");
     #endif
    -        m_iHeight = 0;
    +        m_uiHeight = 0;
             m_bIsValid = false;
         }
     
    -    if (!gst_structure_get_int(str, "encoded-width", &m_iEncodedWidth)) {
    -        m_iEncodedWidth = m_iWidth;
    +    if (!gst_structure_get_int(str, "encoded-width", (int*)&m_uiEncodedWidth)) {
    +        m_uiEncodedWidth = m_uiWidth;
         }
    -    if (!gst_structure_get_int(str, "encoded-height", &m_iEncodedHeight)) {
    -        m_iEncodedHeight = m_iHeight;
    +    if (!gst_structure_get_int(str, "encoded-height", (int*)&m_uiEncodedHeight)) {
    +        m_uiEncodedHeight = m_uiHeight;
         }
     
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    +    Reset();
     
    -    unsigned long expectedSize = 0;
         switch (m_typeFrame) {
             case YCbCr_420p: {
    -            int offset;
    -            m_iPlaneCount = 3;
    +            unsigned int offset = 0;
    +            SetPlaneCount(3);
     
    -            if (!gst_structure_get_int(str, "stride-y", &m_piPlaneStrides[0])) {
    -                m_piPlaneStrides[0] = m_iEncodedWidth;
    +            if (!gst_structure_get_int(str, "stride-y", (int*)&m_puiPlaneStrides[0])) {
    +                m_puiPlaneStrides[0] = m_uiEncodedWidth;
                 }
    -            if (!gst_structure_get_int(str, "stride-v", &m_piPlaneStrides[1])) {
    -                m_piPlaneStrides[1] = m_iEncodedWidth/2;
    +            if (!gst_structure_get_int(str, "stride-v", (int*)&m_puiPlaneStrides[1])) {
    +                m_puiPlaneStrides[1] = m_uiEncodedWidth/2;
                 }
    -            if (!gst_structure_get_int(str, "stride-u", &m_piPlaneStrides[2])) {
    -                m_piPlaneStrides[2] = m_piPlaneStrides[1];
    +            if (!gst_structure_get_int(str, "stride-u", (int*)&m_puiPlaneStrides[2])) {
    +                m_puiPlaneStrides[2] = m_puiPlaneStrides[1];
                 }
     
    -            offset = 0;
    -            gst_structure_get_int(str, "offset-y", &offset);
    -            m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    -            m_pvPlaneData[0] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[0];
    +            gst_structure_get_int(str, "offset-y", (int*)&offset);
    +            m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &m_bIsValid);
    +            m_pvPlaneData[0] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[0], m_ulBufferSize, &m_bIsValid);
     
                 //
                 // Chroma offsets assume YV12 ordering
                 //
                 offset += m_pulPlaneSize[0];
    -            gst_structure_get_int(str, "offset-v", &offset);
    -            m_pulPlaneSize[1] = m_piPlaneStrides[1] * (m_iEncodedHeight/2);
    -            m_pvPlaneData[1] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[1];
    +            gst_structure_get_int(str, "offset-v", (int*)&offset);
    +            m_pulPlaneSize[1] = CalcSize(m_puiPlaneStrides[1], (m_uiEncodedHeight/2), &m_bIsValid);
    +            m_pvPlaneData[1] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[1], m_ulBufferSize, &m_bIsValid);
     
                 offset += m_pulPlaneSize[1];
    -            gst_structure_get_int(str, "offset-u", &offset);
    -            m_pulPlaneSize[2] = m_piPlaneStrides[2] * (m_iEncodedHeight/2);
    -            m_pvPlaneData[2] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[2];
    +            gst_structure_get_int(str, "offset-u", (int*)&offset);
    +            m_pulPlaneSize[2] = CalcSize(m_puiPlaneStrides[2], (m_uiEncodedHeight/2), &m_bIsValid);
    +            m_pvPlaneData[2] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[2], m_ulBufferSize, &m_bIsValid);
     
                 // process alpha channel (before we potentially swap Cb/Cr)
                 if (m_bHasAlpha) {
    -                m_iPlaneCount++;
    -                if (!gst_structure_get_int(str, "stride-a", &m_piPlaneStrides[3])) {
    -                    m_piPlaneStrides[3] = m_piPlaneStrides[0];
    +                SetPlaneCount(GetPlaneCount() + 1);
    +                if (!gst_structure_get_int(str, "stride-a", (int*)&m_puiPlaneStrides[3])) {
    +                    m_puiPlaneStrides[3] = m_puiPlaneStrides[0];
                     }
     
                     offset += m_pulPlaneSize[2];
    -                gst_structure_get_int(str, "offset-a", &offset);
    -                m_pulPlaneSize[3] = m_piPlaneStrides[3] * m_iEncodedHeight;
    -                m_pvPlaneData[3] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -                expectedSize += m_pulPlaneSize[3];
    +                gst_structure_get_int(str, "offset-a", (int*)&offset);
    +                m_pulPlaneSize[3] = CalcSize(m_puiPlaneStrides[3], m_uiEncodedHeight, &m_bIsValid);
    +                m_pvPlaneData[3] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[3], m_ulBufferSize, &m_bIsValid);
                 }
     
                 //
    @@ -299,21 +301,19 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
             }
     
             default:
    -            m_iPlaneCount = 1;
    -            if (!gst_structure_get_int(str, "line_stride", &m_piPlaneStrides[0])) {
    +            SetPlaneCount(1);
    +            if (!gst_structure_get_int(str, "line_stride", (int*)&m_puiPlaneStrides[0])) {
                     if (m_typeFrame == YCbCr_422) {
    -                    m_piPlaneStrides[0] = m_iEncodedWidth * 2; // 16 bpp
    +                    m_puiPlaneStrides[0] = m_uiEncodedWidth * 2; // 16 bpp
                     } else {
    -                    m_piPlaneStrides[0] = m_iEncodedWidth * 4; // 32 bpp
    +                    m_puiPlaneStrides[0] = m_uiEncodedWidth * 4; // 32 bpp
                     }
                 }
    -            m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    -            m_pvPlaneData[0] = m_pvBufferBaseAddress;
    -            expectedSize += m_pulPlaneSize[0];
    +            m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &m_bIsValid);
    +            m_pvPlaneData[0] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, 0,
    +                                                m_pulPlaneSize[0], m_ulBufferSize, &m_bIsValid);
                 break;
         }
    -
    -    m_bIsValid = m_bIsValid && (expectedSize <= m_ulBufferSize);
     }
     
     bool CGstVideoFrame::IsValid()
    @@ -381,9 +381,10 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
         GstBuffer *destBuffer = NULL;
         GstCaps *destCaps = NULL;
         GstMapInfo info;
    -    gint stride = m_iEncodedWidth * 4;
    -    int u_index, v_index;
    -    int status;
    +    guint stride = 0;
    +    guint alloc_size = 0;
    +    unsigned int u_index, v_index = 0;
    +    int status = 0;
     
         if (m_bIsI420) {
             u_index = 1;
    @@ -393,8 +394,26 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
             v_index = 1;
         }
     
    -    stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    -    destBuffer = alloc_aligned_buffer(stride * m_iEncodedHeight);
    +    // Make sure we do not have an integer overflow
    +    if (m_uiEncodedWidth <= (G_MAXUINT / 4)) {
    +        stride = m_uiEncodedWidth * 4;
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (stride <= (G_MAXUINT - 16)) {
    +        stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (m_uiEncodedHeight > 0 && stride <= (G_MAXUINT / m_uiEncodedHeight)) {
    +        alloc_size = stride * m_uiEncodedHeight;
    +    } else {
    +        return NULL;
    +    }
    +
    +    destBuffer = alloc_aligned_buffer(alloc_size);
         if (!destBuffer) {
             return NULL;
         }
    @@ -415,49 +434,49 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
             if (m_bHasAlpha) {
                 status = ColorConvert_YCbCr420p_to_ARGB32(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
                             (const uint8_t*)m_pvPlaneData[3],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index], m_piPlaneStrides[3]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index], m_puiPlaneStrides[3]);
             } else {
                 status = ColorConvert_YCbCr420p_to_ARGB32_no_alpha(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index]);
             }
         } else {
             if (m_bHasAlpha) {
                 status = ColorConvert_YCbCr420p_to_BGRA32(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
                             (const uint8_t*)m_pvPlaneData[3],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index], m_piPlaneStrides[3]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index], m_puiPlaneStrides[3]);
             } else {
                 status = ColorConvert_YCbCr420p_to_BGRA32_no_alpha(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index]);
             }
         }
     
         gst_buffer_unmap(destBuffer, &info);
     
    -    destCaps = create_RGB_caps(destType, m_iWidth, m_iHeight, m_iEncodedWidth, m_iEncodedHeight, stride);
    +    destCaps = create_RGB_caps(destType, m_uiWidth, m_uiHeight, m_uiEncodedWidth, m_uiEncodedHeight, stride);
         if (!destCaps) {
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer);
    @@ -476,12 +495,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
     
         if (0 == status && destSample) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_sample_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
     
         return NULL;
    @@ -493,16 +516,35 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
         GstBuffer *destBuffer;
         GstCaps *destCaps;
         GstMapInfo info;
    -    gint stride = m_iEncodedWidth * 4;
    -    int status = 1;
    +    guint stride = 0;
    +    guint alloc_size = 0;
    +    int status = 0;
     
         // Not handling alpha ...
         if (m_bHasAlpha) {
             return NULL;
         }
     
    -    stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    -    destBuffer = alloc_aligned_buffer(stride * m_iEncodedHeight);
    +    // Make sure we do not have an integer overflow
    +    if (m_uiEncodedWidth <= (G_MAXUINT / 4)) {
    +        stride = m_uiEncodedWidth * 4;
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (stride <= (G_MAXUINT - 16)) {
    +        stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (m_uiEncodedHeight > 0 && stride <= (G_MAXUINT / m_uiEncodedHeight)) {
    +        alloc_size = stride * m_uiEncodedHeight;
    +    } else {
    +        return NULL;
    +    }
    +
    +    destBuffer = alloc_aligned_buffer(alloc_size);
         if (!destBuffer) {
             return NULL;
         }
    @@ -521,23 +563,23 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
         // now do the conversion
         if (destType == ARGB) {
             status = ColorConvert_YCbCr422p_to_ARGB32_no_alpha(info.data, stride,
    -                                                           m_iEncodedWidth, m_iEncodedHeight,
    +                                                           m_uiEncodedWidth, m_uiEncodedHeight,
                                                                (uint8_t*)m_pvPlaneData[0] + 1,
                                                                (uint8_t*)m_pvPlaneData[0] + 2,
                                                                (uint8_t*)m_pvPlaneData[0],
    -                                                           m_piPlaneStrides[0], m_piPlaneStrides[0]);
    +                                                           m_puiPlaneStrides[0], m_puiPlaneStrides[0]);
         } else {
             status = ColorConvert_YCbCr422p_to_BGRA32_no_alpha(info.data, stride,
    -                                                           m_iEncodedWidth, m_iEncodedHeight,
    +                                                           m_uiEncodedWidth, m_uiEncodedHeight,
                                                                (uint8_t*)m_pvPlaneData[0] + 1,
                                                                (uint8_t*)m_pvPlaneData[0] + 2,
                                                                (uint8_t*)m_pvPlaneData[0],
    -                                                           m_piPlaneStrides[0], m_piPlaneStrides[0]);
    +                                                           m_puiPlaneStrides[0], m_puiPlaneStrides[0]);
         }
     
         gst_buffer_unmap(destBuffer, &info);
     
    -    destCaps = create_RGB_caps(destType, m_iWidth, m_iHeight, m_iEncodedWidth, m_iEncodedHeight, stride);
    +    destCaps = create_RGB_caps(destType, m_uiWidth, m_uiHeight, m_uiEncodedWidth, m_uiEncodedHeight, stride);
         if (!destCaps) {
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer);
    @@ -556,12 +598,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
     
         if (0 == status && destBuffer) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
     
         return NULL;
    @@ -574,7 +620,7 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
         GstCaps *srcCaps, *dstCaps;
         GstMapInfo srcInfo, destInfo;
         GstStructure* str;
    -    gint xx, yy, size;
    +    guint xx, yy, size;
         guint32 *srcData, *dstData;
     
         size = gst_buffer_get_size(m_pBuffer);
    @@ -610,7 +656,7 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
                 break;
             default:
                 // shouldn't have gotten this far...
    -// INLINE - gst_buffer_unref()
    +            // INLINE - gst_buffer_unref()
                 gst_buffer_unref(destBuffer);
                 gst_caps_unref(dstCaps);
                 return NULL;
    @@ -646,18 +692,18 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
         // Now copy data from src to dest, byteswapping as we copy
         srcData = (guint32*)srcInfo.data;
         dstData = (guint32*)destInfo.data;
    -    if (!(m_piPlaneStrides[0] & 3)) {
    +    if (!(m_puiPlaneStrides[0] & 3)) {
             // four byte alignment on the entire buffer, we can just loop once
             for (xx = 0; xx < size; xx += 4) {
                 *dstData++ = swap_uint32(*srcData++); // NOTE: SSE could be used here instead
             }
         } else {
    -        for (yy = 0; yy < m_iHeight; yy++) {
    -            for (xx = 0; xx < m_iWidth; xx++) {
    +        for (yy = 0; yy < m_uiHeight; yy++) {
    +            for (xx = 0; xx < m_uiWidth; xx++) {
                     dstData[xx] = swap_uint32(srcData[xx]); // NOTE: SSE could be used here instead
                 }
    -            dstData += m_piPlaneStrides[0];
    -            srcData += m_piPlaneStrides[0];
    +            dstData += m_puiPlaneStrides[0];
    +            srcData += m_puiPlaneStrides[0];
             }
         }
     
    @@ -666,12 +712,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
     
         if (destBuffer) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
         return NULL;
     }
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/avf/AVFMediaPlayer.mm+5 1 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2014, 2022, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -704,6 +704,10 @@ - (void) sendPixelBuffer:(CVPixelBufferRef)buf frameTime:(double)frameTime hostT
             return;
         }
     
    +    if (frame == NULL) {
    +        return;
    +    }
    +
         if (previousWidth < 0 || previousHeight < 0
             || previousWidth != frame->GetWidth() || previousHeight != frame->GetHeight())
         {
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/CVVideoFrame.h+4 3 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -40,8 +40,9 @@ class CVVideoFrame : public CVideoFrame {
         virtual CVideoFrame *ConvertToFormat(FrameType type);
     
     private:
    -    CVPixelBufferRef pixelBuffer;
    -    uint64_t frameHostTime;
    +    bool m_bDisposePixelBuffer;
    +    CVPixelBufferRef m_pixelBuffer;
    +    uint64_t m_frameHostTime;
     
         void PrepareChunky();
         void PreparePlanar();
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/CVVideoFrame.mm+62 39 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -39,32 +39,43 @@
         return false;
     }
     
    -CVVideoFrame::CVVideoFrame(CVPixelBufferRef buf, double frameTime, uint64_t frameHostTime)
    +CVVideoFrame::CVVideoFrame(CVPixelBufferRef pixelBuffer, double frameTime, uint64_t frameHostTime)
     :
    -  pixelBuffer(buf),
    -  frameHostTime(frameHostTime)
    +  m_bDisposePixelBuffer(false),
    +  m_pixelBuffer(pixelBuffer)
     {
         // We can assume that buf is retained at least for the duration of this ctor
         // So postpone retaining it until we're absolutely sure we'll use it
     
         // fail fast
    -    OSType type = CVPixelBufferGetPixelFormatType(buf);
    +    OSType type = CVPixelBufferGetPixelFormatType(pixelBuffer);
         if (!IsFormatSupported(type)) {
             throw "CVVideoFrame: Invalid PixelFormat";
         }
     
         m_dTime = frameTime;
    -    m_iWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    -    m_iHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    +    m_frameHostTime = frameHostTime;
    +    m_uiWidth = (unsigned int)CVPixelBufferGetWidth(pixelBuffer);
    +    m_uiHeight = (unsigned int)CVPixelBufferGetHeight(pixelBuffer);
     
         size_t extLeft, extRight, extTop, extBottom;
         CVPixelBufferGetExtendedPixels(pixelBuffer, &extLeft, &extRight, &extTop, &extBottom);
    -    m_iEncodedWidth = m_iWidth + (int)extLeft + (int)extRight;
    -    m_iEncodedHeight = m_iHeight + (int)extBottom; // ignore top, since 0,0 is where base addr starts
     
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    +    if (m_uiWidth <= (UINT_MAX - (unsigned int)extLeft) &&
    +          (m_uiWidth + (unsigned int)extLeft) <= (UINT_MAX - (unsigned int)extRight)) {
    +        m_uiEncodedWidth = m_uiWidth + (unsigned int)extLeft + (unsigned int)extRight;
    +    } else {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
    +
    +    if (m_uiHeight <= (UINT_MAX - (unsigned int)extBottom)) {
    +        // ignore top, since 0,0 is where base addr starts
    +        m_uiEncodedHeight = m_uiHeight + (unsigned int)extBottom;
    +    } else {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
    +
    +    Reset();
         m_FrameDirty = false;
     
         if (type != gLastFormat) {
    @@ -95,7 +106,8 @@
         }
     
         // Now retain the pixelBuffer so it doesn't go away
    -    CVPixelBufferRetain(pixelBuffer);
    +    m_bDisposePixelBuffer = true;
    +    CVPixelBufferRetain(m_pixelBuffer);
     }
     
     CVVideoFrame::~CVVideoFrame()
    @@ -105,39 +117,49 @@
     
     void CVVideoFrame::PrepareChunky()
     {
    -    m_iPlaneCount = 1;
    -    m_iWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    -    m_iHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    +    SetPlaneCount(1);
    +    m_uiWidth = (unsigned int)CVPixelBufferGetWidth(m_pixelBuffer);
    +    m_uiHeight = (unsigned int)CVPixelBufferGetHeight(m_pixelBuffer);
         m_bHasAlpha = (m_typeFrame == BGRA_PRE);
     
         // We MUST lock the base address during the lifetime of this object
         // or else we could cause a crash
    -    CVReturn cr = CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +    CVReturn cr = CVPixelBufferLockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
         if (kCVReturnSuccess != cr) {
             throw "CVVideoFrame: Unable to lock PixelBuffer base address";
         }
     
    -    m_pvPlaneData[0] = CVPixelBufferGetBaseAddress(pixelBuffer);
    -    m_piPlaneStrides[0] = (int)CVPixelBufferGetBytesPerRow(pixelBuffer);
    -    m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    +    m_pvPlaneData[0] = CVPixelBufferGetBaseAddress(m_pixelBuffer);
    +    m_puiPlaneStrides[0] = (int)CVPixelBufferGetBytesPerRow(m_pixelBuffer);
    +
    +    bool bValid = true; // CalcSize() requires bValid to be true when called
    +    m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &bValid);
    +    if (!bValid) {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
     }
     
     void CVVideoFrame::PreparePlanar()
     {
    -    m_iPlaneCount = (int)CVPixelBufferGetPlaneCount(pixelBuffer);
    +    SetPlaneCount((unsigned int)CVPixelBufferGetPlaneCount(m_pixelBuffer));
         m_bHasAlpha = false;
     
         // We MUST lock the base address during the lifetime of this object
         // or else we could cause a crash
    -    CVReturn cr = CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +    CVReturn cr = CVPixelBufferLockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
         if (kCVReturnSuccess != cr) {
             throw "CVVideoFrame: Unable to lock PixelBuffer base address";
         }
     
    -    for (int index = 0; index < m_iPlaneCount; index++) {
    -        m_piPlaneStrides[index] = (int)CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, index);
    -        m_pulPlaneSize[index] = CVPixelBufferGetHeightOfPlane(pixelBuffer, index) * m_piPlaneStrides[index];
    -        m_pvPlaneData[index] = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, index);
    +    bool bValid = true; // CalcSize() requires bValid to be true when called
    +    for (int index = 0; index < GetPlaneCount(); index++) {
    +        m_puiPlaneStrides[index] = (unsigned int)CVPixelBufferGetBytesPerRowOfPlane(m_pixelBuffer, index);
    +        unsigned long ulHeightOfPlane = (unsigned long)CVPixelBufferGetHeightOfPlane(m_pixelBuffer, index);
    +        m_pulPlaneSize[index] = CalcSize(ulHeightOfPlane, m_puiPlaneStrides[index], &bValid);
    +        if (!bValid) {
    +            throw "CVVideoFrame: Invalid frame size";
    +        }
    +        m_pvPlaneData[index] = CVPixelBufferGetBaseAddressOfPlane(m_pixelBuffer, index);
         }
     
         if (m_typeFrame == YCbCr_420p) {
    @@ -148,34 +170,35 @@
     
     void CVVideoFrame::Dispose()
     {
    -    if (pixelBuffer) {
    -        CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    -        CVPixelBufferRelease(pixelBuffer);
    -        pixelBuffer = 0;
    +    if (m_bDisposePixelBuffer) {
    +        CVPixelBufferUnlockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +        CVPixelBufferRelease(m_pixelBuffer);
    +        m_pixelBuffer = NULL;
    +        m_bDisposePixelBuffer = false;
         }
     }
     
     CVideoFrame *CVVideoFrame::ConvertToFormat(FrameType type)
     {
    -    if(YCbCr_422 == m_typeFrame && BGRA_PRE == type) {
    +    if (YCbCr_422 == m_typeFrame && BGRA_PRE == type) {
             CVPixelBufferRef destPixelBuffer = NULL;
    -        if(kCVReturnSuccess == CVPixelBufferCreate(NULL, m_iEncodedWidth, m_iEncodedHeight,
    +        if (kCVReturnSuccess == CVPixelBufferCreate(NULL, m_uiEncodedWidth, m_uiEncodedHeight,
                                                        k32BGRAPixelFormat, NULL, &destPixelBuffer)) {
    -            if(kCVReturnSuccess == CVPixelBufferLockBaseAddress(destPixelBuffer, 0)) {
    +            if (kCVReturnSuccess == CVPixelBufferLockBaseAddress(destPixelBuffer, 0)) {
                     uint8_t* bgra = (uint8_t*)CVPixelBufferGetBaseAddress(destPixelBuffer);
                     int32_t bgraStride = (int32_t)CVPixelBufferGetBytesPerRow(destPixelBuffer);
                     uint8_t* srcData = (uint8_t*)m_pvPlaneData[0];
     
    -                if(0 == ColorConvert_YCbCr422p_to_BGRA32_no_alpha(bgra,
    +                if (0 == ColorConvert_YCbCr422p_to_BGRA32_no_alpha(bgra,
                                                                       bgraStride,
    -                                                                  m_iEncodedWidth,
    -                                                                  m_iEncodedHeight,
    +                                                                  m_uiEncodedWidth,
    +                                                                  m_uiEncodedHeight,
                                                                       srcData + 1,
                                                                       srcData + 2,
                                                                       srcData,
    -                                                                  m_piPlaneStrides[0],
    -                                                                  m_piPlaneStrides[0])) {
    -                    return new CVVideoFrame(destPixelBuffer, m_dTime, frameHostTime);
    +                                                                  m_puiPlaneStrides[0],
    +                                                                  m_puiPlaneStrides[0])) {
    +                    return new CVVideoFrame(destPixelBuffer, m_dTime, m_frameHostTime);
                     }
                 }
             }
    
0a52a4cf1d12

8313105: Improved media framing

https://github.com/openjdk/jfxAlexander MatveevSep 7, 2023via ghsa
7 files changed · +362 190
  • modules/javafx.media/src/main/native/jfxmedia/jni/NativeVideoBuffer.cpp+3 3 modified
    @@ -71,8 +71,8 @@ JNIEXPORT jobject JNICALL Java_com_sun_media_jfxmediaimpl_NativeVideoBuffer_nati
     {
         CVideoFrame *frame = (CVideoFrame*)jlong_to_ptr(nativeHandle);
         if (frame) {
    -        void *dataPtr = frame->GetDataForPlane((int)plane);
    -        jlong capacity = (jlong)frame->GetSizeForPlane((int)plane);
    +        void *dataPtr = frame->GetDataForPlane((unsigned int)plane);
    +        jlong capacity = (jlong)frame->GetSizeForPlane((unsigned int)plane);
             jobject buffer = env->NewDirectByteBuffer(dataPtr, capacity);
             if (env->ExceptionCheck()) {
                 env->ExceptionClear();
    @@ -215,7 +215,7 @@ JNIEXPORT jintArray JNICALL Java_com_sun_media_jfxmediaimpl_NativeVideoBuffer_na
                 return NULL;
             }
     
    -        for (int ii=0; ii < count; ii++) {
    +        for (unsigned int ii=0; ii < count; ii++) {
                 strideArray[ii] = frame->GetStrideForPlane(ii);
             }
     
    
  • modules/javafx.media/src/main/native/jfxmedia/PipelineManagement/VideoFrame.cpp+103 32 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -23,50 +23,51 @@
      * questions.
      */
     
    +// For UINT_MAX, we cannot use GLib here, since it is shared code between
    +// GStreamer and AVFoundation.
    +#include <limits.h>
    +
     #include "VideoFrame.h"
     #include <Common/VSMemory.h>
     
     //*************************************************************************************************
     //********** class CVideoFrame
     //*************************************************************************************************
     CVideoFrame::CVideoFrame()
    -:   m_iWidth(0),
    -    m_iHeight(0),
    -    m_iEncodedWidth(0),
    -    m_iEncodedHeight(0),
    +:   m_uiWidth(0),
    +    m_uiHeight(0),
    +    m_uiEncodedWidth(0),
    +    m_uiEncodedHeight(0),
         m_typeFrame(UNKNOWN),
         m_bHasAlpha(false),
         m_dTime(0.0),
    -    m_FrameDirty(false),
    -    m_iPlaneCount(1)
    +    m_FrameDirty(false)
     {
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    +    Reset();
     }
     
     CVideoFrame::~CVideoFrame()
     {
     }
     
    -int CVideoFrame::GetWidth()
    +unsigned int CVideoFrame::GetWidth()
     {
    -    return m_iWidth;
    +    return m_uiWidth;
     }
     
    -int CVideoFrame::GetHeight()
    +unsigned int CVideoFrame::GetHeight()
     {
    -    return m_iHeight;
    +    return m_uiHeight;
     }
     
    -int CVideoFrame::GetEncodedWidth()
    +unsigned int CVideoFrame::GetEncodedWidth()
     {
    -    return m_iEncodedWidth;
    +    return m_uiEncodedWidth;
     }
     
    -int CVideoFrame::GetEncodedHeight()
    +unsigned int CVideoFrame::GetEncodedHeight()
     {
    -    return m_iEncodedHeight;
    +    return m_uiEncodedHeight;
     }
     
     CVideoFrame::FrameType CVideoFrame::GetType()
    @@ -84,31 +85,41 @@ double CVideoFrame::GetTime()
         return m_dTime;
     }
     
    -int CVideoFrame::GetPlaneCount()
    +unsigned int CVideoFrame::GetPlaneCount()
     {
    -    return m_iPlaneCount;
    +    return m_uiPlaneCount;
    +}
    +
    +void CVideoFrame::SetPlaneCount(unsigned int count)
    +{
    +    if (count <= MAX_PLANE_COUNT) {
    +        m_uiPlaneCount = count;
    +    } else {
    +        // Should never happen
    +        m_uiPlaneCount = MAX_PLANE_COUNT;
    +    }
     }
     
    -void* CVideoFrame::GetDataForPlane(int planeIndex)
    +void* CVideoFrame::GetDataForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    +    if (planeIndex < MAX_PLANE_COUNT) {
             return m_pvPlaneData[planeIndex];
         }
         return NULL;
     }
     
    -unsigned long CVideoFrame::GetSizeForPlane(int planeIndex)
    +unsigned long CVideoFrame::GetSizeForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    +    if (planeIndex < MAX_PLANE_COUNT) {
             return m_pulPlaneSize[planeIndex];
         }
         return 0;
     }
     
    -int CVideoFrame::GetStrideForPlane(int planeIndex)
    +unsigned int CVideoFrame::GetStrideForPlane(unsigned int planeIndex)
     {
    -    if (planeIndex < 4 && planeIndex >= 0) {
    -        return m_piPlaneStrides[planeIndex];
    +    if (planeIndex < MAX_PLANE_COUNT) {
    +        return m_puiPlaneStrides[planeIndex];
         }
         return 0;
     }
    @@ -118,12 +129,22 @@ CVideoFrame *CVideoFrame::ConvertToFormat(FrameType type)
         return NULL;
     }
     
    -void CVideoFrame::SwapPlanes(int aa, int bb)
    +void CVideoFrame::Reset()
    +{
    +    m_uiPlaneCount = 0;
    +    for (int i = 0; i < MAX_PLANE_COUNT; i++) {
    +        m_puiPlaneStrides[i] = 0;
    +        m_pulPlaneSize[i] = 0;
    +        m_pvPlaneData[i] = NULL;
    +    }
    +}
    +
    +void CVideoFrame::SwapPlanes(unsigned int aa, unsigned int bb)
     {
    -    if (aa != bb && aa >= 0 && aa < m_iPlaneCount && bb >= 0 && bb < m_iPlaneCount) {
    -        int stride = m_piPlaneStrides[aa];
    -        m_piPlaneStrides[aa] = m_piPlaneStrides[bb];
    -        m_piPlaneStrides[bb] = stride;
    +    if (aa != bb && aa < m_uiPlaneCount && bb < m_uiPlaneCount) {
    +        unsigned int stride = m_puiPlaneStrides[aa];
    +        m_puiPlaneStrides[aa] = m_puiPlaneStrides[bb];
    +        m_puiPlaneStrides[bb] = stride;
     
             unsigned long size = m_pulPlaneSize[aa];
             m_pulPlaneSize[aa] = m_pulPlaneSize[bb];
    @@ -134,3 +155,53 @@ void CVideoFrame::SwapPlanes(int aa, int bb)
             m_pvPlaneData[bb] = vptr;
         }
     }
    +
    +unsigned long CVideoFrame::CalcSize(unsigned int a, unsigned int b, bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return 0;
    +    }
    +
    +    if (b > 0 && a <= (UINT_MAX / b)) {
    +        return (a * b);
    +    }
    +
    +    *(pbValid) = false;
    +    return 0;
    +}
    +
    +unsigned long CVideoFrame::AddSize(unsigned long a, unsigned long b, bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return 0;
    +    }
    +
    +    // unsigned long can be 32-bit or 64-bit, make sure it is no more then UINT_MAX
    +    if (a <= UINT_MAX && b <= UINT_MAX && a <= (UINT_MAX - b)) {
    +        return (a + b);
    +    }
    +
    +    *(pbValid) = false;
    +    return 0;
    +}
    +
    +void* CVideoFrame::CalcPlanePointer(intptr_t baseAddress, unsigned int offset,
    +                                    unsigned long planeSize, unsigned long baseSize,
    +                                    bool *pbValid)
    +{
    +    if (pbValid == NULL || *(pbValid) == false) {
    +        return NULL;
    +    }
    +
    +    // We will read planeSize bytes from baseAddress starting with offset, so
    +    // make sure we do not read pass baseSize.
    +    unsigned long endOfPlane = AddSize(offset, planeSize, pbValid);
    +    if (*(pbValid)) { // Make sure AddSize() did not failed.
    +        if (endOfPlane <= baseSize) {
    +            return (void*)(baseAddress + offset);
    +        }
    +    }
    +
    +    *(pbValid) = false;
    +    return NULL;
    +}
    
  • modules/javafx.media/src/main/native/jfxmedia/PipelineManagement/VideoFrame.h+41 18 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -27,6 +27,9 @@
     #define _VIDEO_FRAME_H_
     
     #include <stdlib.h>
    +#include <stdint.h>
    +
    +#define MAX_PLANE_COUNT 4
     
     /**
      * class CVideoFrame
    @@ -56,41 +59,61 @@ class CVideoFrame
     
         double              GetTime();
     
    -    int                 GetWidth();
    -    int                 GetHeight();
    -    int                 GetEncodedWidth();
    -    int                 GetEncodedHeight();
    +    unsigned int        GetWidth();
    +    unsigned int        GetHeight();
    +    unsigned int        GetEncodedWidth();
    +    unsigned int        GetEncodedHeight();
     
         FrameType           GetType();
         bool                HasAlpha();
     
    -    int                 GetPlaneCount();
    -    void*               GetDataForPlane(int planeIndex);
    -    unsigned long       GetSizeForPlane(int planeIndex);
    -    int                 GetStrideForPlane(int planeIndex);
    +    unsigned int        GetPlaneCount();
    +    void                SetPlaneCount(unsigned int count);
    +    void*               GetDataForPlane(unsigned int planeIndex);
    +    unsigned long       GetSizeForPlane(unsigned int planeIndex);
    +    unsigned int        GetStrideForPlane(unsigned int planeIndex);
     
         virtual CVideoFrame *ConvertToFormat(FrameType type);
     
         bool                GetFrameDirty() { return m_FrameDirty; }
         void                SetFrameDirty(bool dirty) { m_FrameDirty = dirty; }
     
     protected:
    -    int                 m_iWidth;
    -    int                 m_iHeight;
    -    int                 m_iEncodedWidth;
    -    int                 m_iEncodedHeight;
    +    unsigned int        m_uiWidth;
    +    unsigned int        m_uiHeight;
    +    unsigned int        m_uiEncodedWidth;
    +    unsigned int        m_uiEncodedHeight;
         FrameType           m_typeFrame;
         bool                m_bHasAlpha;
         double              m_dTime;
         bool                m_FrameDirty;
     
         // frame data buffers
    -    int                 m_iPlaneCount;
    -    void*               m_pvPlaneData[4];
    -    unsigned long       m_pulPlaneSize[4];
    -    int                 m_piPlaneStrides[4];
    +    void*               m_pvPlaneData[MAX_PLANE_COUNT];
    +    unsigned long       m_pulPlaneSize[MAX_PLANE_COUNT];
    +    unsigned int        m_puiPlaneStrides[MAX_PLANE_COUNT];
    +
    +    void Reset();
    +    void SwapPlanes(unsigned int aa, unsigned int bb);
    +
    +    // CalcSize(), AddSize(), CalcPlanePointer() requires bValid to be set to
    +    // true initially, if bValid is false these functions do nothing. It is
    +    // implemented this way, so all these functions can be chain called without
    +    // checking bValid after each call. bValid will be set to false only if
    +    // calculation failed and will never be set to true.
    +    // Multiplies a and b, bValid set to false if integer overflow detected.
    +    unsigned long CalcSize(unsigned int a, unsigned int b, bool *pbValid);
    +    // Adds a and b, bValid set to false if integer overflow detected.
    +    unsigned long AddSize(unsigned long a, unsigned long b, bool *pbValid);
    +    // Calculates plane pointer (baseAddress + offset) and checks that calculated
    +    // pointer within buffer. Returns NULL and sets bValid to false if calculated
    +    // pointer is invalid.
    +    void* CalcPlanePointer(intptr_t baseAddress, unsigned int offset,
    +                           unsigned long planeSize, unsigned long baseSize,
    +                           bool *pbValid);
     
    -    void SwapPlanes(int aa, int bb);
    +private:
    +    unsigned int        m_uiPlaneCount;
     };
     
     #endif  //_VIDEO_FRAME_H_
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/gstreamer/GstVideoFrame.cpp+144 94 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -56,18 +56,23 @@ static GstBuffer *alloc_aligned_buffer(guint size)
     
         // allocate a buffer large enough to accommodate 16 byte alignment
         alignedSize = size;
    -    size += 16;
    +    if (size <= (G_MAXUINT - 16)) {
    +        size += 16;
    +    } else {
    +        return NULL;
    +    }
    +
         newData = (guint8*)g_try_malloc(size);
         if (NULL == newData) {
             return NULL;
         }
     
         alignedData = (guint8*)(((intptr_t)newData + 15) & ~15);
     
    -    return gst_buffer_new_wrapped_full((GstMemoryFlags)0, alignedData, alignedSize, 0, 0, newData, free_aligned_buffer);
    +    return gst_buffer_new_wrapped_full((GstMemoryFlags)0, alignedData, alignedSize, 0, alignedSize, newData, free_aligned_buffer);
     }
     
    -GstCaps *create_RGB_caps(CVideoFrame::FrameType type, gint width, gint height, gint encodedWidth, gint encodedHeight, gint stride)
    +GstCaps *create_RGB_caps(CVideoFrame::FrameType type, guint width, guint height, guint encodedWidth, guint encodedHeight, guint stride)
     {
         gint red_mask, green_mask, blue_mask, alpha_mask;
         GstCaps *newCaps;
    @@ -160,7 +165,8 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
         const GstStructure* str = gst_caps_get_structure(newCaps, 0);
         const gchar* sFormatFourCC = gst_structure_get_string(str, "format");
     
    -    // default to success
    +    // We should always start with success. See CalcSize(), AddSize() and
    +    // CalcPlanePointer() on how this flag is being used.
         m_bIsValid = true;
     
         // FIXME: make format type strings conformant with constant types
    @@ -209,83 +215,79 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
             m_bIsValid = false;
         }
     
    -    if(!gst_structure_get_int(str, "width", &m_iWidth))
    +    if (!gst_structure_get_int(str, "width", (int*)&m_uiWidth))
         {
     #if JFXMEDIA_DEBUG
             g_warning("width could not be retrieved from GstBuffer\n");
     #endif
    -        m_iWidth = 0;
    +        m_uiWidth = 0;
             m_bIsValid = false;
         }
    -    if(!gst_structure_get_int(str, "height", &m_iHeight))
    +    if (!gst_structure_get_int(str, "height", (int*)&m_uiHeight))
         {
     #if JFXMEDIA_DEBUG
             g_warning("height could not be retrieved from GstBuffer\n");
     #endif
    -        m_iHeight = 0;
    +        m_uiHeight = 0;
             m_bIsValid = false;
         }
     
    -    if (!gst_structure_get_int(str, "encoded-width", &m_iEncodedWidth)) {
    -        m_iEncodedWidth = m_iWidth;
    +    if (!gst_structure_get_int(str, "encoded-width", (int*)&m_uiEncodedWidth)) {
    +        m_uiEncodedWidth = m_uiWidth;
         }
    -    if (!gst_structure_get_int(str, "encoded-height", &m_iEncodedHeight)) {
    -        m_iEncodedHeight = m_iHeight;
    +    if (!gst_structure_get_int(str, "encoded-height", (int*)&m_uiEncodedHeight)) {
    +        m_uiEncodedHeight = m_uiHeight;
         }
     
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    +    Reset();
     
    -    unsigned long expectedSize = 0;
         switch (m_typeFrame) {
             case YCbCr_420p: {
    -            int offset;
    -            m_iPlaneCount = 3;
    +            unsigned int offset = 0;
    +            SetPlaneCount(3);
     
    -            if (!gst_structure_get_int(str, "stride-y", &m_piPlaneStrides[0])) {
    -                m_piPlaneStrides[0] = m_iEncodedWidth;
    +            if (!gst_structure_get_int(str, "stride-y", (int*)&m_puiPlaneStrides[0])) {
    +                m_puiPlaneStrides[0] = m_uiEncodedWidth;
                 }
    -            if (!gst_structure_get_int(str, "stride-v", &m_piPlaneStrides[1])) {
    -                m_piPlaneStrides[1] = m_iEncodedWidth/2;
    +            if (!gst_structure_get_int(str, "stride-v", (int*)&m_puiPlaneStrides[1])) {
    +                m_puiPlaneStrides[1] = m_uiEncodedWidth/2;
                 }
    -            if (!gst_structure_get_int(str, "stride-u", &m_piPlaneStrides[2])) {
    -                m_piPlaneStrides[2] = m_piPlaneStrides[1];
    +            if (!gst_structure_get_int(str, "stride-u", (int*)&m_puiPlaneStrides[2])) {
    +                m_puiPlaneStrides[2] = m_puiPlaneStrides[1];
                 }
     
    -            offset = 0;
    -            gst_structure_get_int(str, "offset-y", &offset);
    -            m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    -            m_pvPlaneData[0] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[0];
    +            gst_structure_get_int(str, "offset-y", (int*)&offset);
    +            m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &m_bIsValid);
    +            m_pvPlaneData[0] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[0], m_ulBufferSize, &m_bIsValid);
     
                 //
                 // Chroma offsets assume YV12 ordering
                 //
                 offset += m_pulPlaneSize[0];
    -            gst_structure_get_int(str, "offset-v", &offset);
    -            m_pulPlaneSize[1] = m_piPlaneStrides[1] * (m_iEncodedHeight/2);
    -            m_pvPlaneData[1] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[1];
    +            gst_structure_get_int(str, "offset-v", (int*)&offset);
    +            m_pulPlaneSize[1] = CalcSize(m_puiPlaneStrides[1], (m_uiEncodedHeight/2), &m_bIsValid);
    +            m_pvPlaneData[1] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[1], m_ulBufferSize, &m_bIsValid);
     
                 offset += m_pulPlaneSize[1];
    -            gst_structure_get_int(str, "offset-u", &offset);
    -            m_pulPlaneSize[2] = m_piPlaneStrides[2] * (m_iEncodedHeight/2);
    -            m_pvPlaneData[2] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -            expectedSize += m_pulPlaneSize[2];
    +            gst_structure_get_int(str, "offset-u", (int*)&offset);
    +            m_pulPlaneSize[2] = CalcSize(m_puiPlaneStrides[2], (m_uiEncodedHeight/2), &m_bIsValid);
    +            m_pvPlaneData[2] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[2], m_ulBufferSize, &m_bIsValid);
     
                 // process alpha channel (before we potentially swap Cb/Cr)
                 if (m_bHasAlpha) {
    -                m_iPlaneCount++;
    -                if (!gst_structure_get_int(str, "stride-a", &m_piPlaneStrides[3])) {
    -                    m_piPlaneStrides[3] = m_piPlaneStrides[0];
    +                SetPlaneCount(GetPlaneCount() + 1);
    +                if (!gst_structure_get_int(str, "stride-a", (int*)&m_puiPlaneStrides[3])) {
    +                    m_puiPlaneStrides[3] = m_puiPlaneStrides[0];
                     }
     
                     offset += m_pulPlaneSize[2];
    -                gst_structure_get_int(str, "offset-a", &offset);
    -                m_pulPlaneSize[3] = m_piPlaneStrides[3] * m_iEncodedHeight;
    -                m_pvPlaneData[3] = (void*)((intptr_t)m_pvBufferBaseAddress + offset);
    -                expectedSize += m_pulPlaneSize[3];
    +                gst_structure_get_int(str, "offset-a", (int*)&offset);
    +                m_pulPlaneSize[3] = CalcSize(m_puiPlaneStrides[3], m_uiEncodedHeight, &m_bIsValid);
    +                m_pvPlaneData[3] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, offset,
    +                                                m_pulPlaneSize[3], m_ulBufferSize, &m_bIsValid);
                 }
     
                 //
    @@ -299,21 +301,19 @@ void CGstVideoFrame::SetFrameCaps(GstCaps *newCaps)
             }
     
             default:
    -            m_iPlaneCount = 1;
    -            if (!gst_structure_get_int(str, "line_stride", &m_piPlaneStrides[0])) {
    +            SetPlaneCount(1);
    +            if (!gst_structure_get_int(str, "line_stride", (int*)&m_puiPlaneStrides[0])) {
                     if (m_typeFrame == YCbCr_422) {
    -                    m_piPlaneStrides[0] = m_iEncodedWidth * 2; // 16 bpp
    +                    m_puiPlaneStrides[0] = m_uiEncodedWidth * 2; // 16 bpp
                     } else {
    -                    m_piPlaneStrides[0] = m_iEncodedWidth * 4; // 32 bpp
    +                    m_puiPlaneStrides[0] = m_uiEncodedWidth * 4; // 32 bpp
                     }
                 }
    -            m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    -            m_pvPlaneData[0] = m_pvBufferBaseAddress;
    -            expectedSize += m_pulPlaneSize[0];
    +            m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &m_bIsValid);
    +            m_pvPlaneData[0] = CalcPlanePointer((intptr_t)m_pvBufferBaseAddress, 0,
    +                                                m_pulPlaneSize[0], m_ulBufferSize, &m_bIsValid);
                 break;
         }
    -
    -    m_bIsValid = m_bIsValid && (expectedSize <= m_ulBufferSize);
     }
     
     bool CGstVideoFrame::IsValid()
    @@ -381,9 +381,10 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
         GstBuffer *destBuffer = NULL;
         GstCaps *destCaps = NULL;
         GstMapInfo info;
    -    gint stride = m_iEncodedWidth * 4;
    -    int u_index, v_index;
    -    int status;
    +    guint stride = 0;
    +    guint alloc_size = 0;
    +    unsigned int u_index, v_index = 0;
    +    int status = 0;
     
         if (m_bIsI420) {
             u_index = 1;
    @@ -393,8 +394,26 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
             v_index = 1;
         }
     
    -    stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    -    destBuffer = alloc_aligned_buffer(stride * m_iEncodedHeight);
    +    // Make sure we do not have an integer overflow
    +    if (m_uiEncodedWidth <= (G_MAXUINT / 4)) {
    +        stride = m_uiEncodedWidth * 4;
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (stride <= (G_MAXUINT - 16)) {
    +        stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (m_uiEncodedHeight > 0 && stride <= (G_MAXUINT / m_uiEncodedHeight)) {
    +        alloc_size = stride * m_uiEncodedHeight;
    +    } else {
    +        return NULL;
    +    }
    +
    +    destBuffer = alloc_aligned_buffer(alloc_size);
         if (!destBuffer) {
             return NULL;
         }
    @@ -415,49 +434,49 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
             if (m_bHasAlpha) {
                 status = ColorConvert_YCbCr420p_to_ARGB32(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
                             (const uint8_t*)m_pvPlaneData[3],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index], m_piPlaneStrides[3]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index], m_puiPlaneStrides[3]);
             } else {
                 status = ColorConvert_YCbCr420p_to_ARGB32_no_alpha(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index]);
             }
         } else {
             if (m_bHasAlpha) {
                 status = ColorConvert_YCbCr420p_to_BGRA32(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
                             (const uint8_t*)m_pvPlaneData[3],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index], m_piPlaneStrides[3]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index], m_puiPlaneStrides[3]);
             } else {
                 status = ColorConvert_YCbCr420p_to_BGRA32_no_alpha(
                             info.data, stride,
    -                        m_iEncodedWidth, m_iEncodedHeight,
    +                        m_uiEncodedWidth, m_uiEncodedHeight,
                             (const uint8_t*)m_pvPlaneData[0],
                             (const uint8_t*)m_pvPlaneData[v_index],
                             (const uint8_t*)m_pvPlaneData[u_index],
    -                        m_piPlaneStrides[0], m_piPlaneStrides[v_index],
    -                        m_piPlaneStrides[u_index]);
    +                        m_puiPlaneStrides[0], m_puiPlaneStrides[v_index],
    +                        m_puiPlaneStrides[u_index]);
             }
         }
     
         gst_buffer_unmap(destBuffer, &info);
     
    -    destCaps = create_RGB_caps(destType, m_iWidth, m_iHeight, m_iEncodedWidth, m_iEncodedHeight, stride);
    +    destCaps = create_RGB_caps(destType, m_uiWidth, m_uiHeight, m_uiEncodedWidth, m_uiEncodedHeight, stride);
         if (!destCaps) {
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer);
    @@ -476,12 +495,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr420p(FrameType destType)
     
         if (0 == status && destSample) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_sample_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
     
         return NULL;
    @@ -493,16 +516,35 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
         GstBuffer *destBuffer;
         GstCaps *destCaps;
         GstMapInfo info;
    -    gint stride = m_iEncodedWidth * 4;
    -    int status = 1;
    +    guint stride = 0;
    +    guint alloc_size = 0;
    +    int status = 0;
     
         // Not handling alpha ...
         if (m_bHasAlpha) {
             return NULL;
         }
     
    -    stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    -    destBuffer = alloc_aligned_buffer(stride * m_iEncodedHeight);
    +    // Make sure we do not have an integer overflow
    +    if (m_uiEncodedWidth <= (G_MAXUINT / 4)) {
    +        stride = m_uiEncodedWidth * 4;
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (stride <= (G_MAXUINT - 16)) {
    +        stride = ((stride + 15) & ~15); // round up to multiple of 16 bytes
    +    } else {
    +        return NULL;
    +    }
    +
    +    if (m_uiEncodedHeight > 0 && stride <= (G_MAXUINT / m_uiEncodedHeight)) {
    +        alloc_size = stride * m_uiEncodedHeight;
    +    } else {
    +        return NULL;
    +    }
    +
    +    destBuffer = alloc_aligned_buffer(alloc_size);
         if (!destBuffer) {
             return NULL;
         }
    @@ -521,23 +563,23 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
         // now do the conversion
         if (destType == ARGB) {
             status = ColorConvert_YCbCr422p_to_ARGB32_no_alpha(info.data, stride,
    -                                                           m_iEncodedWidth, m_iEncodedHeight,
    +                                                           m_uiEncodedWidth, m_uiEncodedHeight,
                                                                (uint8_t*)m_pvPlaneData[0] + 1,
                                                                (uint8_t*)m_pvPlaneData[0] + 2,
                                                                (uint8_t*)m_pvPlaneData[0],
    -                                                           m_piPlaneStrides[0], m_piPlaneStrides[0]);
    +                                                           m_puiPlaneStrides[0], m_puiPlaneStrides[0]);
         } else {
             status = ColorConvert_YCbCr422p_to_BGRA32_no_alpha(info.data, stride,
    -                                                           m_iEncodedWidth, m_iEncodedHeight,
    +                                                           m_uiEncodedWidth, m_uiEncodedHeight,
                                                                (uint8_t*)m_pvPlaneData[0] + 1,
                                                                (uint8_t*)m_pvPlaneData[0] + 2,
                                                                (uint8_t*)m_pvPlaneData[0],
    -                                                           m_piPlaneStrides[0], m_piPlaneStrides[0]);
    +                                                           m_puiPlaneStrides[0], m_puiPlaneStrides[0]);
         }
     
         gst_buffer_unmap(destBuffer, &info);
     
    -    destCaps = create_RGB_caps(destType, m_iWidth, m_iHeight, m_iEncodedWidth, m_iEncodedHeight, stride);
    +    destCaps = create_RGB_caps(destType, m_uiWidth, m_uiHeight, m_uiEncodedWidth, m_uiEncodedHeight, stride);
         if (!destCaps) {
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer);
    @@ -556,12 +598,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertFromYCbCr422(FrameType destType)
     
         if (0 == status && destBuffer) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
     
         return NULL;
    @@ -574,7 +620,7 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
         GstCaps *srcCaps, *dstCaps;
         GstMapInfo srcInfo, destInfo;
         GstStructure* str;
    -    gint xx, yy, size;
    +    guint xx, yy, size;
         guint32 *srcData, *dstData;
     
         size = gst_buffer_get_size(m_pBuffer);
    @@ -610,7 +656,7 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
                 break;
             default:
                 // shouldn't have gotten this far...
    -// INLINE - gst_buffer_unref()
    +            // INLINE - gst_buffer_unref()
                 gst_buffer_unref(destBuffer);
                 gst_caps_unref(dstCaps);
                 return NULL;
    @@ -646,18 +692,18 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
         // Now copy data from src to dest, byteswapping as we copy
         srcData = (guint32*)srcInfo.data;
         dstData = (guint32*)destInfo.data;
    -    if (!(m_piPlaneStrides[0] & 3)) {
    +    if (!(m_puiPlaneStrides[0] & 3)) {
             // four byte alignment on the entire buffer, we can just loop once
             for (xx = 0; xx < size; xx += 4) {
                 *dstData++ = swap_uint32(*srcData++); // NOTE: SSE could be used here instead
             }
         } else {
    -        for (yy = 0; yy < m_iHeight; yy++) {
    -            for (xx = 0; xx < m_iWidth; xx++) {
    +        for (yy = 0; yy < m_uiHeight; yy++) {
    +            for (xx = 0; xx < m_uiWidth; xx++) {
                     dstData[xx] = swap_uint32(srcData[xx]); // NOTE: SSE could be used here instead
                 }
    -            dstData += m_piPlaneStrides[0];
    -            srcData += m_piPlaneStrides[0];
    +            dstData += m_puiPlaneStrides[0];
    +            srcData += m_puiPlaneStrides[0];
             }
         }
     
    @@ -666,12 +712,16 @@ CGstVideoFrame *CGstVideoFrame::ConvertSwapRGB(FrameType destType)
     
         if (destBuffer) {
             CGstVideoFrame *newFrame = new CGstVideoFrame();
    -        bool result = newFrame->Init(destSample);
    +        bool result = newFrame->Init(destSample) && newFrame->IsValid();
             // INLINE - gst_buffer_unref()
             gst_buffer_unref(destBuffer); // else we'll have a massive memory leak!
             // INLINE - gst_sample_unref()
             gst_sample_unref(destSample); // else we'll have a massive memory leak!
    -        return result ? newFrame : NULL;
    +        if (result) {
    +            return newFrame;
    +        } else {
    +            delete newFrame;
    +        }
         }
         return NULL;
     }
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/avf/AVFMediaPlayer.mm+5 1 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2014, 2022, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -704,6 +704,10 @@ - (void) sendPixelBuffer:(CVPixelBufferRef)buf frameTime:(double)frameTime hostT
             return;
         }
     
    +    if (frame == NULL) {
    +        return;
    +    }
    +
         if (previousWidth < 0 || previousHeight < 0
             || previousWidth != frame->GetWidth() || previousHeight != frame->GetHeight())
         {
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/CVVideoFrame.h+4 3 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2014, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -40,8 +40,9 @@ class CVVideoFrame : public CVideoFrame {
         virtual CVideoFrame *ConvertToFormat(FrameType type);
     
     private:
    -    CVPixelBufferRef pixelBuffer;
    -    uint64_t frameHostTime;
    +    bool m_bDisposePixelBuffer;
    +    CVPixelBufferRef m_pixelBuffer;
    +    uint64_t m_frameHostTime;
     
         void PrepareChunky();
         void PreparePlanar();
    
  • modules/javafx.media/src/main/native/jfxmedia/platform/osx/CVVideoFrame.mm+62 39 modified
    @@ -1,5 +1,5 @@
     /*
    - * Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2010, 2023, Oracle and/or its affiliates. All rights reserved.
      * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
      *
      * This code is free software; you can redistribute it and/or modify it
    @@ -39,32 +39,43 @@
         return false;
     }
     
    -CVVideoFrame::CVVideoFrame(CVPixelBufferRef buf, double frameTime, uint64_t frameHostTime)
    +CVVideoFrame::CVVideoFrame(CVPixelBufferRef pixelBuffer, double frameTime, uint64_t frameHostTime)
     :
    -  pixelBuffer(buf),
    -  frameHostTime(frameHostTime)
    +  m_bDisposePixelBuffer(false),
    +  m_pixelBuffer(pixelBuffer)
     {
         // We can assume that buf is retained at least for the duration of this ctor
         // So postpone retaining it until we're absolutely sure we'll use it
     
         // fail fast
    -    OSType type = CVPixelBufferGetPixelFormatType(buf);
    +    OSType type = CVPixelBufferGetPixelFormatType(pixelBuffer);
         if (!IsFormatSupported(type)) {
             throw "CVVideoFrame: Invalid PixelFormat";
         }
     
         m_dTime = frameTime;
    -    m_iWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    -    m_iHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    +    m_frameHostTime = frameHostTime;
    +    m_uiWidth = (unsigned int)CVPixelBufferGetWidth(pixelBuffer);
    +    m_uiHeight = (unsigned int)CVPixelBufferGetHeight(pixelBuffer);
     
         size_t extLeft, extRight, extTop, extBottom;
         CVPixelBufferGetExtendedPixels(pixelBuffer, &extLeft, &extRight, &extTop, &extBottom);
    -    m_iEncodedWidth = m_iWidth + (int)extLeft + (int)extRight;
    -    m_iEncodedHeight = m_iHeight + (int)extBottom; // ignore top, since 0,0 is where base addr starts
     
    -    m_pvPlaneData[0] = m_pvPlaneData[1] = m_pvPlaneData[2] = m_pvPlaneData[3] = NULL;
    -    m_piPlaneStrides[0] = m_piPlaneStrides[1] = m_piPlaneStrides[2] = m_piPlaneStrides[3] = 0;
    -    m_pulPlaneSize[0] = m_pulPlaneSize[1] = m_pulPlaneSize[2] = m_pulPlaneSize[3] = 0;
    +    if (m_uiWidth <= (UINT_MAX - (unsigned int)extLeft) &&
    +          (m_uiWidth + (unsigned int)extLeft) <= (UINT_MAX - (unsigned int)extRight)) {
    +        m_uiEncodedWidth = m_uiWidth + (unsigned int)extLeft + (unsigned int)extRight;
    +    } else {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
    +
    +    if (m_uiHeight <= (UINT_MAX - (unsigned int)extBottom)) {
    +        // ignore top, since 0,0 is where base addr starts
    +        m_uiEncodedHeight = m_uiHeight + (unsigned int)extBottom;
    +    } else {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
    +
    +    Reset();
         m_FrameDirty = false;
     
         if (type != gLastFormat) {
    @@ -95,7 +106,8 @@
         }
     
         // Now retain the pixelBuffer so it doesn't go away
    -    CVPixelBufferRetain(pixelBuffer);
    +    m_bDisposePixelBuffer = true;
    +    CVPixelBufferRetain(m_pixelBuffer);
     }
     
     CVVideoFrame::~CVVideoFrame()
    @@ -105,39 +117,49 @@
     
     void CVVideoFrame::PrepareChunky()
     {
    -    m_iPlaneCount = 1;
    -    m_iWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    -    m_iHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    +    SetPlaneCount(1);
    +    m_uiWidth = (unsigned int)CVPixelBufferGetWidth(m_pixelBuffer);
    +    m_uiHeight = (unsigned int)CVPixelBufferGetHeight(m_pixelBuffer);
         m_bHasAlpha = (m_typeFrame == BGRA_PRE);
     
         // We MUST lock the base address during the lifetime of this object
         // or else we could cause a crash
    -    CVReturn cr = CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +    CVReturn cr = CVPixelBufferLockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
         if (kCVReturnSuccess != cr) {
             throw "CVVideoFrame: Unable to lock PixelBuffer base address";
         }
     
    -    m_pvPlaneData[0] = CVPixelBufferGetBaseAddress(pixelBuffer);
    -    m_piPlaneStrides[0] = (int)CVPixelBufferGetBytesPerRow(pixelBuffer);
    -    m_pulPlaneSize[0] = m_piPlaneStrides[0] * m_iEncodedHeight;
    +    m_pvPlaneData[0] = CVPixelBufferGetBaseAddress(m_pixelBuffer);
    +    m_puiPlaneStrides[0] = (int)CVPixelBufferGetBytesPerRow(m_pixelBuffer);
    +
    +    bool bValid = true; // CalcSize() requires bValid to be true when called
    +    m_pulPlaneSize[0] = CalcSize(m_puiPlaneStrides[0], m_uiEncodedHeight, &bValid);
    +    if (!bValid) {
    +        throw "CVVideoFrame: Invalid frame size";
    +    }
     }
     
     void CVVideoFrame::PreparePlanar()
     {
    -    m_iPlaneCount = (int)CVPixelBufferGetPlaneCount(pixelBuffer);
    +    SetPlaneCount((unsigned int)CVPixelBufferGetPlaneCount(m_pixelBuffer));
         m_bHasAlpha = false;
     
         // We MUST lock the base address during the lifetime of this object
         // or else we could cause a crash
    -    CVReturn cr = CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +    CVReturn cr = CVPixelBufferLockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
         if (kCVReturnSuccess != cr) {
             throw "CVVideoFrame: Unable to lock PixelBuffer base address";
         }
     
    -    for (int index = 0; index < m_iPlaneCount; index++) {
    -        m_piPlaneStrides[index] = (int)CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, index);
    -        m_pulPlaneSize[index] = CVPixelBufferGetHeightOfPlane(pixelBuffer, index) * m_piPlaneStrides[index];
    -        m_pvPlaneData[index] = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, index);
    +    bool bValid = true; // CalcSize() requires bValid to be true when called
    +    for (int index = 0; index < GetPlaneCount(); index++) {
    +        m_puiPlaneStrides[index] = (unsigned int)CVPixelBufferGetBytesPerRowOfPlane(m_pixelBuffer, index);
    +        unsigned long ulHeightOfPlane = (unsigned long)CVPixelBufferGetHeightOfPlane(m_pixelBuffer, index);
    +        m_pulPlaneSize[index] = CalcSize(ulHeightOfPlane, m_puiPlaneStrides[index], &bValid);
    +        if (!bValid) {
    +            throw "CVVideoFrame: Invalid frame size";
    +        }
    +        m_pvPlaneData[index] = CVPixelBufferGetBaseAddressOfPlane(m_pixelBuffer, index);
         }
     
         if (m_typeFrame == YCbCr_420p) {
    @@ -148,34 +170,35 @@
     
     void CVVideoFrame::Dispose()
     {
    -    if (pixelBuffer) {
    -        CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
    -        CVPixelBufferRelease(pixelBuffer);
    -        pixelBuffer = 0;
    +    if (m_bDisposePixelBuffer) {
    +        CVPixelBufferUnlockBaseAddress(m_pixelBuffer, kCVPixelBufferLock_ReadOnly);
    +        CVPixelBufferRelease(m_pixelBuffer);
    +        m_pixelBuffer = NULL;
    +        m_bDisposePixelBuffer = false;
         }
     }
     
     CVideoFrame *CVVideoFrame::ConvertToFormat(FrameType type)
     {
    -    if(YCbCr_422 == m_typeFrame && BGRA_PRE == type) {
    +    if (YCbCr_422 == m_typeFrame && BGRA_PRE == type) {
             CVPixelBufferRef destPixelBuffer = NULL;
    -        if(kCVReturnSuccess == CVPixelBufferCreate(NULL, m_iEncodedWidth, m_iEncodedHeight,
    +        if (kCVReturnSuccess == CVPixelBufferCreate(NULL, m_uiEncodedWidth, m_uiEncodedHeight,
                                                        k32BGRAPixelFormat, NULL, &destPixelBuffer)) {
    -            if(kCVReturnSuccess == CVPixelBufferLockBaseAddress(destPixelBuffer, 0)) {
    +            if (kCVReturnSuccess == CVPixelBufferLockBaseAddress(destPixelBuffer, 0)) {
                     uint8_t* bgra = (uint8_t*)CVPixelBufferGetBaseAddress(destPixelBuffer);
                     int32_t bgraStride = (int32_t)CVPixelBufferGetBytesPerRow(destPixelBuffer);
                     uint8_t* srcData = (uint8_t*)m_pvPlaneData[0];
     
    -                if(0 == ColorConvert_YCbCr422p_to_BGRA32_no_alpha(bgra,
    +                if (0 == ColorConvert_YCbCr422p_to_BGRA32_no_alpha(bgra,
                                                                       bgraStride,
    -                                                                  m_iEncodedWidth,
    -                                                                  m_iEncodedHeight,
    +                                                                  m_uiEncodedWidth,
    +                                                                  m_uiEncodedHeight,
                                                                       srcData + 1,
                                                                       srcData + 2,
                                                                       srcData,
    -                                                                  m_piPlaneStrides[0],
    -                                                                  m_piPlaneStrides[0])) {
    -                    return new CVVideoFrame(destPixelBuffer, m_dTime, frameHostTime);
    +                                                                  m_puiPlaneStrides[0],
    +                                                                  m_puiPlaneStrides[0])) {
    +                    return new CVVideoFrame(destPixelBuffer, m_dTime, m_frameHostTime);
                     }
                 }
             }
    

Vulnerability mechanics

Generated on May 9, 2026. Inputs: CWE entries + fix-commit diffs from this CVE's patches. Citations validated against bundle.

References

9

News mentions

0

No linked articles in our index yet.