VK_QCOM_image_processing2

Other Extension Metadata

Last Modified Date

2023-03-10

Interactions and External Dependencies

• This extension provides API support for GL_QCOM_image_processing2

Contributors

• Jeff Leger, Qualcomm Technologies, Inc.

Description

This extension enables support for the SPIR-V TextureBlockMatch2QCOM capability. It builds on the functionality of QCOM_image_processing with the addition of 4 new image processing operations.

• The opImageBlockMatchWindowSADQCOM` SPIR-V instruction builds upon the functionality of opImageBlockMatchSADQCOM` by repeatedly performing block match operations across a 2D window. The 2D windowExtent and compareMode are specified by VkSamplerBlockMatchWindowCreateInfoQCOM in the sampler used to create the target image. Like OpImageBlockMatchSADQCOM, opImageBlockMatchWindowSADQCOM computes an error metric, that describes whether a block of texels in the target image matches a corresponding block of texels in the reference image. Unlike OpImageBlockMatchSADQCOM, this instruction computes an error metric at each (X,Y) location within the 2D window and returns either the minimum or maximum error. The instruction only supports single-component formats. Refer to the pseudocode below for details.
• The opImageBlockMatchWindowSSDQCOM follows the same pattern, computing the SSD error metric at each location within the 2D window.
• The opImageBlockMatchGatherSADQCOM builds upon OpImageBlockMatchSADQCOM. This instruction computes an error metric, that describes whether a block of texels in the target image matches a corresponding block of texels in the reference image. The instruction computes the SAD error metric at 4 texel offsets and returns the error metric for each offset in the X,Y,Z,and W components. The instruction only supports single-component texture formats. Refer to the pseudocode below for details.
• The opImageBlockMatchGatherSSDQCOM follows the same pattern, computing the SSD error metric for 4 offsets.

Each of the above 4 image processing instructions are limited to single-component formats.

Below is the pseudocode for GLSL built-in function textureWindowBlockMatchSADQCOM. The pseudocode for textureWindowBlockMatchSSD is identical other than replacing all instances of "SAD" with "SSD".

vec4 textureBlockMatchWindowSAD( sampler2D target,
                                 uvec2 targetCoord,
                                 samler2D reference,
                                 uvec2 refCoord,
                                 uvec2 blocksize) {
    // compareMode (MIN or MAX) comes from the vkSampler associated with `target`
    // uvec2 window  comes from the vkSampler associated with `target`
    minSAD = INF;
    maxSAD = -INF;
    uvec2 minCoord;
    uvec2 maxCoord;

    for (uint x=0, x < window.width; x++) {
        for (uint y=0; y < window.height; y++) {
            float SAD = textureBlockMatchSAD(target,
                                            targetCoord + uvec2(x, y),
                                            reference,
                                            refCoord,
                                            blocksize).x;
            // Note: the below comparison operator will produce undefined results
            // if SAD is a denorm value.
            if (SAD < minSAD) {
                minSAD = SAD;
                minCoord = uvec2(x,y);
            }
            if (SAD > maxSAD) {
                maxSAD = SAD;
                maxCoord = uvec2(x,y);
            }
        }
    }
    if (compareMode=MIN) {
        return vec4(minSAD, minCoord.x, minCoord.y, 0.0);
    } else {
        return vec4(maxSAD, maxCoord.x, maxCoord.y, 0.0);
    }
}

Below is the pseudocode for textureBlockMatchGatherSADQCOM. The pseudocode for textureBlockMatchGatherSSD follows an identical pattern.

vec4 textureBlockMatchGatherSAD( sampler2D target,
                                 uvec2 targetCoord,
                                 samler2D reference,
                                 uvec2 refCoord,
                                 uvec2 blocksize) {
    vec4 out;
    for (uint x=0, x<4; x++) {
            float SAD = textureBlockMatchSAD(target,
                                            targetCoord + uvec2(x, 0),
                                            reference,
                                            refCoord,
                                            blocksize).x;
            out[x] = SAD;
    }
    return out;
}

New Structures

• Extending VkPhysicalDeviceFeatures2, VkDeviceCreateInfo:
  • VkPhysicalDeviceImageProcessing2FeaturesQCOM
• Extending VkPhysicalDeviceProperties2:
  • VkPhysicalDeviceImageProcessing2PropertiesQCOM
• Extending VkSamplerCreateInfo:
  • VkSamplerBlockMatchWindowCreateInfoQCOM

New Enums

• VkBlockMatchWindowCompareModeQCOM

New Enum Constants

• VK_QCOM_IMAGE_PROCESSING_2_EXTENSION_NAME
• VK_QCOM_IMAGE_PROCESSING_2_SPEC_VERSION
• Extending VkStructureType:
  • VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_PROCESSING_2_FEATURES_QCOM
  • VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_PROCESSING_2_PROPERTIES_QCOM
  • VK_STRUCTURE_TYPE_SAMPLER_BLOCK_MATCH_WINDOW_CREATE_INFO_QCOM

Issues

1) What is the precision of the min/max comparison checks?

RESOLVED: Intermediate computations for the new operations are performed at 16-bit floating-point precision. If the value of "float SAD" in the above code sample is a 16-bit denorm value, then behavior of the MIN/MAX comparison is undefined.

Version History

• Revision 1, 2023-03-10 (Jeff Leger)