Devices and Queues
Once Vulkan is initialized, devices and queues are the primary objects used to interact with a Vulkan implementation.
Physical Devices
Devices
Device objects represent logical connections to physical devices. Each device exposes a number of queue families each having one or more queues. All queues in a queue family support the same operations.
As described in Physical Devices, a Vulkan application will first query for all physical devices in a system. Each physical device can then be queried for its capabilities, including its queue and queue family properties. Once an acceptable physical device is identified, an application will create a corresponding logical device. The created logical device is then the primary interface to the physical device.
How to enumerate the physical devices in a system and query those physical devices for their queue family properties is described in the Physical Device Enumeration section above.
A single logical device can be created from multiple physical devices, if those physical devices belong to the same device group. A device group is a set of physical devices that support accessing each other’s memory and recording a single command buffer that can be executed on all the physical devices. Device groups are enumerated by calling vkEnumeratePhysicalDeviceGroups, and a logical device is created from a subset of the physical devices in a device group by passing the physical devices through VkDeviceGroupDeviceCreateInfo. For two physical devices to be in the same device group, they must support identical extensions, features, and properties.
Physical devices in the same device group must be so similar because there
are no rules for how different features/properties would interact.
They must return the same values for nearly every invariant
vkGetPhysicalDevice* feature, property, capability, etc., but could
potentially differ for certain queries based on things like having a
different display connected, or a different compositor.
The specification does not attempt to enumerate which state is in each
category, because such a list would quickly become out of date.
Device Creation
The number of shader cores used by all the queues of a device can be
controlled by adding a VkDeviceQueueShaderCoreControlCreateInfoARM
structure to the pNext chain of the VkDeviceCreateInfo
structure.
Device Use
The following is a high-level list of VkDevice uses along with
references on where to find more information:
- Creation of queues. See the Queues section below for further details.
- Creation and tracking of various synchronization constructs. See Synchronization and Cache Control for further details.
- Allocating, freeing, and managing memory. See Memory Allocation and Resource Creation for further details.
- Creation and destruction of command buffers and command buffer pools. See Command Buffers for further details.
- Creation, destruction, and management of graphics state. See Pipelines and Resource Descriptors, among others, for further details.
Lost Device
A logical device may become lost for a number of implementation-specific reasons, indicating that pending and future command execution may fail and cause resources and backing memory to become undefined:.
Typical reasons for device loss will include things like execution timing out (to prevent denial of service), power management events, platform resource management, implementation errors.
Applications not adhering to valid usage may also result in device loss being reported, however this is not guaranteed. Even if device loss is reported, the system may be in an unrecoverable state, and further usage of the API is still considered invalid.
When this happens, certain commands will return VK_ERROR_DEVICE_LOST.
After any such event, the logical device is considered lost.
It is not possible to reset the logical device to a non-lost state, however
the lost state is specific to a logical device (VkDevice), and the
corresponding physical device (VkPhysicalDevice) may be otherwise
unaffected.
In some cases, the physical device may also be lost, and attempting to
create a new logical device will fail, returning VK_ERROR_DEVICE_LOST.
This is usually indicative of a problem with the underlying implementation,
or its connection to the host.
If the physical device has not been lost, and a new logical device is
successfully created from that physical device, it must be in the non-lost
state.
Whilst logical device loss may be recoverable, in the case of physical device loss, it is unlikely that an application will be able to recover unless additional, unaffected physical devices exist on the system. The error is largely informational and intended only to inform the user that a platform issue has occurred, and should be investigated further. For example, underlying hardware may have developed a fault or become physically disconnected from the rest of the system. In many cases, physical device loss may cause other more serious issues such as the operating system crashing; in which case it may not be reported via the Vulkan API.
When a device is lost, its child objects are not implicitly destroyed and their handles are still valid. Those objects must still be destroyed before their parents or the device can be destroyed (see the Object Lifetime section). The host address space corresponding to device memory mapped using vkMapMemory is still valid, and host memory accesses to these mapped regions are still valid, but the contents are undefined:. It is still legal to call any API command on the device and child objects.
Once a device is lost, command execution may fail, and certain commands
that return a VkResult may return VK_ERROR_DEVICE_LOST.
These commands can be identified by the inclusion of
VK_ERROR_DEVICE_LOST in the Return Codes section for each command.
Commands that do not allow runtime errors must still operate correctly for
valid usage and, if applicable, return valid data.
Commands that wait indefinitely for device execution (namely
vkDeviceWaitIdle, vkQueueWaitIdle, vkWaitForFences
or vkAcquireNextImageKHR
with a maximum timeout, and vkGetQueryPoolResults with the
VK_QUERY_RESULT_WAIT_BIT bit set in flags) must return in
finite time even in the case of a lost device, and return either
VK_SUCCESS or VK_ERROR_DEVICE_LOST.
For any command that may return VK_ERROR_DEVICE_LOST, for the purpose
of determining whether a command buffer is in the
pending state, or whether resources are
considered in-use by the device, a return value of
VK_ERROR_DEVICE_LOST is equivalent to VK_SUCCESS.
If a device was created with the maintenance5 feature enabled, and any device command returns
VK_ERROR_DEVICE_LOST, then all device commands for which
VK_ERROR_DEVICE_LOST is a valid return value and which happen-after it
on the same host thread must return VK_ERROR_DEVICE_LOST.
Device commands executing on other threads must begin returning
VK_ERROR_DEVICE_LOST within finite time.
The content of any external memory objects that have been exported from or
imported to a lost device become undefined:.
Objects on other logical devices or in other APIs which are associated with
the same underlying memory resource as the external memory objects on the
lost device are unaffected other than their content becoming undefined:.
The layout of subresources of images on other logical devices that are bound
to VkDeviceMemory objects associated with the same underlying memory
resources as external memory objects on the lost device becomes
VK_IMAGE_LAYOUT_UNDEFINED.
The state of VkSemaphore objects on other logical devices created by
importing a semaphore payload with
temporary permanence which was exported from the lost device is undefined:.
The state of VkSemaphore objects on other logical devices that
permanently share a semaphore payload with a VkSemaphore object on the
lost device is undefined:, and remains undefined: following any subsequent
signal operations.
Implementations must ensure pending and subsequently submitted wait
operations on such semaphores behave as defined in
Semaphore State Requirements For
Wait Operations for external semaphores not in a valid state for a wait
operation.
Device Destruction
Queues
Queue Family Properties
As discussed in the Physical Device Enumeration section above, the vkGetPhysicalDeviceQueueFamilyProperties command is used to retrieve details about the queue families and queues supported by a device.
Each index in the pQueueFamilyProperties array returned by
vkGetPhysicalDeviceQueueFamilyProperties describes a unique queue
family on that physical device.
These indices are used when creating queues, and they correspond directly
with the queueFamilyIndex that is passed to the vkCreateDevice
command via the VkDeviceQueueCreateInfo structure as described in the
Queue Creation section below.
Grouping of queue families within a physical device is implementation-dependent.
The general expectation is that a physical device groups all queues of matching capabilities into a single family. However, while implementations should do this, it is possible that a physical device may return two separate queue families with the same capabilities.
Once an application has identified a physical device with the queue(s) that it desires to use, it will create those queues in conjunction with a logical device. This is described in the following section.
Queue Creation
Queues with higher system priority may be allotted more processing time than queues with lower priority. An implementation may allow a higher-priority queue to starve a lower-priority queue until the higher-priority queue has no further commands to execute.
Priorities imply no ordering or scheduling constraints.
No specific guarantees are made about higher priority queues receiving more processing time or better quality of service than lower priority queues.
The global priority level of a queue takes precedence over the per-process
queue priority (VkDeviceQueueCreateInfo::pQueuePriorities).
Abuse of this feature may result in starving the rest of the system of
implementation resources.
Therefore, the driver implementation may deny requests to acquire a
priority above the default priority
(VK_QUEUE_GLOBAL_PRIORITY_MEDIUM_KHR) if the caller does not have
sufficient privileges.
In this scenario VK_ERROR_NOT_PERMITTED_KHR is returned.
The driver implementation may fail the queue allocation request if
resources required to complete the operation have been exhausted (either by
the same process or a different process).
In this scenario VK_ERROR_INITIALIZATION_FAILED is returned.
If the globalPriorityQuery feature
is enabled and the requested global priority is not reported via
VkQueueFamilyGlobalPriorityPropertiesKHR, the driver implementation
must fail the queue creation.
In this scenario, VK_ERROR_INITIALIZATION_FAILED is returned.
Queue Family Index
The queue family index is used in multiple places in Vulkan in order to tie operations to a specific family of queues.
When retrieving a handle to the queue via vkGetDeviceQueue, the queue
family index is used to select which queue family to retrieve the
VkQueue handle from as described in the previous section.
When creating a VkCommandPool object (see
Command Pools), a queue family index is specified
in the VkCommandPoolCreateInfo structure.
Command buffers from this pool can only be submitted on queues
corresponding to this queue family.
When creating VkImage (see Images) and
VkBuffer (see Buffers) resources, a set of queue
families is included in the VkImageCreateInfo and
VkBufferCreateInfo structures to specify the queue families that can
access the resource.
When inserting a VkBufferMemoryBarrier or VkImageMemoryBarrier (see Pipeline Barriers), a source and destination queue family index is specified to allow the ownership of a buffer or image to be transferred from one queue family to another. See the Resource Sharing section for details.
Queue Priority
Each queue is assigned a priority, as set in the VkDeviceQueueCreateInfo structures when creating the device. The priority of each queue is a normalized floating point value between 0.0 and 1.0, which is then translated to a discrete priority level by the implementation. Higher values indicate a higher priority, with 0.0 being the lowest priority and 1.0 being the highest.
Within the same device, queues with higher priority may be allotted more processing time than queues with lower priority. The implementation makes no guarantees with regards to ordering or scheduling among queues with the same priority, other than the constraints defined by any explicit synchronization primitives. The implementation makes no guarantees with regards to queues across different devices.
An implementation may allow a higher-priority queue to starve a
lower-priority queue on the same VkDevice until the higher-priority
queue has no further commands to execute.
The relationship of queue priorities must not cause queues on one
VkDevice to starve queues on another VkDevice.
No specific guarantees are made about higher priority queues receiving more processing time or better quality of service than lower priority queues.
Queue Submission
Work is submitted to a queue via queue submission commands such as vkQueueSubmit2 or vkQueueSubmit. Queue submission commands define a set of queue operations to be executed by the underlying physical device, including synchronization with semaphores and fences.
Submission commands take as parameters a target queue, zero or more batches of work, and an optional fence to signal upon completion. Each batch consists of three distinct parts:
- Zero or more semaphores to wait on before execution of the rest of the
batch.
- If present, these describe a semaphore wait operation.
- Zero or more work items to execute.
- If present, these describe a queue operation matching the work described.
- Zero or more semaphores to signal upon completion of the work items.
- If present, these describe a semaphore signal operation.
If a fence is present in a queue submission, it describes a fence signal operation.
All work described by a queue submission command must be submitted to the queue before the command returns.
Sparse Memory Binding
In Vulkan it is possible to sparsely bind memory to buffers and images as
described in the Sparse Resource chapter.
Sparse memory binding is a queue operation.
A queue whose flags include the VK_QUEUE_SPARSE_BINDING_BIT must be
able to support the mapping of a virtual address to a physical address on
the device.
This causes an update to the page table mappings on the device.
This update must be synchronized on a queue to avoid corrupting page table
mappings during execution of graphics commands.
By binding the sparse memory resources on queues, all commands that are
dependent on the updated bindings are synchronized to only execute after the
binding is updated.
See the Synchronization and Cache Control chapter for
how this synchronization is accomplished.
Queue Destruction
Queues are created along with a logical device during vkCreateDevice.
All queues associated with a logical device are destroyed when
vkDestroyDevice is called on that device.