Usage

General Usage

The ML Emulation Layer for Vulkan® is loaded as two explicit Vulkan® layers. The platform-specific sections below show the exact commands for each operating system, but the same setup sequence applies on all platforms:

  1. Make the Vulkan® loader discover the layer manifest files.

  2. Make the platform dynamic loader discover the graph and tensor layer libraries.

  3. Enable the graph layer before the tensor layer, either during Vulkan® instance creation or with VK_INSTANCE_LAYERS.

  4. Configure any optional logging or profiling environment variables before starting the application.

The layer names are:

  • VK_LAYER_ML_Graph_Emulation

  • VK_LAYER_ML_Tensor_Emulation

The manifest files are:

  • VkLayer_Graph.json

  • VkLayer_Tensor.json

For more information about using explicit Vulkan® layers, see the Vulkan® Layer Documentation.

Logging

You can enable logging using environment variables. Logging must be set before the application is started. Logging severity can be one of error, warning, info, or debug. Logging severity is set independently for the graph and tensor layer.

Using shell:

export VMEL_GRAPH_SEVERITY=debug
export VMEL_TENSOR_SEVERITY=info

Using PowerShell:

$env:VMEL_GRAPH_SEVERITY="debug"
$env:VMEL_TENSOR_SEVERITY="info"

Common severity for both layers can be set using the following variable:

export VMEL_COMMON_SEVERITY=debug

$env:VMEL_COMMON_SEVERITY="debug"

Graph Profiling

You can enable per-pipeline graph profiling with Vulkan® timestamp queries using environment variables before starting the application. Profiling covers TOSA graph operators, MotionEngine graph operators, and optical-flow compute pipelines. Profiling is disabled by default. When enabled, graph command-buffer submits remain asynchronous. Timestamp results are collected when the application waits on fences, waits for a queue or device to become idle, or when the profiling property is queried. Profiling results are saved only as a queryable data graph pipeline property.

Using shell:

export VMEL_GRAPH_PROFILING=1

Using PowerShell:

$env:VMEL_GRAPH_PROFILING="1"

The profiling property returns JSON with a samples array containing one entry per profiled internal compute dispatch, including pipeline_kind, operator_name, raw cycle counts, and time_ms, plus a by_operator summary with total, average, minimum, and maximum time per profiled pipeline.

Graph profiling requires a queue family with non-zero timestampValidBits. When the selected Vulkan® driver does not expose timestamp queries, graph execution remains available but no timestamp samples can be collected.

Usage on Linux

You can enable the graph and tensor layers using environment variables only, without modifying the Vulkan® application. The following environment variables are used:

  • Use the LD_LIBRARY_PATH environment variable to point at the VkLayer_Graph and VkLayer_Tensor libraries.

  • Use the VK_ADD_LAYER_PATH environment variable to point at the VkLayer_Graph.json and VkLayer_Tensor.json manifest file.

    • If your loader ignores VK_ADD_LAYER_PATH (older SDKs before 1.4.328.1), use VK_LAYER_PATH.

  • You must enable the graph layer before the tensor layer. To do this, use the VK_INSTANCE_LAYERS environment variable.

If you have installed the ML Emulation Layer for Vulkan® into a deploy folder, use the following environment variables to enable the layers:

export LD_LIBRARY_PATH=$PWD/deploy/lib:$LD_LIBRARY_PATH
export VK_ADD_LAYER_PATH=$PWD/deploy/share/vulkan/explicit_layer.d
export VK_INSTANCE_LAYERS=VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation

Usage on Windows®

You can enable the graph and tensor layers using environment variables only, without modifying the Vulkan® application. The following environment variables are used:

  • Use the VK_ADD_LAYER_PATH environment variable to point at the VkLayer_Graph.json and VkLayer_Tensor.json manifest files.

  • You must enable the graph layer before the tensor layer. To do this, use the VK_INSTANCE_LAYERS environment variable.

If you have installed the ML Emulation Layer for Vulkan® into a deploy folder, use the following environment variables to enable the layers:

$env:VK_LAYER_PATH="$PWD\deploy\bin"
$env:VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation;VK_LAYER_ML_Tensor_Emulation"

Alternatively, you can use the Windows® registry keys to load the manifest files. This can be done using the Windows® GUI. Or, if you have installed the ML Emulation Layer for Vulkan® into a deploy folder, you set the path to the manifest files using:

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\Vulkan\ExplicitLayers /v `
{ABSOLUTE_PATH}\deploy\bin /t REG_DWORD /d 0 /f

$env:VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation;VK_LAYER_ML_Tensor_Emulation"

Note

If running a Windows® terminal with elevated permissions, VK_ADD_LAYER_PATH is ignored for security reasons. However, if VK_ADD_LAYER_PATH is set and not ignored, then Vulkan skips searching the registry keys for manifest files.

Building for Android™

The Android™ NDK toolset is required to build the ML Emulation Layer for Vulkan® for an Android™ device. The Android™ device must have Vulkan® API 1.3 support.

To build the ML Emulation Layer for Vulkan®, run:

cmake -B build
   -DCMAKE_TOOLCHAIN_FILE=${NDK}/build/cmake/android.toolchain.cmake \
   -DANDROID_ABI=arm64-v8a                                           \
   -DGLSLANG_PATH=${REPO}/dependencies/glslang                       \
   -DSPIRV_CROSS_PATH=${REPO}/dependencies/SPIRV-Cross               \
   -DSPIRV_HEADERS_PATH=${REPO}/dependencies/SPIRV-Headers           \
   -DSPIRV_TOOLS_PATH=${REPO}/dependencies/SPIRV-Tools               \
   -DVULKAN_HEADERS_PATH=${REPO}/dependencies/Vulkan-Headers

cmake --build build

Usage on Android™

You can pack the graph and tensor layer libraries into the Application Package Kit (APK) or push to the /data/local/debug/vulkan directory for Android™ to discover the ML Emulation Layer for Vulkan®. Applications can enable the layers during Vulkan instance creation or you can enable the layers without modifying the application by using following commands:

adb shell settings put global enable_gpu_debug_layers 1
adb shell settings put global gpu_debug_app $TARGET_APP_PKG
adb shell settings put global gpu_debug_layers \
    VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation

APK Packaging

If you want to package the ML Emulation Layer for Vulkan® as an Android™ APK, set the following variables first:

export EMULATION_LAYER_ROOT=/path/to/emulation-layer
export NDK=/path/to/android-ndk
export ANDROID_HOME=/path/to/android-sdk
export TARGET_APP_PKG=com.example.targetapp

The Android™ packaging flow in scripts/build.py requires the Android™ NDK toolchain for the native build and Gradle 8.4 or later with ANDROID_HOME set for APK generation. The Android™ SDK installation pointed to by ANDROID_HOME should include build-tools;34.0.0 and platforms;android-34, or other compatible versions. A typical APK packaging command looks like:

python3 $EMULATION_LAYER_ROOT/scripts/build.py \
    --build-type Android \
    --target-platform android \
    --cmake-toolchain-for-android $NDK/build/cmake/android.toolchain.cmake \
    --install $EMULATION_LAYER_ROOT/apk_install \
    --package-type apk \
    -j $(nproc)

This produces an Android™ project in apk_package/ and Gradle builds the debug APK from there. The layer APK package name is currently com.arm.ai_ml_emulation_layer_for_vulkan.

To enable the packaged layers for a target application, use Android™ GPU debug layer settings:

adb shell settings put global enable_gpu_debug_layers 1
adb shell settings put global gpu_debug_app $TARGET_APP_PKG
adb shell settings put global gpu_debug_layers \
    VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation
adb shell settings put global gpu_debug_layer_app \
    com.arm.ai_ml_emulation_layer_for_vulkan

If you only want to enable a single layer, the command will likely look like:

adb shell settings put global gpu_debug_layers VK_LAYER_KHRONOS_validation

The layer package must also be visible to the debug app. On Android™ 11 and later, if the target app does not already query the layer package, add a package visibility entry such as:

<queries>
    <package android:name="com.arm.ai_ml_emulation_layer_for_vulkan" />
</queries>

Refer to the Android™ validation layer guide for background on APK packaging, debug layer settings, and package visibility: Use Vulkan validation layers on Android.

Building for Darwin

Install the LunarG Vulkan SDK to obtain the Vulkan® Loader. Recent SDK releases can also install KosmicKrisp as an opt-in technical preview; check that SDK release’s host requirements before selecting it.

To build the ML Emulation Layer for Vulkan®, run:

python3 "$SDK_PATH/sw/emulation-layer/scripts/build.py" \
    --install "$SDK_PATH/deploy"

For Vulkan® SDK installation and driver requirements, see the LunarG getting-started guide.

Cross Compilation for AArch64

The shader pre-compilation step requires a glslang compiler. There are three ways to accomplish this when cross-compiling:

  1. Provide a custom glslang executable. You can direct CMake to a custom glslang executable file using the GLSLANG_EXECUTABLE option. First, build glslang inside its repo. When the repository is initialized using the repo manifest, the glslang source is checked out in <repo_root>/dependencies/glslang/ For building glslang, see Building (CMake).

  2. Install glslang to the system. Under cross compilation, when no custom glslang executable is provided, it will be searched from the system using CMake’s find_package. On Ubuntu, you can install it with sudo apt install glslang-tools or from the source code following the previously mentioned documentation. Note that we require version > 15.4.0, which may not yet be available in Ubuntu’s official package repositories.

  3. Disable shader pre-compilation. This can be done by adding the flag --disable-precompile-shaders to the build script command. By doing so, the shaders will be compiled at runtime.

An example build flow using the option 1 would be:

First, build the glslang standalone under <repo_root>/dependencies/glslang/:

cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DENABLE_GLSLANG_BINARIES=ON -DENABLE_OPT=OFF -DBUILD_SHARED_LIBS=OFF

cmake --build build --target glslang-standalone

After building, the binary will be at <repo_root>/dependencies/glslang/build/StandAlone/glslang. Then run the following under <repo_root>/sw/emulation-layer/:

cmake -B build \
   -DCMAKE_TOOLCHAIN_FILE=${REPO}/sw/emulation-layer/cmake/toolchain/linux-aarch64-gcc.cmake \
   -DGLSLANG_PATH=${REPO}/dependencies/glslang \
   -DSPIRV_CROSS_PATH=${REPO}/dependencies/SPIRV-Cross \
   -DSPIRV_HEADERS_PATH=${REPO}/dependencies/SPIRV-Headers \
   -DSPIRV_TOOLS_PATH=${REPO}/dependencies/SPIRV-Tools \
   -DVULKAN_HEADERS_PATH=${REPO}/dependencies/Vulkan-Headers \
   -DGLSLANG_EXECUTABLE=${REPO}/dependencies/glslang/build/StandAlone/glslang

cmake --build build

PyPI

The ML Emulation Layer for Vulkan® is available on PyPI as the ai-ml-emulation-layer-for-vulkan package.

Install the published package:

pip install ai-ml-emulation-layer-for-vulkan

To build and install the host layers from an ML SDK checkout, run from this repository root:

pip install .

Troubleshooting

All zero output from AMD GPUs on Linux

Some workloads may cause silent GPU crashes due to timeout errors. You can check for related kernel messages with the following command:

dmesg | grep -i amdgpu

To change the timeout, follow these steps (applies if your system uses GRUB as the bootloader):

  1. Edit the GRUB configuration file:

    sudo nano /etc/default/grub

  2. Add or modify the GRUB_CMDLINE_LINUX line to include a longer timeout value in milliseconds:

    GRUB_CMDLINE_LINUX="quiet splash amdgpu.lockup_timeout=20000"

  3. Update the GRUB configuration:

    sudo update-grub

  4. Reboot the system:

    sudo reboot