Performance data and analysis

Scenario Runner provides several methods for collecting timing and diagnostic information.

Method	Use it for	Output
Runtime profiling	Measuring Vulkan® command execution	JSON
Performance counters	Measuring host-side setup and preparation	JSON
Emulation Layer graph profiling	Measuring individual operators inside the Emulation Layer	JSON
Neural accelerator statistics	Collecting implementation-specific accelerator statistics	Binary
Neural accelerator debug database	Driver-specific advanced debugging	Binary

Runtime profiling

Use --profiling-dump-path to collect Vulkan® timestamp information:

scenario-runner \
    --scenario scenario.json \
    --profiling-dump-path profiling.json

The output contains the command name and type, timestamp values, timestamp period, execution time in milliseconds, and iteration number.

Total execution time [ms] records the total execution time across all iterations.

For data graph pipelines, the output can also contain Memory Usage entries. Session memory [bytes] is the session-memory requirement reported for the pipeline, not the total or peak GPU memory usage.

When --dry-run is used, Scenario Runner can report session-memory requirements, but it does not produce execution timestamps.

Host-side performance counters

Use --perf-counters-dump-path to collect host-side timings:

scenario-runner \
    --scenario scenario.json \
    --perf-counters-dump-path counters.json

The output uses microseconds and groups measurements into categories such as scenario setup, pipeline setup, running the scenario, loading or saving the pipeline cache, and saving results.

Time to Inference is the preparation time before the first inference starts.

Total Scenario Time is the sum of all performance counters recorded by Scenario Runner.

These counters measure host-side time rather than individual GPU command execution. Use runtime profiling for command-level Vulkan® execution times.

Emulation Layer graph profiling

Enable Emulation Layer graph profiling before starting Scenario Runner:

export VMEL_GRAPH_PROFILING=1

Then provide an existing output directory:

scenario-runner \
    --scenario scenario.json \
    --emulation-layer-profiling-dump-dir profiling-output

Scenario Runner writes one JSON file per graph pipeline:

EmulationLayerPipeline_0.json
EmulationLayerPipeline_1.json

Each sample contains:

The submission and graph-dispatch indexes.
The internal pipeline index and kind.
The operator name.
Timestamp values and their difference.
The calculated execution time in milliseconds.

The by_operator section groups samples by pipeline kind and operator name. It provides the dispatch count and the total, average, minimum, and maximum execution times.

Graph profiling requires a Vulkan® queue family with non-zero timestampValidBits. If timestamp queries are unavailable, graph execution can continue, but no profiling samples are collected.

These results describe execution inside the Emulation Layer and are not representative of native neural accelerator performance.

Neural accelerator statistics

Scenario Runner supports VK_ARM_data_graph_neural_accelerator_statistics for data graph pipelines. When this extension and its dataGraphNeuralAcceleratorStatistics feature are available, Scenario Runner can request neural accelerator debug database and statistics output from graph pipelines. Scenario Runner fails before creating the Vulkan® device if either dump is requested without both.

Use --neural-statistics-dump-dir to enable statistics collection and dump the resulting statistics memory after execution.

--neural-statistics-mode selects the statistics mode, using 0 for VK_NEURAL_ACCELERATOR_STATISTICS_MODE_STATISTICS_0_ARM and 1 for VK_NEURAL_ACCELERATOR_STATISTICS_MODE_STATISTICS_1_ARM.

For example:

scenario-runner \
    --scenario scenario.json \
    --neural-statistics-dump-dir statistics-output \
    --neural-statistics-mode 0

Scenario Runner writes files such as:

Graph_Pipeline_0_Statistics_0.bin
Graph_Pipeline_0_Statistics_1.bin

The contents and interpretation of these dumps are implementation-specific. Scenario Runner writes the data returned by the Vulkan® implementation to disk without interpreting it.

The Emulation Layer does not implement this hardware-specific extension.

Neural accelerator debug database

Use --neural-debug-database-dump-dir to dump the neural accelerator debug database for graph pipelines that expose one:

scenario-runner \
    --scenario scenario.json \
    --neural-debug-database-dump-dir debug-output

Scenario Runner writes files such as:

Graph_Pipeline_0_Debug_Database.bin

The debug database is supplied by the driver and saved without interpretation. If a pipeline does not expose a debug database, no file is created.

Choosing a measurement

Use runtime profiling for command-level Vulkan® execution times.

Use performance counters for host-side setup, pipeline creation, caching, and other Scenario Runner operations.

Use Emulation Layer graph profiling for per-operator measurements while using the Emulation Layer.

Use neural accelerator statistics and debug databases for hardware-specific diagnostic data.