Build Script 'Test' Subcommand

Purpose

Testing all toolchain configurations across all possible build environments is a tedious, time-consuming manual job. The test subcommand automates this process by executing a matrix of build and run tests across toolchains, build types, and execution environments — ensuring that every combination compiles and runs correctly.

Originally implemented as a standalone script (test-build.py), this functionality was integrated into build.py as a first-class subcommand to leverage the existing subcommand infrastructure (SubCommandNative, SubCommandWine, SubCommandDocker, SubCommandRun) and share configuration, logging, and argument parsing.

Why This Subcommand Exists

Cross-platform validation: The project supports multiple compilers (GNU, MinGW, MSVC, cross-compiled targets) across different execution contexts (native Linux, native Windows, Docker containers, Wine). Manually verifying each combination is error-prone and slow.
Regression safety: When toolchains, libraries, or build configurations change, the matrix test catches regressions early by exercising every configured preset.
Idempotent execution: Previously successful commands are tracked via SHA-256 hashes in a log file, so re-running the test suite skips already-passed commands and only executes new or changed ones.
Configuration-driven: The entire test matrix is defined in build.ini, making it easy to add, remove, or modify profiles and groups without changing code.

Architecture

The test subcommand is implemented in cmake/lib/bin/build-test.py as the class SubCommandTest, which extends main.SubCommand. It is only registered at the top level — it does not appear when running inside Docker or Wine containers, preventing nested test execution.

Core Components

Component	Description
Matrix Configuration	A `Dict[str, List[dict]]` structure mapping profile names (e.g., `lnx`, `dck`, `win`) to lists of build groups, each with a `wrapper`, `toolchain`, and `build_types`.
Profile Detection	Auto-detects which profile(s) match the current `sys.platform` by comparing against the `platform` attribute in the matrix. Multiple profiles can match (e.g., both `lnx` and `dck` on Linux).
Wrapper Dispatch	Routes build and run commands through the appropriate wrapper: `native`, `wine`, `docker`, or `docker_wine`. Each wrapper peels its own arguments, passing the inner command through.
Command Deduplication	Uses SHA-256 hashes of command strings to track previously successful builds and runs in a log file, avoiding redundant execution.
Timed Logging	Each successful command is logged with its execution time in `HH:MM:SS` format, providing a performance history.

The Wrapper Layer Model

The subcommand supports four wrapper types, which can be nested. Each layer receives only its own options in args_left, while the inner command(s) go in args_right. This allows each nested layer to peel away its arguments up to --.

Wrapper	Description	Example Command
`native`	Direct build/run on the host system	`build.py --build gnu-debug`
`wine`	Build/run inside Wine (Windows compatibility layer)	`build.py wine --git-server -- --build gnu-debug`
`docker`	Build/run inside a Docker container	`build.py docker --build-dir tmp -- --build gnu-debug`
`docker_wine`	Build/run inside Docker, then Wine (nested)	`build.py docker --build-dir tmp -- wine -- --build gnu-debug`

Layered Argument Peeling

For a docker_wine wrapper, the command flows through three layers:

Docker layer receives --build-dir tmp in args_left, and ["wine", "--", "--build", "gnu-debug"] in args_right.
Wine layer (invoked inside Docker) receives -- as the separator, and ["--build", "gnu-debug"] as the inner command.
Native layer (invoked inside Wine) receives ["--build", "gnu-debug"] and executes the build.

Each subcommand class (SubCommandDocker, SubCommandWine, SubCommandNative) is invoked programmatically via its handle() method, not as a subprocess.

Configuration

The subcommand reads its configuration from two sections in build.ini:

`[config-test]` — Test Command Settings

Key	Description	Default
`build_types`	Comma-separated list of build types to test	`debug,release,coverage`
`apps_linux`	Comma-separated list of Linux applications to run	`./hello-world.bin`
`apps_windows`	Comma-separated list of Windows applications to run	`hello-world.exe`
`build_arguments`	Arguments passed to the build step	`--build`
`run_arguments`	Arguments passed to the run command	`--preset`
`wine_arguments`	Arguments passed to the Wine subcommand	(empty)
`docker_arguments`	Arguments passed to the Docker subcommand	`--build-dir tmp`

`[config-test-matrix]` — Build Matrix Definition

The matrix defines profiles and their build groups using a flat key-value format:

<profile>/<index>/<attribute> = <value>

Where: - profile — Profile identifier (e.g., lnx, dck, win) - index — 0-based group position within the profile - attribute — One of wrapper, toolchain, or build_types

Profile-level metadata uses a two-part key:

<profile>/description = Human-readable description
<profile>/platform = Required platform (e.g., linux, win32)

Example Configuration

[__config-test-matrix]
; Linux profile – native + cross-compile + Wine.
lnx/platform=linux
lnx/description=Linux native builds including Windows using Wine.
lnx/0/wrapper=native
lnx/0/toolchain=gnu
lnx/0/build_types=debug,release,coverage
lnx/1/wrapper=native
lnx/1/toolchain=gw
lnx/1/build_types=debug,release,coverage
lnx/2/wrapper=wine
lnx/2/toolchain=mingw
lnx/2/build_types=debug,release
lnx/3/wrapper=wine
lnx/3/toolchain=msvc
lnx/3/build_types=debug,release

The matrix is cached after the first load, so repeated calls (e.g., detect_profile(), options(), handle()) do not re-parse the configuration.

Hardcoded Defaults

If the [config-test-matrix] section is absent or empty, the subcommand falls back to hardcoded defaults stored as Dict[str, str] in the code, parsed by the same _parse_matrix_kv() method. This ensures the subcommand always has a valid matrix.

Usage

Command-Line Options

Option	Description
`-r`, `--run`	Execute the build and test matrix (required for actual execution)
`-V`, `--verbose`	Enable verbose debug output
`-i`, `--info`	Display all commands for all profiles without executing
`-p PROFILE`, `--profile PROFILE`	Override auto-detected profile(s)
`-f FILE`, `--file FILE`	Export the current matrix configuration to a new INI file
`-x`, `--stop-on-fail`	Stop execution on the first failure

Common Commands

# Run tests for all auto-detected profiles (e.g., lnx + dck on Linux)
./build.py test -r

# Show all commands for all profiles (dry-run info)
./build.py test -r -i

# Run only the Docker profile
./build.py test -r -p dck

# Run with stop-on-fail
./build.py test -r -x

# Export current matrix configuration to a file
./build.py test -f my-matrix.ini

Profile Auto-Detection

When no -p flag is provided, the subcommand determines which profile(s) to execute by matching sys.platform against the platform attribute in the matrix:

On Linux (sys.platform == "linux"): Both lnx and dck profiles match, so both are executed.
On Windows (sys.platform == "win32"): Only the win profile matches.

If no profile matches, the subcommand reports an error and suggests using -p to override.

Platform Validation

Before executing, the subcommand validates that the current system matches each profile's required platform. For example, running -p win on Linux will fail with:

! Profile 'win' requires platform 'win32', but current is 'linux'.

Execution Flow

Parse arguments — Check for --file (export mode), --run (execution mode), and --info (display mode).
Load configuration — Read [config-test] for build types, apps, and arguments.
Detect profile(s) — Auto-detect or use the -p override.
Validate platform — Ensure the current system matches each profile's platform requirement.
Build the matrix — Parse [config-test-matrix] or use hardcoded defaults.
Load success log — Read previously successful command hashes from the log file.
Execute the matrix — For each profile, group, build type, and app:
Compute the command hash.
If the hash is in the success log, skip the command.
Otherwise, invoke the appropriate wrapper's handle() method.
On success, record the hash with timing information.
On failure, increment the failure counter and optionally stop.
Print summary — Report success, skip, and failure counts with elapsed time.

Log File

The subcommand maintains a log file (build-test.log) in the project's run directory. Each successful command is recorded as:

# HH:MM:SS | <command-string>
<sha256-hash>

Example:

# 00:01:30 | --build gnu-debug
a1b2c3d4e5f6...
# 00:00:45 | run -p gnu-debug -- ./hello-world.bin
f6e5d4c3b2a1...

The log file enables: - Idempotent re-runs: Previously successful commands are skipped. - Performance tracking: Execution times are recorded for each command. - Audit trail: The command history shows what was tested and when.

Matrix Export

The --file (-f) option exports the current matrix configuration to a new INI file with:

A header explaining the format.
The [config-test] section with current configuration values.
The [config-test-matrix] section with all profile descriptions, platforms, and group configurations sorted alphabetically.

The file is never overwritten — if the target file exists, the command fails with an error. This is useful for: - Backing up the current matrix configuration. - Sharing test configurations with other developers. - Creating custom matrix files for specific test scenarios.

Design Decisions

Programmatic Subcommand Invocation

Instead of spawning subprocesses, the subcommand invokes SubCommandNative, SubCommandWine, SubCommandDocker, and SubCommandRun programmatically via their handle() methods. This: - Shares the same process, configuration, and logger. - Avoids the overhead of subprocess creation. - Allows precise control over argument passing through the args_left/args_right mechanism.

Configuration-Driven Matrix

The entire test matrix is defined in build.ini, not in code. This: - Allows adding new profiles or groups without code changes. - Makes the test configuration visible and editable. - Enables different projects to have different test matrices.

Command Hashing for Deduplication

SHA-256 hashes of command strings provide a stable, content-based identifier. This: - Avoids re-running commands that have already succeeded. - Automatically detects when a command has changed (e.g., new build type). - Keeps the log file compact and easy to parse.

Profile-Based Platform Detection

Instead of hardcoding platform-to-profile mappings, the matrix's platform attribute drives detection. This: - Allows profiles to be added or removed without code changes. - Supports multiple profiles per platform (e.g., lnx and dck on Linux). - Makes the platform requirement explicit and visible in the configuration.