Regression tests in CI

A trace is a measurement, so a build can be gated on it: capture the firmware on a bench runner, compare the recording against a committed baseline, fail the job when the timing, the CPU load or the health counters move. The pieces are query fingerprint, query compare, and exit codes a job runner understands.

Fingerprints

A fingerprint is a small, git-committable .vafp.json distilled from a recording: selected metrics with per-metric tolerances plus capture provenance (OS, CPU clock, capture source). Counts are normalised to per-second rates so runs of different length compare fairly.

$ viewalyzer-cli query fingerprint --recording good.vadb --out app.vafp.json
{"schema_version":2,"recording_id":"328d49aa4eb6","query":"fingerprint","out":"app.vafp.json","runs":1,"sections":["comms","health","summary","tasks","timers","traces"]}

$ viewalyzer-cli query fingerprint --recording good1.vadb --runs good2.vadb,good3.vadb --out app.vafp.json    # learn an envelope from several good runs

Sections: summary (cpu_load_percent, events_per_second, context_switches_hz, preemptions_hz, exact total_tasks and trace_channels, heap scalars), tasks (per lane: cpu_percent, run_hz, avg_run_us, avg_period_us, max_jitter_us, preemption_hz, exact priority, stack, sync-op and failed_ops_hz), traces (per channel: rate_hz, mean_value, rms, std_dev, advisory min and max), timers (fires_hz, violations, lateness), comms (per path: rate_hz, median_us, p99_us, blocked_hz; per resource: peak_pending, still_pending, empty_receives_hz), health (corrupt_bytes, corrupt_runs, lost_events, seq_gaps: baselines are normally 0, so any loss fails).

The file is meant to be edited: change a metric's tol_pct or warn_pct, set abs_tol or abs_warn directly, delete items you do not want pinned.

Compare

$ viewalyzer-cli query compare --recording candidate.vadb --baseline app.vafp.json
{"schema_version":2,"recording_id":"bf37d1669b82","query":"compare","baseline":"app.vafp.json","verdict":"pass",
 "counts":{"pass":47,"warn":0,"fail":0,"missing":0,"new":0},"provenance_warnings":[],"baseline_runs":1,
 "results":[{"section":"summary","item":"_","metric":"context_switches_hz","status":"pass","current":2871.79,"baseline":2871.74,"min":2871.74,"max":2871.74,"delta_outside":0.04,"abs_warn":574.35,"abs_tol":717.94,"unit":"hz"},
            {"section":"health","item":"_","metric":"lost_events","status":"pass","current":0.0,"baseline":0.0,"min":0.0,"max":0.0,"unit":"count"}, ...]}

A metric passes inside the baseline's [min, max] envelope or within its warn band, warns up to the fail band, fails beyond; exact metrics must match. A baseline item missing from the run is missing (fail); an item the baseline never saw is new (warn). Provenance mismatches never fail; they land in provenance_warnings. Results are sorted fail, missing, warn, new, pass, with counts and the overall verdict.

Exit codes: 0 pass or warn, 2 regression, 1 error. --baseline also accepts any recording, fingerprinted on the fly.

A bench job

Capture on a self-hosted runner with the board attached, compare against the committed fingerprint, keep the recording as an artifact:

# .github/workflows/trace.yml
jobs:
  trace:
    runs-on: [self-hosted, bench-g474]
    steps:
      - uses: actions/checkout@v4
      - name: Flash
        run: python build.py --flash --serial ${{ vars.PROBE_SERIAL }}
      - name: Capture 30 s
        run: |
          viewalyzer-cli capture --config nucleo_g474_rambuf.vacf --stlink-serial ${{ vars.PROBE_SERIAL }} \
            --elf build/rambuf/firmware.elf --output run.vadb --duration 30 --no-register \
            | tee capture.log
          test "$(jq -r '.summary.lost_events' <<< "$(tail -1 capture.log)")" = "0"
      - name: Compare
        run: viewalyzer-cli query compare --recording run.vadb --baseline ci/app.vafp.json | tee compare.json
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: trace, path: "run.vadb\ncompare.json\ncapture.log" }

Pin the probe serial (--stlink-serial) on any machine that can have two probes, pass --no-register so the runner's index does not fill up, and read lost_events from the envelope: a capture that lost events is not a measurement of the firmware.

Assertions in Python

The Python SDK turns the same pieces into fixtures and asserts (Tests and CI):

from viewalyzer_sdk import ViewAlyzer

def test_isr_budget(tmp_path):
    va = ViewAlyzer()
    rec = va.record("nucleo_g474_rambuf.vacf", output=tmp_path / "run.vadb", duration_s=10, elf="firmware.elf")
    assert rec.is_clean                                   # no lost events, no corrupt bytes
    lanes = {t["name"]: t for t in rec.timeline()["tasks"]}
    assert lanes["ISR:SysTick"]["p99_slice_us"] < 5.0
    assert rec.compare("ci/app.vafp.json")["verdict"] in ("pass", "warn")

Metrics for dashboards

replay run.vadb prints one [metric] category,name,value,unit line per metric (recording info, event counts, per-lane slices and CPU, per-channel statistics, flow summaries, result,status), which a bench harness can scrape into a time series without parsing JSON; --perf adds the engine's stage timings.