feat: score the vPlan from test results as well as coverage

README.md

@@ -86,6 +86,7 @@ You can access it [online at opentitan.org/book/](https://opentitan.org/book/).
 
 * [Testplanner tool](./doc/testplanner.md)
 * [Design document](./doc/design_doc.md)
+* [The `lowrisc-dv-evidence` format](./doc/dv_evidence.md)
 * [Glossary](./doc/glossary.md)
 
 ## How to contribute

doc/dv_evidence.md

@@ -0,0 +1,94 @@
+<!--
+# Copyright lowRISC contributors (OpenTitan project).
+# Licensed under the Apache License, Version 2.0, see LICENSE for details.
+# SPDX-License-Identifier: Apache-2.0
+-->
+# The `lowrisc-dv-evidence` format
+
+A regression tells you which tests passed.
+A verification plan asks a different question: of everything we said we would verify, how much is now backed by something that ran?
+Answering it needs the regression's own outcomes in a form a planning tool can read, rather than a log directory and a human.
+
+This is that form.
+DVSim writes one of these files per simulation flow, and this page specifies what any producer has to write rather than describing one tool's output.
+Anything that can produce it can be scored against a verification plan, whether or not it is DVSim.
+
+## Where DVSim writes it
+
+`<scratch_path>/cov_vplan/dv_evidence.json`, produced by the `cov_vplan` job, and only when the sim config names a `vplan`.
+It is written before the annotation step runs and is archived alongside the reports, so it outlives the scratch area it describes.
+
+## Shape
+
+```json
+{
+  "schema": "lowrisc-dv-evidence",
+  "dut": "hmac",
+  "tool": "xcelium",
+  "produced_by": "dvsim 1.50.1",
+  "revision": "https://github.com/lowRISC/opentitan/tree/a1b2c3d (dirty)",
+  "timestamp": "2026-08-18T09:00:00+00:00",
+  "testcase": {
+    "hmac_smoke": [
+      { "status": "passed", "seed": 1234, "log": "/scratch/hmac/1234.hmac_smoke/run.log" },
+      { "status": "failed", "seed": 5678, "log": "...", "message": "UVM_ERROR", "line": 812 }
+    ],
+    "hmac_stress_all": [
+      { "status": "not_run" }
+    ]
+  }
+}
+```
+
+Fields are omitted when they have no value rather than written as `null`.
+
+### Top level
+
+| Key | Meaning |
+| --- | --- |
+| `schema` | Always `lowrisc-dv-evidence`. Identifies the format to whatever reads the file. |
+| `testcase` | Test name to the list of runs of that test. The only required key. |
+| `dut` | The design the results are about, named as a verification plan addresses it. |
+| `tool` | The simulator that produced them. |
+| `produced_by` | What wrote the file, with its version. |
+| `revision` | The tree the results were produced against, suffixed ` (dirty)` when it was not clean. |
+| `timestamp` | When the run started, as an ISO 8601 datetime with an offset. |
+
+### A run
+
+Every entry under `testcase` is keyed by the test name, because that is the name a plan refers to.
+Reseeds of one test share the key and are told apart by `seed`.
+
+| Key | Meaning |
+| --- | --- |
+| `status` | One of `passed`, `failed`, `killed`, `not_run`. Required. |
+| `seed` | The seed the run used, where the flow randomises. |
+| `log` | Path to the run's log. |
+| `message` | Why it ended that way. Present only on a run that did not pass. |
+| `line` | The log line the failure was first reported at. |
+
+`killed` and `not_run` are separate on purpose.
+A killed test started and was terminated, so the design was exercised and something went wrong.
+A `not_run` test never started, because the scheduler cancelled it once a dependency failed or the run was shut down.
+The two are different answers to "did we verify this", and collapsing them would let a build failure read as a passing plan item.
+
+There is no `waived` status.
+A waiver needs an owner and a date, and a regression can supply neither, so a known failure is recorded as an inspection instead.
+
+## Inspections
+
+The format also carries an `inspection` key, for claims no simulation can measure, such as a parameterisation or a structural fact.
+Those records are written by hand and live in the tree next to the plan they support.
+DVSim never produces them; it only passes their path through to whatever consumes this format, so they are out of scope for this document.
+
+## Consumers
+
+[DVPlan](https://github.com/lowRISC/dvplan) reads it to back-annotate a verification plan.
+It is not the only thing that could: the format carries no DVPlan concepts, and a dashboard or a CI job wanting machine-readable regression results can read the same file.
+
+## Changing it
+
+The pydantic models in `src/dvsim/report/dv_evidence.py` specify what a producer writes, and this document describes them.
+They are not the whole format: the `inspection` half and the rules for scoring a file are the consumer's, and [DVPlan](https://github.com/lowRISC/dvplan) specifies those.
+So the models here reject a file carrying an `inspection` key, which is correct, since DVSim writes evidence and never reads it back.
+A change to the models or to this document is a change to the format, so change both, and bear in mind that a consumer may be reading files this repo wrote months ago.

src/dvsim/flow/base.py

@@ -18,7 +18,7 @@
 
 import dvsim.instrumentation.runtime as instrumentation
 from dvsim.flow.hjson import set_target_attribute
-from dvsim.job.data import CompletedJobStatus, JobSpec, WorkspaceConfig
+from dvsim.job.data import CompletedJobStatus, JobSpec, JobStatusInfo, WorkspaceConfig
 from dvsim.job.status import JobStatus
 from dvsim.logging import log
 from dvsim.scheduler.resources import UnknownResourcePolicy
@@ -491,9 +491,20 @@ def deploy_objects(self) -> Sequence[CompletedJobStatus]:
                 interactive=self.interactive,
                 backend=backend,
                 resource_manager=resource_manager,
+                on_job_completed=self.on_job_completed,
             )
         )
 
+    def on_job_completed(
+        self, spec: JobSpec, status: JobStatus, reason: JobStatusInfo | None
+    ) -> None:
+        """Observe a job reaching a terminal state. Flows that care override this.
+
+        Called by the scheduler as each job finishes, so a flow can record an outcome while the
+        run is still going. Not abstract, since observing this is opt-in.
+        """
+        del spec, status, reason
+
     @abstractmethod
     def gen_results(self, results: Sequence[CompletedJobStatus]) -> None:
         """Generate flow results.

src/dvsim/job/data.py

@@ -10,6 +10,7 @@
 """
 
 from collections.abc import Callable, Mapping, Sequence
+from enum import Enum
 from pathlib import Path
 from typing import TypeAlias
 
@@ -20,12 +21,30 @@
 
 __all__ = (
     "CompletedJobStatus",
+    "DependencyPolicy",
     "JobSpec",
     "JobStatusInfo",
     "WorkspaceConfig",
 )
 
 
+class DependencyPolicy(Enum):
+    """When a job may start, given how the jobs it depends on ended."""
+
+    ALL_PASSING = "all_passing"
+    """Start only if every dependency passed, which is right for a job consuming their output."""
+
+    ANY_PASSING = "any_passing"
+    """Start if at least one dependency passed, for a job gathering whatever results exist."""
+
+    ALWAYS = "always"
+    """Start once every dependency is terminal, whatever they concluded.
+
+    For a job whose input is the outcome itself rather than an artefact a dependency produced, so
+    a regression where nothing passed is still the thing it has to report on.
+    """
+
+
 class WorkspaceConfig(BaseModel):
     """Workspace configuration."""
 
@@ -92,8 +111,8 @@ class JobSpec(BaseModel):
 
     dependencies: list[str]
     """Full names of the other Jobs that this one depends on."""
-    needs_all_dependencies_passing: bool
-    """Wait for dependent jobs to pass before scheduling."""
+    dependency_policy: DependencyPolicy
+    """What the jobs this one depends on must have concluded before it may be scheduled."""
     weight: int
     """Weight to apply to the scheduling priority."""
     timeout_mins: float | None