Rosa Starter Kit - EVIDENCE
===========================

This folder is the versioned benchmark evidence for the two published Rosa
figures (Test 1 and COMPAS). Its purpose is that a third party can REPLAY every
published band from this kit alone.

  benchmark-evidence.json   For each benchmark: the exact engine version and
                            container digest that produced it, the instance
                            profile, the SHA-256 of the shipped training input +
                            config, all ten Training/Inference job IDs, the ten
                            individual per-run values (not just the mean and
                            range), and the published band. Also records known
                            data caveats.
  reproduce-environment.txt The reference Python + package versions + platform
                            the published DOWNSTREAM values were computed in. The
                            reproduce-*.py scripts are plain scikit-learn; pin
                            these to match our numbers exactly.
  CHANGELOG.md              This kit's version history.
  Test1-provenance.md       How the Test 1 synthetic benchmark is generated.
  COMPAS-provenance/        How the COMPAS two-year-cohort split is built from
                            ProPublica's public source: the pinned source + its
                            SHA-256, the licence/citation, the exact cleaning +
                            group-aware split recipe, the leakage audit and a
                            data note - as a human-readable doc and a
                            machine-readable JSON, plus build_compas_benchmark.py,
                            the runnable build script itself, so the split is not
                            just described but independently re-derivable: run the
                            script on the pinned source and get the same splits.
  data-checks/              For each shipped dataset (Test 1 and COMPAS): a
                            <name>-findings.json - the report Rosa's own Data
                            Checker produces on that dataset (COMPAS surfaces the
                            time_in_jail negatives and heavy-tailed count columns;
                            Test 1 comes back clean) - and a <name>-datasheet
                            (.json + .md) declaring the expected column types,
                            valid ranges and any KNOWN retained anomalies. The
                            findings are advisory heuristics generated by that
                            client-side Checker, not a verdict; the datasheet is a
                            declared data contract, not a runtime validator.
  interpreting-results.md   How to read the published figures (why a replication
                            inside our range is agreement, not a discrepancy), the
                            permutation grid's resolution (24 on the trial ->
                            0.04; raise it per instance for evidential work, and
                            why a near-threshold verdict can move between runs),
                            our position on re-baselining, and - in the closing
                            section - the RESIDUAL-RISK STATEMENT: what a "no
                            recoverable bias" result does and does not establish,
                            including that independent probes have recovered the
                            attribute above chance from output this test reported
                            as clean.
  validate-your-own-run.md  The companion to that file, for YOUR OWN run rather
                            than our published ones: six checks to run on a Rosa
                            job you did on your own data. Manifest hashes, row
                            identity, pass-through integrity, lineage, a
                            downstream utility metric, and an external recovery
                            probe. The first four are mechanical and catch the
                            failures that are otherwise silent; the last two
                            answer what the manifest cannot see.

Validating a run of your own
----------------------------
The replay steps below check OUR figures. If what you want is confidence in a job
you ran on YOUR data, go to validate-your-own-run.md instead - start with its
checks 1 to 3, which need only the files and the Run Manifest.

How to replay a figure
----------------------
1. Run Rosa on the benchmark's Training split (see the config in its folder),
   then an Inference job on the Test/held-out split using that training job.
   Download the two <name>_fair.csv debiased outputs.
2. Run the benchmark's reproduce-*.py against the raw splits + your two _fair.csv
   files, in the reference environment above.
3. Compare your per-run value to the per-run values in benchmark-evidence.json
   and to the published band. A single run varies (the engine is stochastic); the
   published figure is the mean of that benchmark's independent runs, and the
   count is recorded beside it.
