Repository navigation
Add benchmark summaries and README translation policy #19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Check skill package | |
| on: [push, pull_request] | |
| permissions: | |
| contents: read | |
| jobs: | |
| check: | |
| runs-on: ubuntu-latest | |
| env: | |
| PYTHONDONTWRITEBYTECODE: '1' | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-python@v5 | |
| with: | |
| python-version: '3.12' | |
| - name: Check skill package | |
| run: python3 -m unittest discover -s tests -p 'test_*.py' | |
| - name: Check benchmark harness | |
| run: python3 -m unittest discover -s benchmarks -p 'test_*.py' | |
| - name: Check offline reporting | |
| run: python3 -m unittest discover -s benchmarks/reporting -p 'test_*.py' | |
| - name: Verify original review evidence without model access | |
| run: | | |
| python3 benchmarks/reporting/verify.py benchmarks/results/2026-10-15-astra-v1 \ | |
| --judges benchmarks/results/2026-10-15-astra-v1/judgments \ | |
| --packets benchmarks/results/2026-10-15-astra-v1/review-packets \ | |
| --require-complete \ | |
| --output "$RUNNER_TEMP/align-original-review-report.json" | |
| - name: Verify corrected primary review without model access | |
| run: | | |
| python3 benchmarks/reporting/verify.py benchmarks/results/2026-10-15-astra-v1 \ | |
| --judges benchmarks/results/2026-10-15-astra-v1/judgments-v2 \ | |
| --packets benchmarks/results/2026-10-15-astra-v1/review-packets-v2 \ | |
| --require-complete \ | |
| --output "$RUNNER_TEMP/align-evidence-report.json" | |
| - name: Check corrected review quotes at their declared sources | |
| run: | | |
| python3 benchmarks/reporting/check_quote_sources.py \ | |
| benchmarks/results/2026-10-15-astra-v1/review-packets-v2 \ | |
| benchmarks/results/2026-10-15-astra-v1/judgments-v2 | |
| - name: Check scripted intervention harness and reporting | |
| run: python3 -m unittest discover -s benchmarks/effort -p 'test_*.py' | |
| - name: Check preference discovery harness and review | |
| run: python3 -m unittest discover -s benchmarks/discovery -p 'test_*.py' | |
| - name: Check preference review replay | |
| run: python3 -m unittest discover -s benchmarks/discovery/reporting -p 'test_*.py' | |
| - name: Verify retained intervention infrastructure failure | |
| run: | | |
| python3 benchmarks/effort/verify.py benchmarks/effort/results/2026-10-15-astra-v1-failed \ | |
| --output "$RUNNER_TEMP/align-effort-failed-verification.json" | |
| - name: Verify scripted intervention results without model access | |
| run: | | |
| python3 benchmarks/effort/verify.py benchmarks/effort/results/2026-10-15-astra-v2 \ | |
| --output "$RUNNER_TEMP/align-effort-verification.json" | |
| - name: Verify preference transfer results without model access | |
| run: | | |
| python3 benchmarks/discovery/verify.py benchmarks/discovery/results/2026-10-15-astra-v1 \ | |
| --output "$RUNNER_TEMP/align-discovery-verification.json" | |
| - name: Verify blinded preference review without model access | |
| run: | | |
| python3 benchmarks/discovery/verify_reviews.py "$PWD/benchmarks/discovery/results/2026-10-15-astra-v1" \ | |
| --packets "$PWD/benchmarks/discovery/results/2026-10-15-astra-v1/review-packets" \ | |
| --reviews "$PWD/benchmarks/discovery/results/2026-10-15-astra-v1/judgments" \ | |
| --output "$RUNNER_TEMP/align-discovery-review-verification.json" | |
| - name: Check preference retention protocol | |
| run: python3 -m unittest discover -s benchmarks/retention -p 'test_*.py' | |
| - name: Check retention review replay | |
| run: python3 -m unittest discover -s benchmarks/retention/reporting -p 'test_*.py' | |
| - name: Verify recovered retention evidence without model access | |
| run: | | |
| python3 benchmarks/retention/recovery/verify.py benchmarks/retention/results/2026-10-15-astra-v1 \ | |
| --output "$RUNNER_TEMP/align-retention-verification.json" | |
| - name: Verify masked retention review without model access | |
| run: | | |
| python3 benchmarks/retention/reporting/verify_review.py benchmarks/retention/results/2026-10-15-astra-v1 \ | |
| --output "$RUNNER_TEMP/align-retention-review-verification.json" |