Repository navigation
Close the evalhub.eval/1.0 window: a 1.0 body is 422 runs_moved, GET /schemas/eval is gone (0.4.0) - #54
Merged
Conversation
…/schemas/eval is gone (0.4.0)
The schema crate's versioning rule keeps the previous major accepted for
one release after the new one ships. evalhub.eval/2.0 shipped in 0.2.0,
and 0.2.0 and 0.3.0 still accepted an evalhub.eval/1.0 body (header and
runs[] in one) on POST /evals/{ns}/{name}, converting it at ingest into a
2.0 header plus run rows and answering with a Deprecation header and
converted_from / converted_runs (#45). 0.3.0 shipped with the window still
open, so every sentence that said "0.3.0 removes this" was false (#51);
the removal is the next minor, 0.4.0.
Visible breaks:
- A body declaring evalhub.eval/1.0, with or without runs, is 422 with
exactly one error, runs_moved at /runs: the error a 2.0 header carrying
runs already gets. No Deprecation header, no conversion, nothing is
written. The hint names the only accepted identifier and both run
endpoints (PUT /evals/{ns}/{name}/runs/{run_id},
POST /evals/{ns}/{name}/runs:batch). The server returns that error
before it decodes attachments[].sha256, so a bad digest on such a body
does not add a second error (it did for a 2.0 header with runs too).
ErrorCode::RunsMoved's doc, and so schemas/error.json and the OpenAPI
document, name the 1.0 case.
- GET /schemas/eval is 404. SCHEMA_NAMES is card, eval-2, run, error,
query; RecordKind::Eval.schema_ids() is only evalhub.eval/2.0 and
schema_for("evalhub.eval/1.0") is None.
- evalhub-schema: eval::v1, schema_for_eval_v1 and schemas/eval.json are
removed. evalhub-core: the eval module (EvalSchema, declared_schema,
split_v1, SplitV1, UNRECORDED, V1_MIGRATION_META_KEY) is removed; the
validator's 1.0 arm (runs[].calls / artifacts[] attachment checks) goes
with it. cargo semver-checks reports a major change for both crates.
- The version envelope loses converted_from and converted_runs, and the
OpenAPI document loses ConvertedRunsDto; the web client types are
regenerated from the new snapshot.
A 0.1.x database must still migrate, so the conversion is not deleted but
moved: split_v1 and its constants live in evalhub_store's
data_migrations::v1, internal to evalhub-store (an unpublished crate;
the module is pub only for the store's own tests), the reading of the
1.0 shape that 0003_runs_split uses, over serde_json::Value and only the
keys the conversion needs. No typed 1.0 struct is kept. Its tests moved from
evalhub-core to evalhub-store/tests/runs_split_v1.rs and need no
database; the migration test is unchanged and green.
In the store, the 1.0 arm of ingest (the split, write_converted_runs,
Converted, Ingested) is removed; create_or_append is the one entry point
again and stores the body it is given, the identifier check being the
validator's. The server drops EVAL_V1_DEPRECATION and the Deprecation
header. The server test for the conversion is now the runs_moved
assertion, the boot test asserts the 404, and the core fixtures that
declare 1.0 expect runs_moved (the 1.0 date-time case went with the arm).
README (API, Upgrading from 0.1.x), the schema crate's versioning section
and every crate doc say the window was open in 0.2.0 and 0.3.0 and
closed in 0.4.0; the 0.1.x to 2.0 mapping stays as history. Version
0.3.0 → 0.4.0 (SDK crates evalhub-schema and evalhub-core break their
public API).
Closes #45
Closes #51
Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0.4.0: the
evalhub.eval/1.0compatibility window is closedCloses #45 and #51. Releases 0.2.0 and 0.3.0 accepted an
evalhub.eval/1.0body (header andruns[]in one document) onPOST /evals/{ns}/{name}and converted it at ingest into a 2.0 header and its runs, answering with aDeprecationheader andconverted_from/converted_runsin the envelope. The schema crate's rule is that the previous major is accepted for one release and then refused; the README and the crate docs said 0.3.0 would refuse it, and 0.3.0 shipped with the window open (#51). 0.4.0 closes it.Behaviour
POST /evals/{ns}/{name}withschema: evalhub.eval/1.0answers422with exactly one error,runs_movedat/runs, with or withoutrunsand whatever else is wrong with the body. It is the error a 2.0 header carryingrunsalready got. NoDeprecationheader, noconverted_from, nothing is written. The hint namesPUT /evals/{ns}/{name}/runs/{run_id}andPOST /evals/{ns}/{name}/runs:batch.runswhoseattachments[].sha256is malformed also answers that singleruns_moved(previously the sha error was appended after it).GET /schemas/eval(the 1.0 document) is gone:404.GET /schemas/eval-2andGET /schemas/runare unchanged.0003_runs_splitkeeps its own reading of the 1.0 shape insideevalhub-store(unpublished), byte-for-byte the conversionevalhub-coreused to expose.422after this deploy; the README's 0.1.x → 0.2.0 mapping stays as the record of what to change.SDK (why 0.4.0)
just semver-checkreports major forevalhub-schemaandevalhub-core;evalhub-queryis unchanged.evalhub-schema:eval::v1(Eval,Run,Outcome) andschema_for_eval_v1()removed;SCHEMA_NAMESis[&str; 5](evaldropped; cargo-semver-checks does not flag the array length);RecordKind::Eval.schema_ids()is["evalhub.eval/2.0"],schema_for("evalhub.eval/1.0")isNone,all_schemas()no longer yieldseval;schemas/eval.jsonremoved;ErrorCode::RunsMoveddoc covers the 1.0 declaration.evalhub-core:EvalSchemaanddeclared_schemaremoved (the 1.0 arm of the validator). Theevalmodule went with them:split_v1,SplitV1,UNRECORDED,V1_MIGRATION_META_KEYnow live inevalhub-storefor the data migration only.validate(Eval, …)on a 1.0 body returns[runs_moved].ConvertedRunsDto,converted_from,converted_runsremoved; the web client (schema.d.ts) regenerated.Docs
Every sentence that promised the removal for 0.3.0 (README §API, §Upgrading from 0.1.x; rustdoc in schema, core, store, server) now records that 0.2.0 and 0.3.0 accepted 1.0 and 0.4.0 closed the window.
web/README.mdnameseval-2instead of the removedeval.Verification
cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test -p evalhub-schema(20) /-core(56) /-query(27) /-store(104, Docker) /-server(86, Docker),cargo insta test -p evalhub-schemaand-p evalhub-server(no pending snapshots),RUSTDOCFLAGS=-D warnings cargo docfor schema / core / store,just semver-check(major for schema and core before the bump; "no semver update required" for all three at 0.4.0),just package(three crates ok),corepack pnpm check/pnpm lintinweb/. RED observed before each behaviour change:schema_ids()still listed 1.0; a 1.0 body validated with zero or unrelated errors;POSTanswered201withconverted_from;GET /schemas/evalanswered200; a malformed sha256 produced two errors.Fly is deployed separately after the release (
fly deploy --ha=false); nothing stored changes, so no pre-deploy data check is needed. Producers still posting 1.0 bodies get422 runs_movedfrom then on.