From a382398a31e79b205f60735eab5e20e6980af92d Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 14 Aug 2026 20:49:38 -0400 Subject: [PATCH] =?UTF-8?q?test(hosted):=20retire=20the=20gem=20catalog=20?= =?UTF-8?q?pin=20=E2=80=94=20production=20unpublished=20its=20last=20free?= =?UTF-8?q?=20gem=20patch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Production withdrew pkg:gem/activestorage@7.0.2.2 (uuid 2535d43d-67ce-4944-be27-c19e113997fb, GHSA-w749-p3v6-hccq) — the only free-tier gem patch it ever published. /patch/view/ now 404s (the npm pin's still 200s) and a version-less probe of every gem name in ruby-advisory-db (464 packages) returns zero free patches, so there is no honest replacement pin: the preflight's "pick a replacement" instruction has nothing to pick. Gem therefore joins the suite's existing pattern for patch-less ecosystems (maven / nuget / composer): * drop the GEM_* catalog entries from the preflight and the merge-state canary (npm / PyPI / cargo remain); * add gem to UNPUBLISHED_ECOSYSTEMS — first candidate is the withdrawn pin itself — so canary_unpublished_ecosystems reports the moment production republishes a gem patch; * retire gem_bundler_hosted_redirect_and_known_install_defect and its gem-only helpers (gem_lock_checksum, installed_gem_dir, gem_registry_base, http_probe, published_patch_files): with no published patch neither its known-defect arm nor its success arm can execute. Restoration recipe is in docs/testing/hosted-production-e2e.md; the rewrite grammar stays covered by e2e_redirect_gem_build.rs against a mock; * ci: drop the hosted-e2e job's now-unneeded ruby/setup-ruby step; * docs: catalog + coverage tables updated, retired-leg section added, the compact-index server-defect entry kept for the record, and the SOCKET_PATCH_HOSTED_E2E_GEM_STRICT knob removed with the leg. Verified live against production: 15 passed / 0 failed (the three previously-broken tests pass; every install-proof leg stays green). NOTE: e2e_gem.rs's pkg:gem/activestorage@5.2.0 pin (4bf7fe0b-dc57-4ea8-945f-bc4a04c47a15) and e2e_vendored_production.rs's pin of the same 7.0.2.2 patch are ALSO unpublished — separate follow-up; neither suite is in CI. Co-Authored-By: Claude Fable 5 --- .github/workflows/ci.yml | 14 +- .../tests/e2e_hosted_production.rs | 456 ++---------------- docs/testing/hosted-production-e2e.md | 83 +++- 3 files changed, 109 insertions(+), 444 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 42379944..c27a7eab 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1090,15 +1090,11 @@ jobs: if: steps.gate.outputs.run == 'true' run: python -m pip install --disable-pip-version-check uv && uv --version - - name: Setup Ruby - if: steps.gate.outputs.run == 'true' - uses: ruby/setup-ruby@319994f95fa847cf3fb3cd3dbe89f6dcde9f178f # v1.295.0 - with: - ruby-version: '3.2.10' - # The gem hosted rewrite pins into the Gemfile.lock CHECKSUMS - # section, which `bundle lock --add-checksums` only emits on >= 2.6. - bundler: '2.6' - bundler-cache: false + # No Ruby setup: the bundler leg was retired in 2026-08 when production + # withdrew its last free-tier gem patch (see + # docs/testing/hosted-production-e2e.md). Reinstate ruby/setup-ruby + # (with bundler >= 2.6 for `bundle lock --add-checksums`) when the gem + # leg is restored. - name: Setup Go if: steps.gate.outputs.run == 'true' diff --git a/crates/socket-patch-cli/tests/e2e_hosted_production.rs b/crates/socket-patch-cli/tests/e2e_hosted_production.rs index 437b3092..361a52d0 100644 --- a/crates/socket-patch-cli/tests/e2e_hosted_production.rs +++ b/crates/socket-patch-cli/tests/e2e_hosted_production.rs @@ -36,13 +36,24 @@ //! | npm | `pkg:npm/minimist@1.2.2` | `80630680-4da6-45f9-bba8-b888e0ffd58c` | GHSA-xvch-5gv4-984h (CVE-2021-44906) | //! | PyPI | `pkg:pypi/urllib3@1.26.18` | *any of three* (see [`PYPI_UUIDS`]) | GHSA-gm62-xv2j-4w53 &co | //! | Cargo | `pkg:cargo/traitobject@0.1.1` | `cf2e6f58-d9fa-4096-9151-c34afa717f89` | GHSA-pp8r-vv2j-9j5v | -//! | gem | `pkg:gem/activestorage@7.0.2.2` | `2535d43d-67ce-4944-be27-c19e113997fb` | GHSA-w749-p3v6-hccq | //! //! `docs/testing/hosted-production-e2e.md` explains how these were chosen and //! how to re-pick one if it is ever withdrawn. //! //! # Ecosystems with no coverage, and why //! +//! * **gem** — hosted mode is implemented (its rewrite grammar stays covered +//! by `e2e_redirect_gem_build.rs` against a mock), and this suite carried a +//! real bundler leg until 2026-08, when production withdrew its **last** +//! free-tier gem patch (`pkg:gem/activestorage@7.0.2.2`, uuid +//! `2535d43d-67ce-4944-be27-c19e113997fb`, GHSA-w749-p3v6-hccq). Production +//! now publishes zero free gem patches — every gem that has ever had a +//! ruby-advisory-db advisory probes empty — so there is no honest pin to +//! replace it with. The retired leg (redirect assertions, the CHECKSUMS +//! re-pin guard, the compact-index known-defect tolerance and its +//! content-verified success arm) lives in git history on this file; restore +//! it the moment [`canary_unpublished_ecosystems`] reports a published gem +//! patch. //! * **maven / nuget / composer** — hosted mode is implemented and documented //! for all three, but production currently publishes **zero** free-tier //! patches for them, so there is nothing real to redirect to. Rather than @@ -56,11 +67,11 @@ //! //! Toolchains (each leg soft-skips if its own toolchain is absent, unless //! `SOCKET_PATCH_HOSTED_E2E_STRICT=1`): `npm`, `pnpm`, `yarn` (classic), -//! `corepack` (berry), `bun`, `uv`, `cargo`, `ruby` + `bundle`, `go`. +//! `corepack` (berry), `bun`, `uv`, `cargo`, `go`. //! //! Network egress to: `patches-api.socket.dev`, `patch.socket.dev`, //! `registry.npmjs.org`, `pypi.org`, `files.pythonhosted.org`, -//! `static.crates.io`, `index.crates.io`, `rubygems.org`. +//! `static.crates.io`, `index.crates.io`. //! //! No API token is used or needed — the suite deliberately runs against the //! **free public proxy**, which is the surface every unauthenticated user @@ -126,18 +137,28 @@ const CARGO_UUID: &str = "cf2e6f58-d9fa-4096-9151-c34afa717f89"; /// that npm/PyPI artifacts carry, so this is the marker to look for. const CARGO_MARKER: &str = "GHSA-pp8r-vv2j-9j5v"; -const GEM_PURL: &str = "pkg:gem/activestorage@7.0.2.2"; -const GEM_NAME: &str = "activestorage"; -const GEM_VERSION: &str = "7.0.2.2"; -const GEM_UUID: &str = "2535d43d-67ce-4944-be27-c19e113997fb"; - /// Header the patch service injects into patched npm / PyPI source files. const PATCH_MARKER: &str = "Socket Community Patch"; /// Ecosystems where hosted mode is implemented but production has no free /// patches to exercise it with. [`canary_unpublished_ecosystems`] watches /// these so coverage can be extended the moment one lights up. +/// +/// `gem` is the newest member: this suite carried a real bundler leg pinned +/// to `pkg:gem/activestorage@7.0.2.2` until production withdrew that patch in +/// 2026-08, leaving the ecosystem with zero free patches (see the module +/// docs). Its first candidate below is the withdrawn pin itself, so the +/// canary lights up fastest on the most likely republish. const UNPUBLISHED_ECOSYSTEMS: &[(&str, &[&str])] = &[ + ( + "gem", + &[ + "pkg:gem/activestorage", + "pkg:gem/rails-html-sanitizer", + "pkg:gem/nokogiri", + "pkg:gem/rack", + ], + ), ( "maven", &[ @@ -558,117 +579,6 @@ async fn published_patch_dates(purl: &str) -> Result, Stri .unwrap_or_default()) } -/// `GET /patch/view/` against the real proxy — the same route the -/// CLI's free-proxy client fetches patch content from. Returns, per file the -/// patch touches, the `(path, beforeHash, afterHash)` triple (hashes are -/// git-blob sha256, `None` for pure additions/deletions respectively). -async fn published_patch_files( - uuid: &str, -) -> Result, Option)>, String> { - let url = format!("{PROXY}/patch/view/{uuid}"); - let resp = reqwest::Client::new() - .get(&url) - .header("Accept", "application/json") - .send() - .await - .map_err(|e| format!("GET {url}: {e}"))?; - let status = resp.status(); - let body = resp - .text() - .await - .map_err(|e| format!("GET {url}: reading body: {e}"))?; - if !status.is_success() { - return Err(format!("GET {url}: HTTP {status}\n{body}")); - } - let v: serde_json::Value = - serde_json::from_str(&body).map_err(|e| format!("GET {url}: bad JSON ({e}):\n{body}"))?; - Ok(v["files"] - .as_object() - .map(|m| { - m.iter() - .map(|(path, f)| { - ( - path.clone(), - f["beforeHash"].as_str().map(str::to_string), - f["afterHash"].as_str().map(str::to_string), - ) - }) - .collect() - }) - .unwrap_or_default()) -} - -/// The `sha256=` hex value the lock's CHECKSUMS section pins for -/// ` ()`, or `None` when the entry is absent. Bundler >= 2.6 -/// writes one two-space-indented ` name (version) sha256=` line per -/// resolved gem. -fn gem_lock_checksum(lock: &str, name: &str, version: &str) -> Option { - let prefix = format!(" {name} ({version}) sha256="); - lock.lines() - .find_map(|l| l.strip_prefix(&prefix).map(|h| h.trim().to_string())) -} - -/// Locate the bundler-installed `gems/-` directory under a -/// `BUNDLE_PATH` root. The `ruby/` segment in between varies by host -/// interpreter, so walk for it (depth-bounded — the layout is only a few -/// levels deep) instead of hardcoding the version. -fn installed_gem_dir(root: &Path, dir_name: &str, depth: usize) -> Option { - for entry in std::fs::read_dir(root).ok()?.flatten() { - let p = entry.path(); - if !p.is_dir() { - continue; - } - if p.file_name().is_some_and(|n| n == dir_name) - && p.parent() - .and_then(|d| d.file_name()) - .is_some_and(|n| n == "gems") - { - return Some(p); - } - if depth > 0 { - if let Some(found) = installed_gem_dir(&p, dir_name, depth - 1) { - return Some(found); - } - } - } - None -} - -/// The Socket patch-registry base URL the gem rewriter pinned into `Gemfile` -/// as `source "" do`, or `None` when no Socket source block is present. -/// -/// Read back out of the rewritten file rather than rebuilt from constants on -/// purpose: the probe must interrogate the *exact* registry bundler was told -/// to use, so a rewriter that emits the wrong base cannot be papered over by a -/// probe that guesses the right one. -fn gem_registry_base(gemfile: &str) -> Option { - const OPEN: &str = "source \""; - let marker = format!("{OPEN}https://{PATCH_HOST}/patch-registry/gem/"); - let at = gemfile.find(&marker)?; - let rest = &gemfile[at + OPEN.len()..]; - let end = rest.find('"')?; - Some(rest[..end].to_string()) -} - -/// `GET ` → `(status, body_len)`, or `Err` on a transport failure. -/// -/// Sends a bundler-shaped `User-Agent` so the probe observes whatever a real -/// `bundle install` would be served. -async fn http_probe(url: &str) -> Result<(u16, usize), String> { - let resp = reqwest::Client::new() - .get(url) - .header("User-Agent", "bundler/2.6.9 rubygems/3.6.9") - .send() - .await - .map_err(|e| e.to_string())?; - let status = resp.status().as_u16(); - let body = resp - .bytes() - .await - .map_err(|e| format!("reading body: {e}"))?; - Ok((status, body.len())) -} - /// Percent-encode a PURL for use as a single path segment. `reqwest` will not /// do this for us — a raw `pkg:npm/...` would be split into path segments and /// 404. @@ -702,7 +612,6 @@ async fn preflight_required_patches_are_published() { (NPM_PURL, vec![NPM_UUID]), (PYPI_PURL, PYPI_UUIDS.to_vec()), (CARGO_PURL, vec![CARGO_UUID]), - (GEM_PURL, vec![GEM_UUID]), ]; let mut failures: Vec = Vec::new(); @@ -755,7 +664,7 @@ async fn canary_patches_name_advisories_so_merge_state_is_inferable() { let mut failures: Vec = Vec::new(); let mut coverage_seen: Vec<(String, String, usize)> = Vec::new(); - for purl in [NPM_PURL, PYPI_PURL, CARGO_PURL, GEM_PURL] { + for purl in [NPM_PURL, PYPI_PURL, CARGO_PURL] { match published_patch_advisory_counts(purl).await { Err(e) => failures.push(format!("{purl}: production probe failed: {e}")), Ok(patches) if patches.is_empty() => { @@ -1607,294 +1516,6 @@ fn cargo_hosted_install_proof() { assert_patched(&lib_rs, CARGO_MARKER, LEG); } -// =========================================================================== -// RubyGems — redirect works; the hosted install is blocked by a SERVER defect -// =========================================================================== - -/// The gem redirect itself is correct and is asserted hard here. -/// -/// The **install** leg is a different story. Socket's gem patch-registry serves -/// a compact index whose `/info/` line declares **no runtime -/// dependencies**, while the `.gem` it serves declares six. Bundler's -/// `ensure_same_dependencies` check fails closed: -/// -/// ```text -/// Bundler::APIResponseMismatchError: Downloading activestorage-7.0.2.2 -/// revealed dependencies not in the API (activesupport (= 7.0.2.2), ...) -/// ``` -/// -/// Compare production's own index, which does emit them: -/// `https://index.rubygems.org/info/activestorage` → -/// `7.0.2.2 actionpack:= 7.0.2.2,activejob:= 7.0.2.2,...|checksum:...` -/// versus `patch.socket.dev/patch-registry/gem///info/activestorage` -/// → `7.0.2.2 |checksum:...`. -/// -/// That is a **server-side** defect, not a CLI one, and it blocks hosted gem -/// mode for any gem with runtime dependencies. Until it is fixed the install -/// leg reports loudly but does not fail the suite; set -/// `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` to promote it to a hard failure -/// (do that as the regression guard once the server is fixed). -/// -/// # Why the tolerance probes the server instead of matching the error text -/// -/// This leg used to accept the install failure by string-matching bundler's -/// message. That couples a CI check to one particular *symptom* of the server -/// bug, and the symptom is a function of which fetcher bundler lands on — -/// which the server keeps changing. Bundler selects one via -/// `available_fetchers.drop_while {|f| !f.available? }` over `[CompactIndex, -/// Dependency, Index]`, so: -/// -/// | server state | `/versions` | `/api/v1/dependencies` | bundler raises | -/// |---|---|---|---| -/// | originally | 200, empty dep segment | — | `Bundler::APIResponseMismatchError` | -/// | after depscan#23630 | 404 `not_built` | **200, zero-byte body** | `ArgumentError: marshal data too short` (classic Marshal) / `NoMethodError: undefined method 'bytes' for nil` (SafeMarshal, ruby 3.4+) | -/// | after the empty-body fix | 404 `not_built` | 404 | `Could not fetch specs from …` | -/// -/// Three different strings for one unchanged server condition. A whitelist of -/// them goes stale on every server deploy and reds the check for a reason that -/// has nothing to do with socket-patch. -/// -/// So the tolerance is decided by the **condition**, not the symptom: probe -/// the pinned registry's `/versions` — the URL bundler was actually given, -/// read back out of the rewritten `Gemfile`. -/// -/// * non-2xx → the documented server defect. The install failure is expected; -/// report loudly and pass (unless `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1`). -/// * 2xx → the compact index is **built**, so hosted gem mode MUST work. An -/// install failure is then a real regression and fails the suite. -/// -/// That is symptom-independent, and it **auto-retires itself**: the moment the -/// server is healthy the 2xx branch starts enforcing a real success assertion, -/// with no stale whitelist and no `NOTE` asking a human to clean up. -#[tokio::test(flavor = "multi_thread")] -#[ignore = "live production API + real rubygems.org. Run with --ignored."] -async fn gem_bundler_hosted_redirect_and_known_install_defect() { - const LEG: &str = "gem_bundler_hosted_redirect_and_known_install_defect"; - if !has_command("ruby") || !has_command("bundle") { - soft_skip!(LEG, "`ruby` and/or `bundle` not on PATH"); - } - let tmp = tempfile::tempdir().expect("tempdir"); - let proj = tmp.path().join("proj"); - std::fs::create_dir_all(&proj).expect("mkdir proj"); - let bundle_path = tmp.path().join("bundle").display().to_string(); - let env = [ - ("BUNDLE_PATH", bundle_path.as_str()), - ("BUNDLE_APP_CONFIG", bundle_path.as_str()), - ]; - - std::fs::write( - proj.join("Gemfile"), - format!("source \"https://rubygems.org\"\ngem \"{GEM_NAME}\", \"{GEM_VERSION}\"\n"), - ) - .expect("write Gemfile"); - - // `--add-checksums` produces the CHECKSUMS section the hosted rewrite pins - // into; it needs bundler >= 2.6. - if !ok(&tool(&proj, "bundle", &["lock", "--add-checksums"], &env)) { - soft_skip!( - LEG, - "`bundle lock --add-checksums` failed (bundler < 2.6 has no \ - CHECKSUMS section)" - ); - } - // Capture the UPSTREAM checksum pin before any redirect. Bundler installs - // whatever matches this pin, so the redirect must replace it with the - // patched artifact's digest — the assertion after the redirect below is - // what makes an inert rewrite (URL repointed, upstream digest kept, so - // bundler verify-and-installs the UNPATCHED gem) go red instead of green. - let pristine_lock = read(&proj.join("Gemfile.lock")); - let upstream_sha = - gem_lock_checksum(&pristine_lock, GEM_NAME, GEM_VERSION).unwrap_or_else(|| { - panic!( - "{LEG}: `bundle lock --add-checksums` wrote no sha256 CHECKSUMS \ - entry for {GEM_NAME} ({GEM_VERSION}):\n{pristine_lock}" - ) - }); - let install = tool(&proj, "bundle", &["install", "--quiet"], &env); - if !ok(&install) { - soft_skip!(LEG, "upstream `bundle install` failed:\n{}", dump(&install)); - } - - let env_json = scan_hosted(&proj, &[]); - assert_redirected(&env_json, "Gemfile.lock"); - - // Hard assertions: the redirect itself must be correct. - let gemfile = read(&proj.join("Gemfile")); - assert!( - gemfile.contains(&format!("https://{PATCH_HOST}/patch-registry/gem/")) - && gemfile.contains(GEM_UUID), - "{LEG}: Gemfile carries no per-dep Socket source block for \ - {GEM_UUID}:\n{gemfile}" - ); - let lock = read(&proj.join("Gemfile.lock")); - assert!( - lock.contains("CHECKSUMS"), - "{LEG}: Gemfile.lock lost its CHECKSUMS section:\n{lock}" - ); - let redirected_sha = gem_lock_checksum(&lock, GEM_NAME, GEM_VERSION).unwrap_or_else(|| { - panic!( - "{LEG}: redirected Gemfile.lock carries no sha256 CHECKSUMS entry \ - for {GEM_NAME} ({GEM_VERSION}):\n{lock}" - ) - }); - assert_ne!( - redirected_sha, upstream_sha, - "{LEG}: the redirect left {GEM_NAME}'s CHECKSUMS pin at the UPSTREAM \ - sha256 — bundler would verify and install the unpatched artifact, so \ - the hosted patch is inert (the same blindspot that let an inert npm \ - patch stay green).\nGemfile.lock:\n{lock}" - ); - - // Known-broken leg: reinstall from the redirected Gemfile. - std::fs::remove_dir_all(&bundle_path).ok(); - let reinstall = tool(&proj, "bundle", &["install"], &env); - let gem_strict = std::env::var("SOCKET_PATCH_HOSTED_E2E_GEM_STRICT") - .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) - .unwrap_or(false); - if ok(&reinstall) { - // The server defect has been fixed. Say so loudly — the guard below - // should be promoted to unconditional and this branch deleted. - println!( - "NOTE {LEG}: `bundle install` from the redirected Gemfile now \ - SUCCEEDS. The gem patch-registry compact-index dependency defect \ - appears to be FIXED — delete the tolerance branch in this test and \ - assert unconditionally." - ); - // Exit 0 proves only that bundler fetched an artifact matching the - // CHECKSUMS pin. Close the loop on CONTENT: fetch the patch's file - // manifest from the proxy and assert every file it rewrites landed - // on disk byte-exact (afterHash is the git-blob sha256 the patch - // service publishes — the same digest the CLI's apply verifies). - let patch_files = published_patch_files(GEM_UUID).await.unwrap_or_else(|e| { - panic!( - "{LEG}: `bundle install` from the redirected Gemfile succeeded \ - but the patch file manifest could not be fetched to verify \ - the installed content: {e}" - ) - }); - let gem_dir = installed_gem_dir( - Path::new(&bundle_path), - &format!("{GEM_NAME}-{GEM_VERSION}"), - 4, - ) - .unwrap_or_else(|| { - panic!( - "{LEG}: `bundle install` succeeded but no \ - gems/{GEM_NAME}-{GEM_VERSION} directory exists under \ - {bundle_path}" - ) - }); - use socket_patch_core::hash::git_sha256::compute_git_sha256_from_bytes; - let mut verified = 0usize; - let mut rewritten = 0usize; - for (path, before, after) in &patch_files { - // No afterHash = the patch deletes the file; nothing to hash. - let Some(after) = after else { continue }; - let rel = path.strip_prefix("package/").unwrap_or(path.as_str()); - let installed = gem_dir.join(rel); - let bytes = std::fs::read(&installed).unwrap_or_else(|e| { - panic!( - "{LEG}: patch {GEM_UUID} rewrites `{path}` but the \ - installed gem has no readable {}: {e}", - installed.display() - ) - }); - assert_eq!( - compute_git_sha256_from_bytes(&bytes), - *after, - "{LEG}: installed {} does not hash to the patch's afterHash — \ - bundler fetched an artifact whose content is NOT the \ - published patch", - installed.display() - ); - verified += 1; - if before.as_deref() != Some(after.as_str()) { - rewritten += 1; - } - } - assert!( - verified >= 1, - "{LEG}: patch {GEM_UUID} names no files with an afterHash, so \ - nothing was content-verified — the install success is vacuous" - ); - assert!( - rewritten >= 1, - "{LEG}: every file in patch {GEM_UUID} has afterHash == \ - beforeHash — the published patch is inert and this install \ - proved nothing" - ); - println!( - "{LEG}: verified {verified} patched file(s) on disk against the \ - published afterHash ({rewritten} differ from upstream)" - ); - return; - } - let detail = dump(&reinstall); - - // Ask the SERVER what state it is in, rather than guessing from bundler's - // error text (see the doc comment above for why the text is untrustworthy). - // The base is read back out of the rewritten Gemfile, so this probes the - // exact registry bundler was pointed at. - let index_base = gem_registry_base(&gemfile).unwrap_or_else(|| { - panic!( - "{LEG}: could not read the Socket registry base back out of the \ - rewritten Gemfile, so the install failure cannot be attributed. \ - The redirect assertions above passed, so the `source \"…\" do` \ - block shape must have changed:\n{gemfile}" - ) - }); - let versions_url = format!("{}/versions", index_base.trim_end_matches('/')); - let probe = http_probe(&versions_url).await; - let probe_note = match &probe { - Ok((status, len)) => { - format!("GET {versions_url} -> HTTP {status}, {len}-byte body") - } - Err(e) => format!("GET {versions_url} -> transport error: {e}"), - }; - - // Strict mode promotes ANY install failure to a hard failure, whatever the - // server state — that is its whole purpose as the regression guard. - assert!( - !gem_strict, - "{LEG}: SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1 and `bundle install` from \ - the redirected Gemfile failed.\n registry probe: {probe_note}\n{detail}" - ); - - // A 2xx `/versions` means the compact index is BUILT and bundler was - // served a usable index. The documented server defect therefore does NOT - // apply, and tolerating the failure here would hide a real regression. - if let Ok((status, _)) = probe { - assert!( - !(200..300).contains(&status), - "{LEG}: `bundle install` from the redirected Gemfile FAILED even \ - though the pinned registry's compact index is SERVING.\n \ - registry probe: {probe_note}\n\ - A 2xx /versions means `package_gem_index_deps` is populated and \ - the index is built, so this is NOT the known server defect \ - (which 404s that route) — it is a real regression in hosted gem \ - mode. The redirect assertions above all passed, so the rewrite \ - itself is fine and the failure is in the install leg.\n{detail}" - ); - } - - // Non-2xx (or an unreachable registry): the documented server defect. - // A transport error is tolerated rather than failed because a network - // blip is the most likely explanation for BOTH the probe and the install - // failing, and a required check must not go red for one. - println!( - "KNOWN PRODUCTION DEFECT {LEG}: the Socket gem patch-registry's \ - compact index is not being served for this patch, so `bundle \ - install` from the redirected Gemfile cannot succeed.\n registry \ - probe: {probe_note}\n\ - Since depscan#23630 the compact-index routes fail closed with 404 \ - `not_built` until the requeued rebuild populates \ - `package_gem_index_deps`. Hosted gem mode stays unusable for gems \ - with dependencies until that completes. Redirect assertions above \ - all passed; this leg will start asserting a real successful install \ - automatically once /versions returns 2xx." - ); -} - // =========================================================================== // Documented negative cases // =========================================================================== @@ -1974,10 +1595,12 @@ fn deno_hosted_is_unsupported() { // Canary — ecosystems whose hosted support has nothing to test against // =========================================================================== -/// maven, nuget and composer all implement hosted mode, but production +/// gem, maven, nuget and composer all implement hosted mode, but production /// publishes no free-tier patches for them, so there is no honest end-to-end -/// leg to write. This probes production every run and reports the moment that -/// changes, so coverage can be extended deliberately rather than by accident. +/// leg to write (gem HAD one until production withdrew its last free gem +/// patch in 2026-08 — see the module docs). This probes production every run +/// and reports the moment that changes, so coverage can be extended (or, for +/// gem, restored from git history) deliberately rather than by accident. /// /// It deliberately does NOT fail when patches appear: production publishing a /// new patch is not a socket-patch regression, and a required check must not @@ -2010,9 +1633,9 @@ async fn canary_unpublished_ecosystems() { if newly_published.is_empty() { println!( - "canary_unpublished_ecosystems: maven / nuget / composer still have \ - no free-tier published patches — their hosted-mode legs remain \ - untestable end-to-end against production." + "canary_unpublished_ecosystems: gem / maven / nuget / composer \ + still have no free-tier published patches — their hosted-mode \ + legs remain untestable end-to-end against production." ); return; } @@ -2020,7 +1643,8 @@ async fn canary_unpublished_ecosystems() { let msg = format!( "production now publishes free patches for previously-empty \ ecosystems:\n - {}\nExtend this suite with real install proofs for \ - them (see docs/testing/hosted-production-e2e.md).", + them — for gem, restore the retired bundler leg from git history \ + (see docs/testing/hosted-production-e2e.md).", newly_published.join("\n - ") ); if std::env::var("SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT").as_deref() == Ok("1") { diff --git a/docs/testing/hosted-production-e2e.md b/docs/testing/hosted-production-e2e.md index 5886e80e..3e1fb72d 100644 --- a/docs/testing/hosted-production-e2e.md +++ b/docs/testing/hosted-production-e2e.md @@ -41,14 +41,19 @@ from the child environment. | npm | `pkg:npm/minimist@1.2.2` | `80630680-4da6-45f9-bba8-b888e0ffd58c` | GHSA-xvch-5gv4-984h / CVE-2021-44906 | all five npm-family legs | | PyPI | `pkg:pypi/urllib3@1.26.18` | `de58c8b8-796c-4b6d-8a48-539b5563db76`, `26242e35-f867-4da8-8789-f0d2ea49e0f1`, `e828efa5-5c6d-43f3-9909-03f5ac232b98` | GHSA-38jv-5279-wg99, GHSA-2xpw-w6gg-jr37, GHSA-gm62-xv2j-4w53 | requirements.txt, uv.lock | | Cargo | `pkg:cargo/traitobject@0.1.1` | `cf2e6f58-d9fa-4096-9151-c34afa717f89` | GHSA-pp8r-vv2j-9j5v | cargo sparse-registry leg | -| RubyGems | `pkg:gem/activestorage@7.0.2.2` | `2535d43d-67ce-4944-be27-c19e113997fb` | GHSA-w749-p3v6-hccq | bundler leg | + +RubyGems is no longer in the catalog: the suite was pinned to +`pkg:gem/activestorage@7.0.2.2` (`2535d43d-67ce-4944-be27-c19e113997fb`, +GHSA-w749-p3v6-hccq) until 2026-08, when production withdrew it — and with it +the **last** free-tier gem patch. See +[RubyGems: the retired bundler leg](#rubygems-the-retired-bundler-leg). urllib3 1.26.18 carries **three** distinct free patches, one per advisory. Which one the resolver returns is a server-side ordering detail, so the suite accepts any of the three rather than pinning one — pinning would go red on an unrelated server-side reorder. -`preflight_required_patches_are_published` checks all four every run and fails +`preflight_required_patches_are_published` checks all three every run and fails first with the offending PURL named, so a withdrawn patch produces one clear failure instead of N confusing ones that look like CLI regressions. @@ -66,6 +71,13 @@ failure instead of N confusing ones that look like CLI regressions. 3. If the new patch does not inject the `// Socket Community Patch` header (Cargo crates do not), pick a marker unique to the patch and set the ecosystem's `*_MARKER` constant. +4. If the ecosystem has **no free patches left at all** (probe every plausible + candidate — for gems, every name in + [ruby-advisory-db](https://github.com/rubysec/ruby-advisory-db) is a decent + sweep), there is nothing honest to re-pin: retire the leg, move the + ecosystem into `UNPUBLISHED_ECOSYSTEMS` so `canary_unpublished_ecosystems` + reports the moment a patch reappears, and note the retirement here. That is + what happened to RubyGems in 2026-08. ## Ecosystem coverage, and the honest gaps @@ -74,22 +86,49 @@ failure instead of N confusing ones that look like CLI regressions. | npm | ✅ | ✅ many | ✅ npm, npm-shrinkwrap, pnpm, yarn classic, yarn berry, bun | | PyPI | ✅ (requirements.txt + uv.lock only) | ✅ many | ✅ requirements.txt, uv.lock | | Cargo | ✅ | ✅ 1 crate | ✅ sparse registry | -| RubyGems | ✅ | ✅ 1 gem | ⚠️ redirect asserted; install blocked by a **server defect** (below) | +| RubyGems | ✅ | ❌ **none** — last one withdrawn 2026-08 | canary only; leg retired (below), mocked capstone `e2e_redirect_gem_build.rs` | | Maven | ✅ | ❌ **none** | canary only | | NuGet | ✅ | ❌ **none** | canary only | | Composer | ✅ | ❌ **none** | canary only | | Go | ❌ [by design](../design/golang-hosted-no-go.md) | ❌ none | negative assertion | | Deno | ❌ not supported | — | negative assertion | -Maven, NuGet and Composer all *implement* hosted mode, but production publishes -**zero** free-tier patches for them, so there is nothing real to redirect to. -Rather than skipping silently, `canary_unpublished_ecosystems` probes production -every run and reports the moment that changes, so coverage can be extended -deliberately. It does not fail when patches appear — production publishing a -patch is not a socket-patch regression — but +RubyGems, Maven, NuGet and Composer all *implement* hosted mode, but production +publishes **zero** free-tier patches for them, so there is nothing real to +redirect to. Rather than skipping silently, `canary_unpublished_ecosystems` +probes production every run and reports the moment that changes, so coverage +can be extended deliberately. It does not fail when patches appear — production +publishing a patch is not a socket-patch regression — but `SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT=1` makes it fail, for use in a scheduled nag run. +### RubyGems: the retired bundler leg + +Until 2026-08 the suite carried a real bundler leg +(`gem_bundler_hosted_redirect_and_known_install_defect`): install +activestorage 7.0.2.2 from rubygems.org, assert the hosted redirect rewrote the +`Gemfile` source block and replaced the `Gemfile.lock` CHECKSUMS pin, then +reinstall from the redirected Gemfile — tolerating (and loudly reporting) the +known compact-index server defect below, and content-verifying the installed +files against the patch's published `afterHash` whenever the install +succeeded. + +Production then withdrew the `activestorage@7.0.2.2` patch. It was the last +free-tier gem patch: as of 2026-08-13 a probe of **every** gem name in +ruby-advisory-db (464 packages) returns zero free patches, and +`/patch/view/2535d43d-67ce-4944-be27-c19e113997fb` 404s, so there was no +honest replacement pin and the leg was retired rather than left red or made +vacuous. The gem hosted rewrite grammar stays covered by +`e2e_redirect_gem_build.rs` against a mock. + +**To restore it**: when the `gem` probes in `UNPUBLISHED_ECOSYSTEMS` light up, +resurrect the leg, its helpers (`gem_lock_checksum`, `installed_gem_dir`, +`gem_registry_base`, `http_probe`, `published_patch_files`), the `GEM_*` +catalog constants, the `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT` knob and CI's +`ruby/setup-ruby` step from git history (the commit that landed this +paragraph deleted all of them together), re-pin to the newly published patch, +and re-add the catalog rows above. + PyPI's poetry / pdm / pipenv locks are **not** rewritten by hosted mode (see the [matrix](../ecosystems.md#mode--ecosystem-matrix)); those flavors are vendored-mode only, so there is no hosted leg to write for them. @@ -107,10 +146,16 @@ Two supported hosted shapes are deliberately **not** covered here: ## Known issues this suite surfaced -Both were found by running against real production, and neither is a test bug. +All were found by running against real production, and none is a test bug. ### 1. `gem` — hosted mode is unusable for gems with dependencies (SERVER) +> **Status 2026-08**: the bundler leg that exercised this defect was retired +> when production withdrew its last free-tier gem patch (see above). The +> defect itself is still believed to stand server-side; nothing free-tier +> remains to probe it against. Kept for the record and for the day the leg is +> restored. + Socket's gem patch-registry serves a compact index whose `/info/` line declares **no runtime dependencies**, while the `.gem` it serves declares six. Bundler's `ensure_same_dependencies` check fails closed: @@ -134,10 +179,11 @@ curl -s "https://patch.socket.dev/patch-registry/gem///info/actives ``` **Fix belongs on the server**: the compact-index generator must emit the -gemspec's runtime dependencies. Until then the suite asserts the redirect (which -is correct) and tolerates the install failure, failing loudly if it fails for -any *other* reason. Set `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` to promote it to -a hard failure — do that as the regression guard once the server is fixed. +gemspec's runtime dependencies. While the leg existed, the suite asserted the +redirect (which was correct) and tolerated the install failure, failing loudly +if it failed for any *other* reason; `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` +promoted it to a hard failure. Both the tolerance and the knob left with the +leg. ### 2. `pnpm` — pnpm 11 rejects hosted lockfiles by default (CLI UX gap) @@ -201,19 +247,18 @@ runs only where it is explicitly asked for. | Variable | Effect | |----------|--------| | `SOCKET_PATCH_HOSTED_E2E_STRICT=1` | Turn every "toolchain missing" soft-skip into a hard failure. **CI sets this** — a required check must never report green on an unexercised leg. | -| `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` | Promote the known gem install defect to a hard failure. | -| `SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT=1` | Fail when maven/nuget/composer gain their first free published patch. | +| `SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT=1` | Fail when gem/maven/nuget/composer gain their first free published patch. | ### Toolchains `npm`, `corepack` (pnpm + yarn classic + yarn berry), `bun`, `uv`, `cargo`, -`ruby` + `bundle` (**≥ 2.6** — `bundle lock --add-checksums` emits the CHECKSUMS -section the gem rewrite pins into), `go`. +`go`. (`ruby` + `bundle` ≥ 2.6 will be needed again when the retired gem leg +is restored.) ### Network egress `patches-api.socket.dev`, `patch.socket.dev`, `registry.npmjs.org`, `pypi.org`, -`files.pythonhosted.org`, `static.crates.io`, `index.crates.io`, `rubygems.org`. +`files.pythonhosted.org`, `static.crates.io`, `index.crates.io`. ## CI: the `hosted-e2e` job