[WIP] CAS draft (adopting to CI/CD, not for review / merge) - #2073
Draft
filimonov wants to merge 4161 commits into
Draft
[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073filimonov wants to merge 4161 commits into
filimonov wants to merge 4161 commits into
Conversation
ilejn
reviewed
Jul 28, 2026
|
|
||
| ```sql | ||
| SYSTEM CONTENT ADDRESSED GC RUN [ON CLUSTER cluster_name] [disk_name] | ||
| ``` |
svb-alt
added
antalya
antalya-26.6
roadmap
strtgbb
requested changes
Jul 31, 2026
… Task 14) Persistent agent working guide absorbing INTENT.md, 02-methodology.md and durable cross-session memory items; deletion of the absorbed docs is now gated on its existence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l-overwrite design The first design added a conditional streaming overwrite. It worked, but it paid the condition's full cost -- a new seam method with a refusal path, overrides in twelve backends, a `declared_size` parameter, and a GCS guard -- for a benefit that turns out to be one saved re-upload on a lost race. The decisive evidence was already in the tree. `resurrectStaged`, the S3-native staging arm of the SAME branch, writes unconditionally, and its comment states the reasoning verbatim: an `If-Match` on the condemned token "would only save a redundant re-upload on a lost race, never prevent data loss". INV-NO-RETURN comes from the fresh incarnation tag, not from the condition; durable references address blobs by content hash and name no incarnation at all. The method's name also misleads, and the spec says so: `resurrectStaged` performs no server-side copy -- it reads the staging object, streams it through the client, and writes with default settings. The server-side copy is in `promoteStaged`. So the two arms differ in exactly one thing: where the reader comes from. Making the reader a parameter collapses them into one operation, and because an unconditional write needs no precondition, it takes the multipart path on GCS too -- the ceiling this path had stops existing rather than being documented. Order of work puts the revert first so the branch never carries two designs at once, and flags one piece of the reverted work to re-apply on its own merits: the explicit upload abort on a refused conditional write, which `putIfAbsentStream` still needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
Task 1 is the revert, and it lands green on its own so the branch never carries two designs at once. It also calls out the one piece of the reverted work worth re-applying separately: the explicit upload abort on a refused conditional write, which the streaming CREATE still needs and which would otherwise vanish with a design it did not belong to. The middle three tasks are ordered so each is independently testable: the source becomes a reader factory (behaviour-preserving), the seam takes a reader instead of a staging key (behaviour-preserving for the staging arm), and only then does the local arm switch and the materialization disappear. Task 5 pins a property rather than changing behaviour: its test passes on arrival, because it exists to stop a later consistency-minded refactor from routing the resurrect back through conditionalWriteSettings and silently restoring a ceiling that no longer applies. The implementer notes lead with INV-NO-RETURN, because the cheapest-looking shortcuts on this path -- reusing the source header, server-side copying the condemned object -- reproduce the condemned ETag and let an already-queued exact-token delete destroy the live resurrection, which no test without a GC round afterwards would catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…ending the body" This reverts commit 784308d.
This reverts commit 531adee.
…s refused Losing the condition is an ordinary outcome on the streaming create path, not an error: another writer legitimately took the slot. Left to the write buffer's destructor it logs "was neither finished nor aborted" every time -- which any test whose server writes to stderr turns into a failure -- and leaves uploaded parts billable until a lifecycle rule reaps them. Found while building the conditional-overwrite design that was since reverted; it stands on its own, because `putIfAbsentStream` loses conditions too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
Every producer already had a reader behind its callback: the production one opened ReadBufferFromFile and copied, the test ones wrote a retained string. Pushing the copy loop into each producer meant a source could only be consumed by something willing to be written INTO -- which is why the two resurrect arms could not share an implementation even though they write the same bytes. A factory rather than one buffer because this path retries: it re-uploads after the object vanishes and re-decides after a lost race, and each attempt must read from the start. The string-backed sources use ReadBufferFromOwnString, not ReadBufferFromString: the latter only BORROWS the bytes it is handed. Today's captures outlive every call, so the borrowing form would work -- and would be a dangling read the first time someone shortened a lifetime. The owning copy costs nothing here, where this helper exists for inline payloads and tests. Behaviour is unchanged; the touched suites pass 103/103. Gates: release and ASan both green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
The method never performed a server-side copy despite its name: in Native mode it opened the staging object, skipped its envelope header, and streamed it through the client into an unconditional write. The server-side copy is `promoteStaged`, the ordinary write-once create. So the two resurrect arms differed in exactly one thing -- where the reader came from -- and making that a parameter collapses them into one operation. The payload offset moves to the caller, which is what knows the staging envelope's shape. One test assertion had to move rather than be dropped. `CASS3Staging` proved "the resurrect reads the staging object, never the condemned blob key" by inspecting the source ARGUMENT; with the caller opening the reader there is no such argument. It now counts what was actually read via `getStream` and asserts the staging key was read and the condemned key was not -- an assertion about I/O rather than about a parameter the backend was merely told, which is the stronger form. `get` is deliberately not counted: the surrounding assertions use it to inspect bodies. Release gate: 2027/2027. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…ource reader The local arm was the last caller that had to materialize a blob body: putOverwrite took a String, so a body larger than memory could not be resurrected at all. It now streams through the same Backend::resurrect the staging arm uses, and drops the condition for the same reason that arm always did -- an If-Match on the condemned token saves a redundant re-upload on a lost race and prevents no data loss. What protects the resurrection is the fresh incarnation tag, which makes the new body's ETag differ from the condemned one so every already-queued exact-token delete misses. The size check moved rather than disappeared: the whole-body path compared what the source produced against source.size BEFORE writing; the streaming path HEADs the incarnation it just wrote and compares lengths, so a source that lies about its size still cannot publish a body that disagrees with the manifest entry naming it. The new open-count test pins TWO opens for the present-condemned shape and says why each bound matters: open #1 is the ordinary conditional-create attempt (streamed, refused at finalize), open #2 is the resurrect. More would mean a hidden materialization pass crept back in; fewer would mean the create attempt stopped streaming -- a protocol change, not an optimization to make silently. Release gate: 2027/2027. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
It existed because one branch could not stream. That branch streams now, so the aggregate it rationed no longer accumulates. Its guarantee was max(capacity, largest single body) and the second term was unbounded, so it never protected against the case that motivated this work -- one blob larger than memory. The two tests that drove it are deleted rather than ported: they asserted that an aggregate is not exceeded and that an overweight admission does not starve, and neither property exists once nothing is admitted. `cas_condemned_upload_memory_bytes` is deleted with no compatibility shim -- pre-release, no persisted data, and an accepted-and-ignored setting reads as still doing something. One deletion here was initially too greedy, and the full gate caught it: gtest_cas_blob_upload_pool_env.cpp looked like the admission's test scaffolding but is the global gtest listener that (a) brings the blob UPLOAD POOL up before every test -- its getter is fail-loud, so 46 tests across 10 suites threw without it -- and (b) shuts that pool down in OnTestProgramEnd, without which the binary deadlocks at exit: gtest_main joins the GlobalThreadPool whose std::threads are occupied by pool workers that only return when the pool is destroyed. The file is restored with only its admission lines removed. Release gate after the restore: 2027/2027. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…tes the cap binds; prune the ledger The regression test locks in what a consistency-minded refactor would silently break: routing the resurrect through conditionalWriteSettings would reintroduce a single-part ceiling this path no longer has. The three user-facing pages now say which writes the cap actually binds -- the conditional write-once create -- instead of implying it bounds every write on GCS. The displacement-alternatives BACKLOG item is removed as resolved: the question it asked was settled by landing the unconditional design. The GCS-rethink item is narrowed to the one path still capped, the conditional create, with the overwrite discussion marked historical. Both design docs and both plans are deleted -- the code and its comments now carry the reasoning, and a superseded spec left in the tree reads as an open question. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
An adversarial review of the increment (verdict REJECT) found two real defects, both invisible to the unit gate because it exercises InMemoryBackend: 1. The size check ran AFTER publication. Moving the source.size comparison from before the write (the old whole-body path counted into the String) to a post-write HEAD left a window where a source truncated after hashing publishes a short body as the CURRENT incarnation; the check then throws, but a racing writer that already resurrected correctly proceeds to flip the meta Clean and promote a tokened dependency without revalidation -- the committed manifest names a truncated body. The old pre-write count let nothing publish at all; the post-write HEAD could also inspect a racing writer's incarnation rather than our own. Fixed by passing `payload_size` into `Backend::resurrect`: the write counts while streaming and, on a mismatch, cancels the upload BEFORE finalize -- nothing becomes current on any backend. 2. Local content-addressed disks could no longer resurrect at all. Local object storage runs `ObjectStorageBackend::Mode::EmulatedSingleProcess`, and the new `resurrect` threw NOT_IMPLEMENTED for every non-Native mode, where the `putOverwrite` it replaced worked. A condemned blob on a local CA disk became unrepairable, and no InMemoryBackend test could notice. Fixed with an emulated implementation serialized by the same `emu_mutex` as every other emulated conditional op. Both regression tests live in the CONTRACT suite, which is parameterized over InMemoryBackend and the real ObjectStorageBackend in emulated mode -- so the mode that hid defect 2 now runs them: `ResurrectWrongSizePublishesNothing` and `ResurrectReplacesBodyAndMintsFreshToken` (the latter also re-pins INV-NO-RETURN through the queued exact-token delete missing the fresh incarnation). Also from the review, the contract prose no longer claims "never overwrites a live blob": two racing resurrections may both run and the loser overwrites the winner's FRESH incarnation -- accepted, not prevented, because payloads are content-identical and durable references name content hashes. The arm comment and the backend doc page now say what is actually guaranteed: a live incarnation of DIFFERENT content is unreachable by the content address itself. Release gate: 2032/2032. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…ERROR test rule at the gate table Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…tyle The three edits carried paragraph-length mechanism recaps into tables whose every other row is one phrase. The mechanism lives in the code comments; the tables state the contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
INV-NO-RETURN / INV-NO-LOSS / INV-NO-DANGLE are development shorthand: to a reader of the public documentation they are unexplained noise. Each occurrence is replaced by the plain statement of what the invariant guarantees, or removed where the sentence already said it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
A source that yields a different byte count than it declared is INPUT -- a staged file truncated after hashing -- not an impossible program state. LOGICAL_ERROR also aborts sanitizer builds, which the ASan gate demonstrated by dying on the guard's own regression test and hiding everything after it. CORRUPTED_DATA states what happened and stays a catchable outcome everywhere. (The change was in the tree when the previous commit's gates ran green; committing it separately was missed and caught by re-review.) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
The second review round caught a guarantee lost in the admission's deletion: the Native resurrect
streams, but the emulated one materializes the whole body, and the upload fan-out runs N resurrect
tasks in parallel -- so a local-disk INSERT resurrecting several large condemned blobs could hold the
SUM of their sizes in RAM. The deleted semaphore bounded exactly that.
Restored as the smallest equivalent: emulated resurrections are serialized by their own process-wide
mutex, so the peak is the largest single body -- the same bound the semaphore's exclusive arm gave.
A dedicated mutex rather than emu_mutex, because the drain may read through the same store. The three
prose sites that contradicted the implementation (the seam promising "never materialized", the header
still claiming NOT_IMPLEMENTED for emulated resurrect, the doc page describing the deleted semaphore)
now say what actually happens.
The proper fix is recorded as debt in BACKLOG {#emulated-resurrect-spill-to-disk}: this backend sits
ON a local disk, so the staging area for a body of any size is a scratch file, not RAM -- spilling
would remove both the materialization and the serialization. Kept as debt rather than done now
because no production deployment runs CAS over local paths.
Release gate: 2030/2030.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
… drop dead setting externs The third review round approved the increment with one prose finding: seven sites still described the pre-serialization world. Two were live code residue -- the extern declarations of the deleted `cas_condemned_upload_memory_bytes` in Server.cpp and LocalServer.cpp (unused symbols, so no linker caught them) and a DisksApp comment describing the deleted admission's derivation. Five were claims that resurrection "streams and never materializes" stated backend-independently; each now says what is true where -- streamed on remote object storage, materialized one body at a time under serialization on the local emulated mode. The review also answered the two safety questions asked of it, with the reasoning worth keeping: the lock graph is acyclic (no path holding `emu_mutex` acquires the resurrect mutex, and emulated `getStream` releases `emu_mutex` before returning), and pool saturation under serialized resurrections is head-of-line blocking, not deadlock -- the mutex owner never waits on queued fan-out work, so its completion always frees both the mutex and a pool slot. Release gate: 2030/2030. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…mmit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
It rode into the CAS branch after a CAS test tripped the class it detects, but it is a generic check in a shared upstream file -- one more line of diff noise in an already loud PR. The check itself is kept aside untracked for a standalone upstream submission. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
Every workflow carried its own splice of upstream lanes plus the CAS lanes -- six copies of the same composition, each a place to forget when the next lane is added. The composition now lives once in AltinityJobConfigs.functional_tests_jobs (a NEW list; altinity_jobs.py is ours, so no upstream file grows CAS knowledge), and each workflow references it with a one-token change. A per-file `JobConfigs.functional_tests_jobs += ...` was considered and rejected with evidence: praktika's mangle loads every workflow module in ONE process, so mutating the shared upstream list would append the CAS lanes once per loaded workflow -- later-loaded workflows would schedule duplicates, and the result would depend on directory iteration order. Equivalence proved, not assumed: the sorted job-name list of every one of the six workflows is byte-identical before and after (loaded standalone, the way praktika loads them); dups=0 in the all-modules-one-process mode as well. pull_request.py's local FUNCTIONAL_TESTS_JOBS list dissolved into the shared one, shrinking its diff against upstream from five edit sites to token swaps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…age-b-remaining) Stage B is complete; the per-plan workspace with its briefs, reports, and evidence bundles is no longer needed on the branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…cessor test Upstream commit bfe5988 ("Iceberg type 'time' support") changed the getSimpleType mapping of the Iceberg `time` type from Int64 to Time64(6), but the test kept the old expectation. The mismatch is inherited from altinity/antalya-26.6 and fails `Unit tests (asan_ubsan)` on every PR against that base. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2159&sha=056488b47a0b3f32ea2ebf83d508b697ff4c0c17&name_0=PR&name_1=Unit%20tests%20%28asan_ubsan%29 PR: #2159 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…on` release at query finish `BlockIO::onFinish` released the whole set of workload resources — including the `MemoryReservation` — before the query pipeline was finalized, while pipeline threads still hold raw pointers to it through `WorkloadResources` in `PipelineExecutor`. Split the release: the query slot is released early (safe, pipeline threads never touch it), the memory reservation only after the pipeline has been finalized and its threads joined. `BlockIO::reset` resets the pipeline before releasing the resources for the same reason. The memory reservation feature (ClickHouse#82414) was backported to 26.6 in `2098704e6d1` on 2026-06-27, but this fix never followed: its cherry-pick ClickHouse#108562 was closed automatically on 2026-07-06 with unresolved conflicts. As a result 26.6 reports the race in `Stress test (amd_tsan)`, for example https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=112727&sha=6120a6fb27efe0d740c8f075aee499215d2f23e1&name_0=BackportPR&name_1=Stress%20test%20%28amd_tsan%29 (cherry picked from commit 9e801ca)
Member
Author
|
TSan: data race + Server died (report) Race on |
…ection stays cheaper The final check asserts the optimizer selects the projection under force_optimize_projection = 1. With randomized MergeTree settings (index_granularity_bytes = 206048, index_granularity = 27075, merge_max_block_size = 5006, enable_block_offset_column = 1 — minimized by the CI randomized-settings diagnosis) the projection needed 202 marks vs 201 for the base table, the optimizer picked the base, and the check threw PROJECTION_NOT_USED. The test is about the multi-block projection rebuild on a content-addressed disk, not the cost model, so pin the default granularity in the CREATE. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=9b5cbc314bc4eeb3d1b3d3017a34cd3816f59e60&name_0=PR&name_1=Stateless+tests+%28amd_debug%2C+parallel%29&name_2=Tests PR: #2073 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
The "proven absent" presence probe and the cold-reader admission both required two whole-catalog reads to be byte-identical (token + content) before trusting an observation. The catalog is pool-global, so under a parallel workload the condition almost never holds and both paths starve: in CI, `01069_database_memory` failed 193 of 194 retries with "catalog changed while probing table-root cleanup completeness" on its `deduplication_logs` namespace. Each catalog read is one token-CAS'd full value, so this namespace's row is what the proof actually needs: - presence probe: row absent in BOTH reads => proven absent (linearized at the second read); a row that appears in between answers present; - cold reader: revalidate THIS row by value; unrelated churn admits, an own-row change still forces a fresh observation. The second cut now runs the ambiguity validation explicitly -- an aliasing incarnation admitted between the reads left the target row byte-identical and was only refused by the whole-catalog comparison it replaces. Two tests pinning whole-catalog stillness updated to the per-row contract; regression tests added for unrelated-churn starvation and the aliasing incarnation. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=95ba719fe763e3d87a3ef4555b9e1b310c768d0c&name_0=PR&name_1=Stateless+tests+%28amd_binary%2C+cas+storage%2C+parallel%29 PR: #2073 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
content addressable storage - draft PR
Documentation entry for user-facing changes
TBD.
Exclude tests:
Regression jobs to run: