Add StoreDigest plugin for content-based deduplication - #152
Merged
Conversation
StorePermalink asks whether a link has been seen. A page that reissues one article under a new URL, and a feed built from an HTML index by CustomFeedWeb, pose the other question: whether this content has been seen, whatever it was published under. StoreDigest answers that one. The fields the Recipe names -- title and description by default -- are read as UTF-8, normalized to NFC with their whitespace runs collapsed, joined with their field names into one canonical string and hashed with SHA-256. The digest is recorded in SQLite and nothing else is: what an item said is StoreFullText's business. Exact matching of the selected content, and only that. There is no fuzzy comparison, no similarity threshold and no algorithm setting. A Recipe that names fields gets those fields, with no fallback to another when the one it asked for is empty, so that what the Recipe says two identical items are is what the plugin obeys. An item whose selected fields are all empty is passed on with a warning rather than stored under the digest of the empty string, which would make every item with no description the same item. A database failure ends the run instead of being rescued, which is a deliberate difference from StoreFullText: an item passed on after its digest failed to store would be published again next run. The digest column carries a unique index, so two overlapping runs cannot both store one digest. The Database mixin supplies the connection, the table and the directory; the digest loop is this plugin's own, because for_each_new_feed is written around a link. Nothing in StorePermalink or StoreFullText changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRMBU24jWQcDTH5GH5NJMy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a new
StoreDigestplugin that deduplicates feed items based on content digest rather than URL, complementing the existingStorePermalinkplugin's URL-based deduplication.Key Changes
New plugin:
StoreDigest(plugins/store/digest.rb)Comprehensive test suite (
spec/plugins/store/digest_spec.rb)Documentation updates (
doc/PLUGINS.md)StoreDigestsection explaining use cases, configuration, and behaviorStorePermalink(URL identity),StoreFullText(URL+title+body), andStoreDigest(content identity)Supporting updates
database.rb,DEPLOYMENT.md,BASIC_DESIGN.md,Gemfile,VERSIONS)Notable Implementation Details
\0) as separator in canonical digest string to prevent field/value collisionsStoreFullTextfor complete deduplication with content preservationhttps://claude.ai/code/session_01PRMBU24jWQcDTH5GH5NJMy