From a6dcc1108f0c00ec942fb47375bdafc0dda99125 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 09:52:54 +0000 Subject: [PATCH] Add the plugin StoreDigest StorePermalink asks whether a link has been seen. A page that reissues one article under a new URL, and a feed built from an HTML index by CustomFeedWeb, pose the other question: whether this content has been seen, whatever it was published under. StoreDigest answers that one. The fields the Recipe names -- title and description by default -- are read as UTF-8, normalized to NFC with their whitespace runs collapsed, joined with their field names into one canonical string and hashed with SHA-256. The digest is recorded in SQLite and nothing else is: what an item said is StoreFullText's business. Exact matching of the selected content, and only that. There is no fuzzy comparison, no similarity threshold and no algorithm setting. A Recipe that names fields gets those fields, with no fallback to another when the one it asked for is empty, so that what the Recipe says two identical items are is what the plugin obeys. An item whose selected fields are all empty is passed on with a warning rather than stored under the digest of the empty string, which would make every item with no description the same item. A database failure ends the run instead of being rescued, which is a deliberate difference from StoreFullText: an item passed on after its digest failed to store would be published again next run. The digest column carries a unique index, so two overlapping runs cannot both store one digest. The Database mixin supplies the connection, the table and the directory; the digest loop is this plugin's own, because for_each_new_feed is written around a link. Nothing in StorePermalink or StoreFullText changes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01PRMBU24jWQcDTH5GH5NJMy --- Gemfile | 2 +- README.md | 6 +- doc/BASIC_DESIGN.md | 4 +- doc/DEPLOYMENT.md | 5 +- doc/PLUGINS.md | 135 +++++++++- doc/VERSIONS | 1 + plugins/store/database.rb | 6 +- plugins/store/digest.rb | 212 +++++++++++++++ spec/plugins/store/digest_spec.rb | 424 ++++++++++++++++++++++++++++++ 9 files changed, 778 insertions(+), 17 deletions(-) create mode 100644 plugins/store/digest.rb create mode 100644 spec/plugins/store/digest_spec.rb diff --git a/Gemfile b/Gemfile index 5df134c..517cd77 100644 --- a/Gemfile +++ b/Gemfile @@ -35,7 +35,7 @@ gemspec # The table of which plugin needs which gem, and which of those plugins still # work, is in doc/DEPLOYMENT.md and doc/PLUGINS.md section 6. -# StorePermalink and StoreFullText, through plugins/store/database.rb. +# StorePermalink, StoreFullText and StoreDigest, through plugins/store/database.rb. group :plugins, :store, optional: true do gem 'activerecord', '>= 7.1', '< 9.0' gem 'sqlite3', '>= 1.7', '< 3.0' diff --git a/README.md b/README.md index 419bb2d..0851ab5 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ a plugin set every part of which still has somewhere to talk to. See - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to wire a pipeline together. -- **35 plugins** across seven categories: subscribe, custom feed, filter, +- **36 plugins** across seven categories: subscribe, custom feed, filter, store, provide, notify, publish — and every one of them has a current use. - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain Markdown document, to a file or to standard output, with no service and no @@ -369,13 +369,13 @@ like a shipped plugin replaces it. ### Which plugins still work -35 plugins ship with the gem. Every one is classified in +36 plugins ship with the gem. Every one is classified in [`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason for its status: | Status | Count | Meaning | | --- | --- | --- | -| **Supported** | 24 | Works on the supported Rubies with current dependencies | +| **Supported** | 25 | Works on the supported Rubies with current dependencies | | **Supported (external)** | 10 | Works, but needs something you provide: a service, a command, a credential, a data file | | **Needs rework** | 1 | The service exists; this plugin speaks a replaced interface | diff --git a/doc/BASIC_DESIGN.md b/doc/BASIC_DESIGN.md index 75a14eb..7cf1cbd 100644 --- a/doc/BASIC_DESIGN.md +++ b/doc/BASIC_DESIGN.md @@ -325,7 +325,9 @@ plugin implementation and the framework does not use it: class's `column_definition` when it is absent, and provides `for_each_new_feed`, which yields only items whose key is not already stored. `StorePermalink` and `StoreFullText` are this mixin plus a model and a column - list. + list. `StoreDigest` takes the database part of it and decides for itself what + has been seen, because it is identified by a digest of an item's content + rather than by the link `for_each_new_feed` reads. ### 4.10 `db/`, `config/`, `assets/` Fallbacks inside the installation, used when the corresponding part of the user diff --git a/doc/DEPLOYMENT.md b/doc/DEPLOYMENT.md index e04c800..7b9931d 100644 --- a/doc/DEPLOYMENT.md +++ b/doc/DEPLOYMENT.md @@ -455,7 +455,7 @@ bundle. `plugins` is every group in the first block at once. | Plugin | Needs | Installed gem | Checkout group | Status | | --- | --- | --- | --- | --- | -| `StorePermalink`, `StoreFullText` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | Supported | +| `StorePermalink`, `StoreFullText`, `StoreDigest` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | Supported | | `FilterImageSource`, `FilterDescriptionLink`, `SubscriptionLink`, `SubscriptionTumblr`, `CustomFeedWeb` | `nokogiri` | `gem install nokogiri` | `html` | Supported (`SubscriptionTumblr` external) | | `PublishMarkdown` | `nokogiri`, for HTML bodies only | `gem install nokogiri` | `html` | Supported; runs without it | | `FilterSanitize` | `sanitize` | `gem install sanitize` | `sanitize` | Supported | @@ -499,8 +499,7 @@ missing, what needs it and how to get it, and the command exits `1`. ```text automatic: The `activerecord` gem is not installed. It is needed by the store -plugins StorePermalink and StoreFullText. Install it with `gem install -activerecord`, ... +plugins. Install it with `gem install activerecord`, ... ``` ## Your own plugins diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index 73d41e5..2171e23 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -1044,12 +1044,17 @@ been through another filter first is no longer a `NoMethodError`. Persist, and drop what has already been seen. A store plugin is what makes a Recipe safe to run repeatedly. -`StorePermalink` and `StoreFullText` keep their records in SQLite through -ActiveRecord. Both gems are these plugins' own optional dependencies rather than -the framework's: `gem install activerecord sqlite3`, or the `store` group in a -checkout. A Recipe that stores nothing needs neither. +`StorePermalink`, `StoreFullText` and `StoreDigest` keep their records in SQLite +through ActiveRecord. Both gems are these plugins' own optional dependencies +rather than the framework's: `gem install activerecord sqlite3`, or the `store` +group in a checkout. A Recipe that stores nothing needs neither. See [`DEPLOYMENT.md`](DEPLOYMENT.md). +They answer different questions, and a Recipe may ask more than one of them: +`StorePermalink` whether this **link** has been seen, `StoreFullText` whether +this link or title has been stored with its body, `StoreDigest` whether this +**content** has been seen, whatever it was published under. + #### StorePermalink — **Supported** `store/permalink.rb`. Records each item's link in SQLite and passes on only the @@ -1074,6 +1079,124 @@ republished article with a new URL is not stored twice. Pair with | --- | --- | --- | | `db` | string | Database file name, under `~/.automatic/db`. Required. | +#### StoreDigest — **Supported** + +`store/digest.rb`. Takes the SHA-256 digest of the item fields the Recipe names, +records it in SQLite, and passes on only the items whose digest was not recorded +already. Content identity, where `StorePermalink` is URL identity: a page that +reissues one article under a new URL is one item here, and one URL whose content +changed is a new item — the opposite of what `StorePermalink` decides in both +cases. Pair it with `CustomFeedWeb`, whose items are whatever an index page +currently lists. + +| Key | Type | Meaning | +| --- | --- | --- | +| `db` | string | Database file name, under `~/.automatic/db`. Required. | +| `fields` | list | The fields the digest is taken over, in the order written. Default: `title`, `description`. | + +```yaml +plugins: + - module: CustomFeedWeb + config: + sites: + - url: https://example.com/news/ + + - module: StoreDigest + config: + db: web-digest.db + + - module: PublishMarkdown + config: + file: ~/.automatic/markdown/web-watch.md + mode: append +``` + +The first run passes on everything the page listed and records a digest for +each. The second passes on nothing, because the page still lists the same +articles. A run in which one article has been added passes on that one. + +**`fields`** names any of `title`, `link`, `description`, `author`, `comments`, +`source` and `content_encoded`. `date` is not among them — an item republished +unchanged carries a new date often enough to defeat the purpose — and neither is +`enclosure`, which is a structure rather than a value. + +- The order is part of the fingerprint. `[title, description]` and + `[description, title]` are two different specifications and produce different + digests; nothing is sorted behind the Recipe's back. +- **The fields named are the fields used.** A Recipe that asks for + `content_encoded` gets `content_encoded`, and an item whose body is empty is + not quietly judged on its title instead. What the Recipe says two identical + items are is what this plugin obeys. +- Anything but an absent `fields` is taken as written: an empty list, a name + that is not a field, a name given twice and a value that is not a list are + each refused with an `ArgumentError` before the database is opened, rather + than corrected into something the Recipe did not ask for. +- `db` is required, and an empty name is refused the same way. + +**What "the same content" means here.** Each value is read as UTF-8, with +invalid and undefined characters replaced, normalized to Unicode NFC, its runs +of whitespace collapsed to one space and its ends trimmed. The values are joined +with their field names into one canonical string — `title`, NUL, the title, NUL, +`description`, NUL, the description — and that string is hashed with SHA-256. +The algorithm is fixed; there is no setting for it and no column recording it. + +Two items are therefore the same item when their selected fields are **exactly** +equal after that normalization, and not otherwise. A difference of case, of +punctuation or of markup is a difference of content. This is not similarity +matching: there is no fuzzy comparison, no edit distance, no embedding and no +semantic judgement anywhere in it, and two articles that report one event in +different words are two items. + +**An item with nothing to hash is passed on, not stored.** Where every field the +Recipe named is empty after normalization — `fields: [description]` on an item +that has no description — there is nothing to identify the item by. Hashing the +empty string would make every such item the same item and silence all but the +first of them for good, so instead the plugin logs a warning naming the item's +link and passes it on unjudged. An item is never lost for having too little +content. A field that is empty while another is not takes part in the digest as +an empty value, so an item with a title and no description differs from the same +item with both. + +**A database failure ends the run.** A failed read or a failed write is not +rescued here, which is deliberate and is a difference from `StoreFullText`: an +item passed on after its digest failed to store would be published again on the +next run, and de-duplication that quietly stops de-duplicating is worse than a +run that stops. The digest column carries a unique index, so two runs of one +Recipe overlapping cannot both store one digest; the second write is rejected +and its item is treated as seen. + +The digest is all that is stored — no title, no body, no URL. Recording what an +item said is `StoreFullText`'s work, and pairing the two is how a Recipe gets +both. + +`StorePermalink` and `StoreDigest` may stand in one Recipe, each with its own +database, and the pair is worth having: the first drops what has been seen at +that URL, the second drops what has been seen under any URL. + +```yaml +plugins: + - module: CustomFeedWeb + config: + sites: + - url: https://example.com/news/ + + - module: FilterFullFeed + + - module: StoreDigest + config: + db: fulltext-digest.db + fields: + - content_encoded + + - module: PublishMarkdown + config: + file: ~/.automatic/markdown/web-watch.md + mode: append +``` + +With the body fetched first, the digest is taken over the article itself, so a +headline edited between runs no longer republishes the article. + #### StoreFile — **Supported** `store/file.rb`. Downloads what each link points at and rewrites the link to a @@ -1434,11 +1557,11 @@ is a claim that the plugin works. | Status | Count | Plugins | | --- | --- | --- | -| Supported | 24 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `StorePermalink`, `StoreFullText`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` | +| Supported | 25 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `StorePermalink`, `StoreFullText`, `StoreDigest`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` | | Supported (external) | 10 | `SubscriptionTumblr`, `CustomFeedSVNLog`, `FilterFullFeed`, `ProvideFluentd`, `NotifyIkachan`, `PublishEject`, `PublishMemcached`, `PublishFluentd`, `PublishInstapaper`, `PublishAmazonS3` | | Needs rework | 1 | `PublishHatenaBookmark` | -Thirty-five plugins. Every one of them either runs, or names the one thing it +Thirty-six plugins. Every one of them either runs, or names the one thing it needs from the operator; the single exception says what is wrong with it and what fixing it would take. diff --git a/doc/VERSIONS b/doc/VERSIONS index c1fa7fe..91d439d 100644 --- a/doc/VERSIONS +++ b/doc/VERSIONS @@ -12,6 +12,7 @@ v26.08 (Release Date: TBD) - Modernize the bundled plugins for current Ruby, libraries and services, and remove the integrations that are no longer viable. - Remove the plugins for Twitter, Pocket, HipChat, Google Calendar, livedoor Weather, So-net G-Guide and Chan-Toru, and Google News link rewriting; their services or APIs no longer exist, and a Recipe naming one now fails at load. - Add the plugin CustomFeedWeb, which builds one feed per HTML index page from the article links it lists, selected with CSS selectors and filtered by host, pattern and count. +- Add the plugin StoreDigest, which drops items by content rather than by URL, on a SHA-256 digest of the item fields a Recipe names. - Rebuild the test and CI strategy for current RSpec and Ruby, with deterministic isolation from user data and external services. - Add Markdown as the primary service-independent publication format, with a documented and tested first-run workflow. - Rebuild the maintained documentation around current usage, architecture, policy, plugins and deployment, and remove superseded historical documents. diff --git a/plugins/store/database.rb b/plugins/store/database.rb index 8157171..cd01b52 100644 --- a/plugins/store/database.rb +++ b/plugins/store/database.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Feb 27, 2012 -# Updated:: Aug 15, 2026 +# Updated:: Aug 17, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. # # The SQLite storage the store plugins share. ActiveRecord and sqlite3 are the @@ -14,9 +14,9 @@ Automatic.require_optional('active_record', gem_name: 'activerecord', - needed_by: 'the store plugins StorePermalink and StoreFullText') + needed_by: 'the store plugins') Automatic.require_optional('sqlite3', - needed_by: 'the store plugins StorePermalink and StoreFullText') + needed_by: 'the store plugins') module Automatic::Plugin module Database diff --git a/plugins/store/digest.rb b/plugins/store/digest.rb new file mode 100644 index 0000000..72e10f2 --- /dev/null +++ b/plugins/store/digest.rb @@ -0,0 +1,212 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Store::Digest +# Description:: Drop items whose content has been seen before, by SHA-256 digest. +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 17, 2026 +# Updated:: Aug 17, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. +# +# Content identity, where StorePermalink is URL identity. The fields the Recipe +# names are normalized, joined into one canonical string and hashed, and the +# digest is what the database holds: an item whose digest is already there has +# been seen, whatever its link says, and an item whose link has been seen before +# under different content has not. +# +# It is exact matching, of the content the Recipe selected. Whitespace runs and +# Unicode composition differences are absorbed; nothing else is. Two articles +# that say the same thing in different words are two articles here, and making +# them one is not this plugin's work -- see doc/PLUGINS.md section 6.4. + +require 'digest' +require_relative 'database' + +module Automatic::Plugin + # Named for what it holds rather than as `Digest`, which is Ruby's own + # hashing module and is what computes the value stored here. + class DigestRecord < ActiveRecord::Base + end + + class StoreDigest + include Automatic::Plugin::Database + + # The item fields a digest may be taken over. `date` is not one: an item + # republished unchanged carries a new date often enough that including it + # would defeat the purpose. `enclosure` is not one either, being a + # structure rather than a value. + FIELDS = %w[title link description author comments source content_encoded].freeze + + # What an item is, absent a Recipe saying otherwise: what it says, not + # where it is. A feed that reissues an article under a new URL is the case + # this plugin exists for. + DEFAULT_FIELDS = %w[title description].freeze + + # The field name is part of what is hashed, and this separates it from its + # value and one field from the next. A byte no title, body or URL contains, + # so that no arrangement of two fields collides with another. + SEPARATOR = "\0" + + def initialize(config, pipeline = []) + @config = config || {} + @pipeline = pipeline + end + + def column_definition + { digest: :string, created_at: :string } + end + + def model_class + Automatic::Plugin::DigestRecord + end + + # Records the digest of each item's selected fields and passes on only the + # digests not already recorded. + # + # Nothing is rescued around the database. A store plugin that failed to + # write and passed the item on anyway would publish it again on the next + # run, which is the one thing this plugin is for. + def run + validate_settings + prepare_database + + @pipeline.each_with_object([]) do |feeds, returned| + next if feeds.nil? + + new_items = feeds.items.select { |item| new_item?(item) } + returned << Automatic::FeedMaker.create_pipeline(new_items) unless new_items.empty? + end + end + + private + + # The table the mixin builds, plus the constraint. Two runs of one Recipe + # overlapping -- a `cron` entry that takes longer than its interval -- would + # otherwise both read a digest as absent and both store it. The unique + # index makes the second write fail instead, which #store below reads as + # what it is. + def create_table + super + ActiveRecord::Base.connection.add_index(model_class.table_name, :digest, unique: true) + end + + # Everything the Recipe has to get right, checked before the database file + # is opened. A Recipe this plugin cannot carry out is the operator's + # mistake and will be the same mistake next hour. + def validate_settings + raise ArgumentError, 'StoreDigest needs a db file name' if @config['db'].to_s.strip.empty? + + fields + end + + # The fields, in the order the Recipe wrote them. That order is part of the + # fingerprint and is not sorted here: a Recipe that changes it has said + # something different, and its digests are different digests. + # + # Only an absent `fields` takes the default. Every other way of getting it + # wrong is refused rather than corrected, because a Recipe's own statement + # of what makes two items the same is what this plugin has to obey. + def fields + @fields ||= validated_fields + end + + def validated_fields + given = @config['fields'] + return DEFAULT_FIELDS if given.nil? + + unless given.is_a?(Array) + raise ArgumentError, "StoreDigest takes a list of fields, not #{given.inspect}" + end + + names = given.map(&:to_s) + raise ArgumentError, 'StoreDigest was given an empty fields list' if names.empty? + + unknown = names - FIELDS + unless unknown.empty? + raise ArgumentError, + "StoreDigest cannot take a digest over #{unknown.join(', ')}; " \ + "the fields are #{FIELDS.join(', ')}" + end + + duplicated = names.tally.select { |_name, count| count > 1 }.keys + unless duplicated.empty? + raise ArgumentError, "StoreDigest was given #{duplicated.join(', ')} twice" + end + + names + end + + # Whether the item goes downstream. An item whose digest was stored here is + # new, one whose digest was already stored is not, and one there is nothing + # to hash goes on unjudged. + def new_item?(item) + digest = digest_for(item) + + if digest.nil? + Automatic::Log.puts('warn', + "StoreDigest: no digestable content for #{item.link}; " \ + 'passing item unchanged') + return true + end + + return false if stored_digest?(digest) + + store(digest, item) + end + + def stored_digest?(digest) + model_class.exists?(digest: digest) + end + + # True when this run is the one that stored the digest. The read above + # answers this on its own for a single run; the rescue is for the run + # overlapping another, where the row appeared between the two statements. + def store(digest, item) + model_class.create!(digest: digest, created_at: Time.now.strftime('%Y/%m/%d %X')) + Automatic::Log.puts('info', "Saving Digest: #{digest} (#{item.link})") + true + rescue ActiveRecord::RecordNotUnique + false + end + + # The digest of the item's selected fields, or nil where the Recipe + # selected nothing the item has. + # + # An item with nothing to hash is not stored under the digest of the empty + # string: every item with no description would then be the same item as + # every other, and the first of them would silence the rest for good. + def digest_for(item) + values = fields.map { |field| [field, normalize(value_of(item, field))] } + return nil if values.all? { |_field, value| value.empty? } + + canonical = values.map { |field, value| "#{field}#{SEPARATOR}#{value}" }.join(SEPARATOR) + ::Digest::SHA256.hexdigest(canonical) + end + + # A field an item does not carry reads as absent rather than raising. + # `content_encoded` is the case that matters: an item built by a filter has + # it, one straight from an RSS 2.0 feed may not. + def value_of(item, field) + item.respond_to?(field) ? item.public_send(field) : nil + end + + # What two spellings of one string have to survive to hash alike: the + # encoding, the composition of accented characters, and how much whitespace + # a feed happens to have put between words. Nothing beyond that -- case, + # punctuation and markup are content here, and an item that differs in them + # is a different item. + # + # `scrub` is not redundant after `encode`: a conversion whose source and + # destination encodings are the same is skipped, invalid bytes and all, and + # `unicode_normalize` raises on what is left. + def normalize(value) + value.to_s. + encode('UTF-8', invalid: :replace, undef: :replace). + scrub. + unicode_normalize(:nfc). + gsub(/\s+/, ' '). + strip + end + end +end diff --git a/spec/plugins/store/digest_spec.rb b/spec/plugins/store/digest_spec.rb new file mode 100644 index 0000000..8f2eed4 --- /dev/null +++ b/spec/plugins/store/digest_spec.rb @@ -0,0 +1,424 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Store::Digest +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 17, 2026 +# Updated:: Aug 17, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') + +# The store plugins keep their records in SQLite through ActiveRecord. Both +# gems are the store plugins' own, declared in the Gemfile's optional :plugins +# and :store groups, so this spec runs only where the operator has installed +# them. See doc/POLICY.md section 5. +return unless AutomaticSpec.optional_dependency?('activerecord') && + AutomaticSpec.optional_dependency?('sqlite3') + +require 'store/digest' +require 'digest' +require 'pathname' + +# Items with the fields these examples are about. The pipeline generator in +# spec_helper builds an item from a positional list that has no +# content_encoded, and a digest taken over a body needs one. +module DigestSpec + module_function + + def feed(*items) + channel = RSS::Rss::Channel.new + items.each { |attributes| channel.items << item(attributes) } + rss = RSS::Rss.new([]) + rss.instance_variable_set(:@channel, channel) + rss + end + + def item(attributes) + item = RSS::Rss::Channel::Item.new + item.link = attributes.fetch(:link, 'https://example.com/news/1') + item.title = attributes[:title] unless attributes[:title].nil? + item.instance_variable_set(:@description, attributes.fetch(:description, '').to_s) + item.author = attributes[:author] unless attributes[:author].nil? + item.content_encoded = attributes[:content_encoded] unless attributes[:content_encoded].nil? + item + end + + # The canonical representation the plugin hashes, written out here rather + # than taken from the plugin, so that an example asserts the format instead + # of agreeing with whatever the plugin currently builds. + def digest(*pairs) + ::Digest::SHA256.hexdigest(pairs.map { |field, value| "#{field}\0#{value}" }.join("\0")) + end +end + +describe Automatic::Plugin::StoreDigest do + let(:db) { 'test_digest.db' } + let(:record) { Automatic::Plugin::DigestRecord } + + # A run of a Recipe: the plugin is built for the pipeline it is handed, and + # the next run is the next instance, as the framework builds it. + def run(config, *feeds) + Automatic::Plugin::StoreDigest.new({ 'db' => db }.merge(config), feeds).run + end + + def reset_database + path = Pathname(AutomaticSpec.db_dir).cleanpath + db + path.delete if path.exist? + end + + # The file is removed and the table rebuilt, so that an example starts having + # seen nothing and can ask the model for a count. + before do + reset_database + Automatic::Plugin::StoreDigest.new('db' => db).run + end + + describe 'what it passes on' do + it 'passes on an item it has not seen' do + returned = run({}, DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned.should have(1).feed + returned.first.items.should have(1).item + returned.first.items.first.title.should eq 'Ruby 4.0 released' + end + + it 'stores the digest of an item it has not seen' do + lambda { + run({}, DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + }.should change(record, :count).by(1) + + record.first.digest.should eq DigestSpec.digest( + ['title', 'Ruby 4.0 released'], + ['description', 'Ruby 4.0 is now available.'] + ) + end + + it 'drops the same item on the next run' do + feed = lambda { + DigestSpec.feed(title: 'Ruby 4.0 released', description: 'Ruby 4.0 is now available.') + } + + run({}, feed.call).should have(1).feed + lambda { + run({}, feed.call).should have(0).feed + }.should change(record, :count).by(0) + end + + it 'passes on the first of two items of one content in one run' do + returned = run({}, DigestSpec.feed( + { link: 'https://example.com/a', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.' }, + { link: 'https://example.com/b', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.' } + )) + + returned.first.items.should have(1).item + record.count.should eq 1 + end + + it 'passes on the first of two feeds of one content in one run' do + returned = run({}, + DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.'), + DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned.should have(1).feed + returned.first.items.should have(1).item + record.count.should eq 1 + end + + it 'takes two links to one content as one item' do + run({}, DigestSpec.feed(link: 'https://example.com/a', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned = run({}, DigestSpec.feed(link: 'https://example.com/b', + title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + returned.should have(0).feed + end + + # The difference from StorePermalink: the link is not what identifies an + # item here, so one URL whose content changed is a new item. + it 'takes one link with new content as a new item' do + run({}, DigestSpec.feed(link: 'https://example.com/a', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned = run({}, DigestSpec.feed(link: 'https://example.com/a', + title: 'Ruby 4.0.1 released', + description: 'Ruby 4.0.1 is now available.')) + returned.should have(1).feed + record.count.should eq 2 + end + end + + describe 'fields' do + it 'takes the digest over title and description by default' do + run({}, DigestSpec.feed(title: 'A title', description: 'A description', + author: 'An author')) + + record.first.digest.should eq DigestSpec.digest(['title', 'A title'], + ['description', 'A description']) + end + + it 'takes the digest over title alone when asked to' do + config = { 'fields' => ['title'] } + run(config, DigestSpec.feed(title: 'A title', description: 'One description')) + + returned = run(config, DigestSpec.feed(title: 'A title', + description: 'Another description')) + returned.should have(0).feed + record.first.digest.should eq DigestSpec.digest(['title', 'A title']) + end + + it 'takes the digest over description alone when asked to' do + config = { 'fields' => ['description'] } + run(config, DigestSpec.feed(title: 'One title', description: 'A description')) + + returned = run(config, DigestSpec.feed(title: 'Another title', + description: 'A description')) + returned.should have(0).feed + record.first.digest.should eq DigestSpec.digest(['description', 'A description']) + end + + it 'takes the digest over the link when asked to' do + config = { 'fields' => ['link'] } + run(config, DigestSpec.feed(link: 'https://example.com/a', title: 'One title')) + + run(config, DigestSpec.feed(link: 'https://example.com/a', + title: 'Another title')).should have(0).feed + run(config, DigestSpec.feed(link: 'https://example.com/b', + title: 'One title')).should have(1).feed + end + + it 'takes the digest over content_encoded when asked to' do + config = { 'fields' => ['content_encoded'] } + run(config, DigestSpec.feed(title: 'One title', content_encoded: '

A body.

')) + + run(config, DigestSpec.feed(title: 'Another title', + content_encoded: '

A body.

')).should have(0).feed + run(config, DigestSpec.feed(title: 'One title', + content_encoded: '

Another body.

')).should have(1).feed + end + + it 'names every field it was given in the canonical representation' do + run({ 'fields' => %w[title link description] }, + DigestSpec.feed(link: 'https://example.com/a', title: 'A title', + description: 'A description')) + + record.first.digest.should eq DigestSpec.digest(['title', 'A title'], + ['link', 'https://example.com/a'], + ['description', 'A description']) + end + + it 'takes the fields in the order the Recipe wrote them' do + item = { title: 'A title', description: 'A description' } + + run({ 'fields' => %w[title description] }, DigestSpec.feed(item)) + forward = record.first.digest + + reset_database + run({ 'fields' => %w[description title] }, DigestSpec.feed(item)) + record.first.digest.should_not eq forward + end + end + + # Two items, one run after the other: one record where the plugin read them + # as one content, two where it read them as two. + describe 'normalization' do + def records_for(first, second) + run({}, DigestSpec.feed(first)) + run({}, DigestSpec.feed(second)) + record.count + end + + it 'reads a run of spaces as one space' do + records_for({ title: 'Ruby 4.0 released', description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + it 'reads a newline as a space' do + records_for({ title: "Ruby 4.0\nreleased", description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + it 'reads a tab as a space' do + records_for({ title: "Ruby 4.0\treleased", description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + it 'ignores whitespace around a value' do + records_for({ title: ' Ruby 4.0 released ', description: "Out now.\n" }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + # The same word composed and decomposed: U+00E9, and e followed by the + # combining acute accent U+0301. + it 'reads the two Unicode compositions of one string as one string' do + records_for({ title: "Café opens", description: 'Out now.' }, + { title: "Café opens", description: 'Out now.' }).should eq 1 + end + + it 'reads a difference in case as different content' do + records_for({ title: 'Ruby 4.0 Released', description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 2 + end + + it 'reads a difference in punctuation as different content' do + records_for({ title: 'Ruby 4.0 released!', description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 2 + end + end + + describe 'an item with nothing to hash' do + let(:config) { { 'fields' => ['description'] } } + + it 'stores no digest when every field it was given is empty' do + lambda { + run(config, DigestSpec.feed(title: 'A title', description: ' ')) + }.should change(record, :count).by(0) + end + + it 'passes the item on rather than losing it' do + returned = run(config, DigestSpec.feed(title: 'A title', description: '')) + + returned.should have(1).feed + returned.first.items.should have(1).item + end + + it 'takes a digest while any field it was given has content' do + lambda { + run({}, DigestSpec.feed(title: 'Ruby 4.0 released', description: '')) + }.should change(record, :count).by(1) + + record.first.digest.should eq DigestSpec.digest(['title', 'Ruby 4.0 released'], + ['description', '']) + end + end + + describe 'a Recipe it cannot carry out' do + def error_for(config) + lambda { + run(config, DigestSpec.feed(title: 'A title', description: 'A description')) + } + end + + it 'refuses an empty fields list' do + error_for('fields' => []).should raise_error(ArgumentError) + end + + it 'refuses a field it has no value for' do + error_for('fields' => %w[title foobar]).should raise_error(ArgumentError, /foobar/) + end + + it 'refuses fields that are not a list' do + error_for('fields' => 'title').should raise_error(ArgumentError) + end + + it 'refuses a field named twice' do + error_for('fields' => %w[title title]).should raise_error(ArgumentError, /title/) + end + + it 'refuses a Recipe with no db' do + lambda { + Automatic::Plugin::StoreDigest.new({}, []).run + }.should raise_error(ArgumentError) + end + + it 'refuses a db that is an empty name' do + lambda { + Automatic::Plugin::StoreDigest.new({ 'db' => '' }, []).run + }.should raise_error(ArgumentError) + end + end + + describe 'the pipeline it returns' do + it 'skips a nil feed' do + returned = run({}, nil, DigestSpec.feed(title: 'A title', description: 'A description')) + + returned.should have(1).feed + returned.first.items.should have(1).item + end + + it 'returns no feed where every item of one has been seen' do + feed = lambda { DigestSpec.feed(title: 'A title', description: 'A description') } + + run({}, feed.call).should have(1).feed + run({}, feed.call).should have(0).feed + end + + it 'returns a feed of the new items where a feed holds both' do + run({}, DigestSpec.feed(title: 'A title', description: 'A description')) + + returned = run({}, DigestSpec.feed( + { link: 'https://example.com/a', title: 'A title', description: 'A description' }, + { link: 'https://example.com/b', title: 'A new title', + description: 'A new description' } + )) + + returned.should have(1).feed + returned.first.items.should have(1).item + returned.first.items.first.title.should eq 'A new title' + end + + it 'judges each of several feeds on its own' do + returned = run({}, + DigestSpec.feed(title: 'One title', description: 'One description'), + DigestSpec.feed(title: 'Another title', + description: 'Another description')) + + returned.should have(2).feeds + record.count.should eq 2 + end + + it 'returns the shape every plugin returns' do + returned = run({}, DigestSpec.feed(title: 'A title', description: 'A description')) + + returned.should be_an(Array) + returned.each { |feed| feed.should respond_to(:items) } + end + end + + describe 'a database that fails' do + let(:feed) { DigestSpec.feed(title: 'A title', description: 'A description') } + + # The point of this plugin is that what it passed on is recorded. An item + # passed on after a failed write would be published again next run, so the + # failure ends the run instead. + it 'does not swallow a failed write' do + record.stub(:create!) { raise ActiveRecord::StatementInvalid, 'no such table' } + + lambda { run({}, feed) }.should raise_error(ActiveRecord::StatementInvalid) + end + + # Two runs of one Recipe overlapping: both read the digest as absent, and + # the unique index is what stops the second from storing it twice. The + # stubbed read is how one process is made to see what the other had not + # committed when it looked. + it 'reads a rejected duplicate write as an item it has seen' do + run({}, feed) + record.stub(:exists?).and_return(false) + + lambda { + run({}, DigestSpec.feed(title: 'A title', description: 'A description')). + should have(0).feed + }.should change(record, :count).by(0) + end + + it 'does not swallow a failed read' do + record.stub(:exists?) { raise ActiveRecord::StatementInvalid, 'database is locked' } + + lambda { run({}, feed) }.should raise_error(ActiveRecord::StatementInvalid) + end + end + + it 'stores a SHA-256 digest, which is 64 hexadecimal characters' do + run({}, DigestSpec.feed(title: 'A title', description: 'A description')) + + record.first.digest.should match(/\A[0-9a-f]{64}\z/) + end +end