diff --git a/Gemfile b/Gemfile index 5df134c..517cd77 100644 --- a/Gemfile +++ b/Gemfile @@ -35,7 +35,7 @@ gemspec # The table of which plugin needs which gem, and which of those plugins still # work, is in doc/DEPLOYMENT.md and doc/PLUGINS.md section 6. -# StorePermalink and StoreFullText, through plugins/store/database.rb. +# StorePermalink, StoreFullText and StoreDigest, through plugins/store/database.rb. group :plugins, :store, optional: true do gem 'activerecord', '>= 7.1', '< 9.0' gem 'sqlite3', '>= 1.7', '< 3.0' diff --git a/README.md b/README.md index 419bb2d..0851ab5 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ a plugin set every part of which still has somewhere to talk to. See - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to wire a pipeline together. -- **35 plugins** across seven categories: subscribe, custom feed, filter, +- **36 plugins** across seven categories: subscribe, custom feed, filter, store, provide, notify, publish — and every one of them has a current use. - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain Markdown document, to a file or to standard output, with no service and no @@ -369,13 +369,13 @@ like a shipped plugin replaces it. ### Which plugins still work -35 plugins ship with the gem. Every one is classified in +36 plugins ship with the gem. Every one is classified in [`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason for its status: | Status | Count | Meaning | | --- | --- | --- | -| **Supported** | 24 | Works on the supported Rubies with current dependencies | +| **Supported** | 25 | Works on the supported Rubies with current dependencies | | **Supported (external)** | 10 | Works, but needs something you provide: a service, a command, a credential, a data file | | **Needs rework** | 1 | The service exists; this plugin speaks a replaced interface | diff --git a/doc/BASIC_DESIGN.md b/doc/BASIC_DESIGN.md index 75a14eb..7cf1cbd 100644 --- a/doc/BASIC_DESIGN.md +++ b/doc/BASIC_DESIGN.md @@ -325,7 +325,9 @@ plugin implementation and the framework does not use it: class's `column_definition` when it is absent, and provides `for_each_new_feed`, which yields only items whose key is not already stored. `StorePermalink` and `StoreFullText` are this mixin plus a model and a column - list. + list. `StoreDigest` takes the database part of it and decides for itself what + has been seen, because it is identified by a digest of an item's content + rather than by the link `for_each_new_feed` reads. ### 4.10 `db/`, `config/`, `assets/` Fallbacks inside the installation, used when the corresponding part of the user diff --git a/doc/DEPLOYMENT.md b/doc/DEPLOYMENT.md index e04c800..7b9931d 100644 --- a/doc/DEPLOYMENT.md +++ b/doc/DEPLOYMENT.md @@ -455,7 +455,7 @@ bundle. `plugins` is every group in the first block at once. | Plugin | Needs | Installed gem | Checkout group | Status | | --- | --- | --- | --- | --- | -| `StorePermalink`, `StoreFullText` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | Supported | +| `StorePermalink`, `StoreFullText`, `StoreDigest` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | Supported | | `FilterImageSource`, `FilterDescriptionLink`, `SubscriptionLink`, `SubscriptionTumblr`, `CustomFeedWeb` | `nokogiri` | `gem install nokogiri` | `html` | Supported (`SubscriptionTumblr` external) | | `PublishMarkdown` | `nokogiri`, for HTML bodies only | `gem install nokogiri` | `html` | Supported; runs without it | | `FilterSanitize` | `sanitize` | `gem install sanitize` | `sanitize` | Supported | @@ -499,8 +499,7 @@ missing, what needs it and how to get it, and the command exits `1`. ```text automatic: The `activerecord` gem is not installed. It is needed by the store -plugins StorePermalink and StoreFullText. Install it with `gem install -activerecord`, ... +plugins. Install it with `gem install activerecord`, ... ``` ## Your own plugins diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index 73d41e5..2171e23 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -1044,12 +1044,17 @@ been through another filter first is no longer a `NoMethodError`. Persist, and drop what has already been seen. A store plugin is what makes a Recipe safe to run repeatedly. -`StorePermalink` and `StoreFullText` keep their records in SQLite through -ActiveRecord. Both gems are these plugins' own optional dependencies rather than -the framework's: `gem install activerecord sqlite3`, or the `store` group in a -checkout. A Recipe that stores nothing needs neither. +`StorePermalink`, `StoreFullText` and `StoreDigest` keep their records in SQLite +through ActiveRecord. Both gems are these plugins' own optional dependencies +rather than the framework's: `gem install activerecord sqlite3`, or the `store` +group in a checkout. A Recipe that stores nothing needs neither. See [`DEPLOYMENT.md`](DEPLOYMENT.md). +They answer different questions, and a Recipe may ask more than one of them: +`StorePermalink` whether this **link** has been seen, `StoreFullText` whether +this link or title has been stored with its body, `StoreDigest` whether this +**content** has been seen, whatever it was published under. + #### StorePermalink — **Supported** `store/permalink.rb`. Records each item's link in SQLite and passes on only the @@ -1074,6 +1079,124 @@ republished article with a new URL is not stored twice. Pair with | --- | --- | --- | | `db` | string | Database file name, under `~/.automatic/db`. Required. | +#### StoreDigest — **Supported** + +`store/digest.rb`. Takes the SHA-256 digest of the item fields the Recipe names, +records it in SQLite, and passes on only the items whose digest was not recorded +already. Content identity, where `StorePermalink` is URL identity: a page that +reissues one article under a new URL is one item here, and one URL whose content +changed is a new item — the opposite of what `StorePermalink` decides in both +cases. Pair it with `CustomFeedWeb`, whose items are whatever an index page +currently lists. + +| Key | Type | Meaning | +| --- | --- | --- | +| `db` | string | Database file name, under `~/.automatic/db`. Required. | +| `fields` | list | The fields the digest is taken over, in the order written. Default: `title`, `description`. | + +```yaml +plugins: + - module: CustomFeedWeb + config: + sites: + - url: https://example.com/news/ + + - module: StoreDigest + config: + db: web-digest.db + + - module: PublishMarkdown + config: + file: ~/.automatic/markdown/web-watch.md + mode: append +``` + +The first run passes on everything the page listed and records a digest for +each. The second passes on nothing, because the page still lists the same +articles. A run in which one article has been added passes on that one. + +**`fields`** names any of `title`, `link`, `description`, `author`, `comments`, +`source` and `content_encoded`. `date` is not among them — an item republished +unchanged carries a new date often enough to defeat the purpose — and neither is +`enclosure`, which is a structure rather than a value. + +- The order is part of the fingerprint. `[title, description]` and + `[description, title]` are two different specifications and produce different + digests; nothing is sorted behind the Recipe's back. +- **The fields named are the fields used.** A Recipe that asks for + `content_encoded` gets `content_encoded`, and an item whose body is empty is + not quietly judged on its title instead. What the Recipe says two identical + items are is what this plugin obeys. +- Anything but an absent `fields` is taken as written: an empty list, a name + that is not a field, a name given twice and a value that is not a list are + each refused with an `ArgumentError` before the database is opened, rather + than corrected into something the Recipe did not ask for. +- `db` is required, and an empty name is refused the same way. + +**What "the same content" means here.** Each value is read as UTF-8, with +invalid and undefined characters replaced, normalized to Unicode NFC, its runs +of whitespace collapsed to one space and its ends trimmed. The values are joined +with their field names into one canonical string — `title`, NUL, the title, NUL, +`description`, NUL, the description — and that string is hashed with SHA-256. +The algorithm is fixed; there is no setting for it and no column recording it. + +Two items are therefore the same item when their selected fields are **exactly** +equal after that normalization, and not otherwise. A difference of case, of +punctuation or of markup is a difference of content. This is not similarity +matching: there is no fuzzy comparison, no edit distance, no embedding and no +semantic judgement anywhere in it, and two articles that report one event in +different words are two items. + +**An item with nothing to hash is passed on, not stored.** Where every field the +Recipe named is empty after normalization — `fields: [description]` on an item +that has no description — there is nothing to identify the item by. Hashing the +empty string would make every such item the same item and silence all but the +first of them for good, so instead the plugin logs a warning naming the item's +link and passes it on unjudged. An item is never lost for having too little +content. A field that is empty while another is not takes part in the digest as +an empty value, so an item with a title and no description differs from the same +item with both. + +**A database failure ends the run.** A failed read or a failed write is not +rescued here, which is deliberate and is a difference from `StoreFullText`: an +item passed on after its digest failed to store would be published again on the +next run, and de-duplication that quietly stops de-duplicating is worse than a +run that stops. The digest column carries a unique index, so two runs of one +Recipe overlapping cannot both store one digest; the second write is rejected +and its item is treated as seen. + +The digest is all that is stored — no title, no body, no URL. Recording what an +item said is `StoreFullText`'s work, and pairing the two is how a Recipe gets +both. + +`StorePermalink` and `StoreDigest` may stand in one Recipe, each with its own +database, and the pair is worth having: the first drops what has been seen at +that URL, the second drops what has been seen under any URL. + +```yaml +plugins: + - module: CustomFeedWeb + config: + sites: + - url: https://example.com/news/ + + - module: FilterFullFeed + + - module: StoreDigest + config: + db: fulltext-digest.db + fields: + - content_encoded + + - module: PublishMarkdown + config: + file: ~/.automatic/markdown/web-watch.md + mode: append +``` + +With the body fetched first, the digest is taken over the article itself, so a +headline edited between runs no longer republishes the article. + #### StoreFile — **Supported** `store/file.rb`. Downloads what each link points at and rewrites the link to a @@ -1434,11 +1557,11 @@ is a claim that the plugin works. | Status | Count | Plugins | | --- | --- | --- | -| Supported | 24 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `StorePermalink`, `StoreFullText`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` | +| Supported | 25 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `StorePermalink`, `StoreFullText`, `StoreDigest`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` | | Supported (external) | 10 | `SubscriptionTumblr`, `CustomFeedSVNLog`, `FilterFullFeed`, `ProvideFluentd`, `NotifyIkachan`, `PublishEject`, `PublishMemcached`, `PublishFluentd`, `PublishInstapaper`, `PublishAmazonS3` | | Needs rework | 1 | `PublishHatenaBookmark` | -Thirty-five plugins. Every one of them either runs, or names the one thing it +Thirty-six plugins. Every one of them either runs, or names the one thing it needs from the operator; the single exception says what is wrong with it and what fixing it would take. diff --git a/doc/VERSIONS b/doc/VERSIONS index c1fa7fe..91d439d 100644 --- a/doc/VERSIONS +++ b/doc/VERSIONS @@ -12,6 +12,7 @@ v26.08 (Release Date: TBD) - Modernize the bundled plugins for current Ruby, libraries and services, and remove the integrations that are no longer viable. - Remove the plugins for Twitter, Pocket, HipChat, Google Calendar, livedoor Weather, So-net G-Guide and Chan-Toru, and Google News link rewriting; their services or APIs no longer exist, and a Recipe naming one now fails at load. - Add the plugin CustomFeedWeb, which builds one feed per HTML index page from the article links it lists, selected with CSS selectors and filtered by host, pattern and count. +- Add the plugin StoreDigest, which drops items by content rather than by URL, on a SHA-256 digest of the item fields a Recipe names. - Rebuild the test and CI strategy for current RSpec and Ruby, with deterministic isolation from user data and external services. - Add Markdown as the primary service-independent publication format, with a documented and tested first-run workflow. - Rebuild the maintained documentation around current usage, architecture, policy, plugins and deployment, and remove superseded historical documents. diff --git a/plugins/store/database.rb b/plugins/store/database.rb index 8157171..cd01b52 100644 --- a/plugins/store/database.rb +++ b/plugins/store/database.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Feb 27, 2012 -# Updated:: Aug 15, 2026 +# Updated:: Aug 17, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. # # The SQLite storage the store plugins share. ActiveRecord and sqlite3 are the @@ -14,9 +14,9 @@ Automatic.require_optional('active_record', gem_name: 'activerecord', - needed_by: 'the store plugins StorePermalink and StoreFullText') + needed_by: 'the store plugins') Automatic.require_optional('sqlite3', - needed_by: 'the store plugins StorePermalink and StoreFullText') + needed_by: 'the store plugins') module Automatic::Plugin module Database diff --git a/plugins/store/digest.rb b/plugins/store/digest.rb new file mode 100644 index 0000000..72e10f2 --- /dev/null +++ b/plugins/store/digest.rb @@ -0,0 +1,212 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Store::Digest +# Description:: Drop items whose content has been seen before, by SHA-256 digest. +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 17, 2026 +# Updated:: Aug 17, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. +# +# Content identity, where StorePermalink is URL identity. The fields the Recipe +# names are normalized, joined into one canonical string and hashed, and the +# digest is what the database holds: an item whose digest is already there has +# been seen, whatever its link says, and an item whose link has been seen before +# under different content has not. +# +# It is exact matching, of the content the Recipe selected. Whitespace runs and +# Unicode composition differences are absorbed; nothing else is. Two articles +# that say the same thing in different words are two articles here, and making +# them one is not this plugin's work -- see doc/PLUGINS.md section 6.4. + +require 'digest' +require_relative 'database' + +module Automatic::Plugin + # Named for what it holds rather than as `Digest`, which is Ruby's own + # hashing module and is what computes the value stored here. + class DigestRecord < ActiveRecord::Base + end + + class StoreDigest + include Automatic::Plugin::Database + + # The item fields a digest may be taken over. `date` is not one: an item + # republished unchanged carries a new date often enough that including it + # would defeat the purpose. `enclosure` is not one either, being a + # structure rather than a value. + FIELDS = %w[title link description author comments source content_encoded].freeze + + # What an item is, absent a Recipe saying otherwise: what it says, not + # where it is. A feed that reissues an article under a new URL is the case + # this plugin exists for. + DEFAULT_FIELDS = %w[title description].freeze + + # The field name is part of what is hashed, and this separates it from its + # value and one field from the next. A byte no title, body or URL contains, + # so that no arrangement of two fields collides with another. + SEPARATOR = "\0" + + def initialize(config, pipeline = []) + @config = config || {} + @pipeline = pipeline + end + + def column_definition + { digest: :string, created_at: :string } + end + + def model_class + Automatic::Plugin::DigestRecord + end + + # Records the digest of each item's selected fields and passes on only the + # digests not already recorded. + # + # Nothing is rescued around the database. A store plugin that failed to + # write and passed the item on anyway would publish it again on the next + # run, which is the one thing this plugin is for. + def run + validate_settings + prepare_database + + @pipeline.each_with_object([]) do |feeds, returned| + next if feeds.nil? + + new_items = feeds.items.select { |item| new_item?(item) } + returned << Automatic::FeedMaker.create_pipeline(new_items) unless new_items.empty? + end + end + + private + + # The table the mixin builds, plus the constraint. Two runs of one Recipe + # overlapping -- a `cron` entry that takes longer than its interval -- would + # otherwise both read a digest as absent and both store it. The unique + # index makes the second write fail instead, which #store below reads as + # what it is. + def create_table + super + ActiveRecord::Base.connection.add_index(model_class.table_name, :digest, unique: true) + end + + # Everything the Recipe has to get right, checked before the database file + # is opened. A Recipe this plugin cannot carry out is the operator's + # mistake and will be the same mistake next hour. + def validate_settings + raise ArgumentError, 'StoreDigest needs a db file name' if @config['db'].to_s.strip.empty? + + fields + end + + # The fields, in the order the Recipe wrote them. That order is part of the + # fingerprint and is not sorted here: a Recipe that changes it has said + # something different, and its digests are different digests. + # + # Only an absent `fields` takes the default. Every other way of getting it + # wrong is refused rather than corrected, because a Recipe's own statement + # of what makes two items the same is what this plugin has to obey. + def fields + @fields ||= validated_fields + end + + def validated_fields + given = @config['fields'] + return DEFAULT_FIELDS if given.nil? + + unless given.is_a?(Array) + raise ArgumentError, "StoreDigest takes a list of fields, not #{given.inspect}" + end + + names = given.map(&:to_s) + raise ArgumentError, 'StoreDigest was given an empty fields list' if names.empty? + + unknown = names - FIELDS + unless unknown.empty? + raise ArgumentError, + "StoreDigest cannot take a digest over #{unknown.join(', ')}; " \ + "the fields are #{FIELDS.join(', ')}" + end + + duplicated = names.tally.select { |_name, count| count > 1 }.keys + unless duplicated.empty? + raise ArgumentError, "StoreDigest was given #{duplicated.join(', ')} twice" + end + + names + end + + # Whether the item goes downstream. An item whose digest was stored here is + # new, one whose digest was already stored is not, and one there is nothing + # to hash goes on unjudged. + def new_item?(item) + digest = digest_for(item) + + if digest.nil? + Automatic::Log.puts('warn', + "StoreDigest: no digestable content for #{item.link}; " \ + 'passing item unchanged') + return true + end + + return false if stored_digest?(digest) + + store(digest, item) + end + + def stored_digest?(digest) + model_class.exists?(digest: digest) + end + + # True when this run is the one that stored the digest. The read above + # answers this on its own for a single run; the rescue is for the run + # overlapping another, where the row appeared between the two statements. + def store(digest, item) + model_class.create!(digest: digest, created_at: Time.now.strftime('%Y/%m/%d %X')) + Automatic::Log.puts('info', "Saving Digest: #{digest} (#{item.link})") + true + rescue ActiveRecord::RecordNotUnique + false + end + + # The digest of the item's selected fields, or nil where the Recipe + # selected nothing the item has. + # + # An item with nothing to hash is not stored under the digest of the empty + # string: every item with no description would then be the same item as + # every other, and the first of them would silence the rest for good. + def digest_for(item) + values = fields.map { |field| [field, normalize(value_of(item, field))] } + return nil if values.all? { |_field, value| value.empty? } + + canonical = values.map { |field, value| "#{field}#{SEPARATOR}#{value}" }.join(SEPARATOR) + ::Digest::SHA256.hexdigest(canonical) + end + + # A field an item does not carry reads as absent rather than raising. + # `content_encoded` is the case that matters: an item built by a filter has + # it, one straight from an RSS 2.0 feed may not. + def value_of(item, field) + item.respond_to?(field) ? item.public_send(field) : nil + end + + # What two spellings of one string have to survive to hash alike: the + # encoding, the composition of accented characters, and how much whitespace + # a feed happens to have put between words. Nothing beyond that -- case, + # punctuation and markup are content here, and an item that differs in them + # is a different item. + # + # `scrub` is not redundant after `encode`: a conversion whose source and + # destination encodings are the same is skipped, invalid bytes and all, and + # `unicode_normalize` raises on what is left. + def normalize(value) + value.to_s. + encode('UTF-8', invalid: :replace, undef: :replace). + scrub. + unicode_normalize(:nfc). + gsub(/\s+/, ' '). + strip + end + end +end diff --git a/spec/plugins/store/digest_spec.rb b/spec/plugins/store/digest_spec.rb new file mode 100644 index 0000000..8f2eed4 --- /dev/null +++ b/spec/plugins/store/digest_spec.rb @@ -0,0 +1,424 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Store::Digest +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 17, 2026 +# Updated:: Aug 17, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') + +# The store plugins keep their records in SQLite through ActiveRecord. Both +# gems are the store plugins' own, declared in the Gemfile's optional :plugins +# and :store groups, so this spec runs only where the operator has installed +# them. See doc/POLICY.md section 5. +return unless AutomaticSpec.optional_dependency?('activerecord') && + AutomaticSpec.optional_dependency?('sqlite3') + +require 'store/digest' +require 'digest' +require 'pathname' + +# Items with the fields these examples are about. The pipeline generator in +# spec_helper builds an item from a positional list that has no +# content_encoded, and a digest taken over a body needs one. +module DigestSpec + module_function + + def feed(*items) + channel = RSS::Rss::Channel.new + items.each { |attributes| channel.items << item(attributes) } + rss = RSS::Rss.new([]) + rss.instance_variable_set(:@channel, channel) + rss + end + + def item(attributes) + item = RSS::Rss::Channel::Item.new + item.link = attributes.fetch(:link, 'https://example.com/news/1') + item.title = attributes[:title] unless attributes[:title].nil? + item.instance_variable_set(:@description, attributes.fetch(:description, '').to_s) + item.author = attributes[:author] unless attributes[:author].nil? + item.content_encoded = attributes[:content_encoded] unless attributes[:content_encoded].nil? + item + end + + # The canonical representation the plugin hashes, written out here rather + # than taken from the plugin, so that an example asserts the format instead + # of agreeing with whatever the plugin currently builds. + def digest(*pairs) + ::Digest::SHA256.hexdigest(pairs.map { |field, value| "#{field}\0#{value}" }.join("\0")) + end +end + +describe Automatic::Plugin::StoreDigest do + let(:db) { 'test_digest.db' } + let(:record) { Automatic::Plugin::DigestRecord } + + # A run of a Recipe: the plugin is built for the pipeline it is handed, and + # the next run is the next instance, as the framework builds it. + def run(config, *feeds) + Automatic::Plugin::StoreDigest.new({ 'db' => db }.merge(config), feeds).run + end + + def reset_database + path = Pathname(AutomaticSpec.db_dir).cleanpath + db + path.delete if path.exist? + end + + # The file is removed and the table rebuilt, so that an example starts having + # seen nothing and can ask the model for a count. + before do + reset_database + Automatic::Plugin::StoreDigest.new('db' => db).run + end + + describe 'what it passes on' do + it 'passes on an item it has not seen' do + returned = run({}, DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned.should have(1).feed + returned.first.items.should have(1).item + returned.first.items.first.title.should eq 'Ruby 4.0 released' + end + + it 'stores the digest of an item it has not seen' do + lambda { + run({}, DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + }.should change(record, :count).by(1) + + record.first.digest.should eq DigestSpec.digest( + ['title', 'Ruby 4.0 released'], + ['description', 'Ruby 4.0 is now available.'] + ) + end + + it 'drops the same item on the next run' do + feed = lambda { + DigestSpec.feed(title: 'Ruby 4.0 released', description: 'Ruby 4.0 is now available.') + } + + run({}, feed.call).should have(1).feed + lambda { + run({}, feed.call).should have(0).feed + }.should change(record, :count).by(0) + end + + it 'passes on the first of two items of one content in one run' do + returned = run({}, DigestSpec.feed( + { link: 'https://example.com/a', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.' }, + { link: 'https://example.com/b', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.' } + )) + + returned.first.items.should have(1).item + record.count.should eq 1 + end + + it 'passes on the first of two feeds of one content in one run' do + returned = run({}, + DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.'), + DigestSpec.feed(title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned.should have(1).feed + returned.first.items.should have(1).item + record.count.should eq 1 + end + + it 'takes two links to one content as one item' do + run({}, DigestSpec.feed(link: 'https://example.com/a', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned = run({}, DigestSpec.feed(link: 'https://example.com/b', + title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + returned.should have(0).feed + end + + # The difference from StorePermalink: the link is not what identifies an + # item here, so one URL whose content changed is a new item. + it 'takes one link with new content as a new item' do + run({}, DigestSpec.feed(link: 'https://example.com/a', title: 'Ruby 4.0 released', + description: 'Ruby 4.0 is now available.')) + + returned = run({}, DigestSpec.feed(link: 'https://example.com/a', + title: 'Ruby 4.0.1 released', + description: 'Ruby 4.0.1 is now available.')) + returned.should have(1).feed + record.count.should eq 2 + end + end + + describe 'fields' do + it 'takes the digest over title and description by default' do + run({}, DigestSpec.feed(title: 'A title', description: 'A description', + author: 'An author')) + + record.first.digest.should eq DigestSpec.digest(['title', 'A title'], + ['description', 'A description']) + end + + it 'takes the digest over title alone when asked to' do + config = { 'fields' => ['title'] } + run(config, DigestSpec.feed(title: 'A title', description: 'One description')) + + returned = run(config, DigestSpec.feed(title: 'A title', + description: 'Another description')) + returned.should have(0).feed + record.first.digest.should eq DigestSpec.digest(['title', 'A title']) + end + + it 'takes the digest over description alone when asked to' do + config = { 'fields' => ['description'] } + run(config, DigestSpec.feed(title: 'One title', description: 'A description')) + + returned = run(config, DigestSpec.feed(title: 'Another title', + description: 'A description')) + returned.should have(0).feed + record.first.digest.should eq DigestSpec.digest(['description', 'A description']) + end + + it 'takes the digest over the link when asked to' do + config = { 'fields' => ['link'] } + run(config, DigestSpec.feed(link: 'https://example.com/a', title: 'One title')) + + run(config, DigestSpec.feed(link: 'https://example.com/a', + title: 'Another title')).should have(0).feed + run(config, DigestSpec.feed(link: 'https://example.com/b', + title: 'One title')).should have(1).feed + end + + it 'takes the digest over content_encoded when asked to' do + config = { 'fields' => ['content_encoded'] } + run(config, DigestSpec.feed(title: 'One title', content_encoded: '

A body.

')) + + run(config, DigestSpec.feed(title: 'Another title', + content_encoded: '

A body.

')).should have(0).feed + run(config, DigestSpec.feed(title: 'One title', + content_encoded: '

Another body.

')).should have(1).feed + end + + it 'names every field it was given in the canonical representation' do + run({ 'fields' => %w[title link description] }, + DigestSpec.feed(link: 'https://example.com/a', title: 'A title', + description: 'A description')) + + record.first.digest.should eq DigestSpec.digest(['title', 'A title'], + ['link', 'https://example.com/a'], + ['description', 'A description']) + end + + it 'takes the fields in the order the Recipe wrote them' do + item = { title: 'A title', description: 'A description' } + + run({ 'fields' => %w[title description] }, DigestSpec.feed(item)) + forward = record.first.digest + + reset_database + run({ 'fields' => %w[description title] }, DigestSpec.feed(item)) + record.first.digest.should_not eq forward + end + end + + # Two items, one run after the other: one record where the plugin read them + # as one content, two where it read them as two. + describe 'normalization' do + def records_for(first, second) + run({}, DigestSpec.feed(first)) + run({}, DigestSpec.feed(second)) + record.count + end + + it 'reads a run of spaces as one space' do + records_for({ title: 'Ruby 4.0 released', description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + it 'reads a newline as a space' do + records_for({ title: "Ruby 4.0\nreleased", description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + it 'reads a tab as a space' do + records_for({ title: "Ruby 4.0\treleased", description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + it 'ignores whitespace around a value' do + records_for({ title: ' Ruby 4.0 released ', description: "Out now.\n" }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 1 + end + + # The same word composed and decomposed: U+00E9, and e followed by the + # combining acute accent U+0301. + it 'reads the two Unicode compositions of one string as one string' do + records_for({ title: "Café opens", description: 'Out now.' }, + { title: "Café opens", description: 'Out now.' }).should eq 1 + end + + it 'reads a difference in case as different content' do + records_for({ title: 'Ruby 4.0 Released', description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 2 + end + + it 'reads a difference in punctuation as different content' do + records_for({ title: 'Ruby 4.0 released!', description: 'Out now.' }, + { title: 'Ruby 4.0 released', description: 'Out now.' }).should eq 2 + end + end + + describe 'an item with nothing to hash' do + let(:config) { { 'fields' => ['description'] } } + + it 'stores no digest when every field it was given is empty' do + lambda { + run(config, DigestSpec.feed(title: 'A title', description: ' ')) + }.should change(record, :count).by(0) + end + + it 'passes the item on rather than losing it' do + returned = run(config, DigestSpec.feed(title: 'A title', description: '')) + + returned.should have(1).feed + returned.first.items.should have(1).item + end + + it 'takes a digest while any field it was given has content' do + lambda { + run({}, DigestSpec.feed(title: 'Ruby 4.0 released', description: '')) + }.should change(record, :count).by(1) + + record.first.digest.should eq DigestSpec.digest(['title', 'Ruby 4.0 released'], + ['description', '']) + end + end + + describe 'a Recipe it cannot carry out' do + def error_for(config) + lambda { + run(config, DigestSpec.feed(title: 'A title', description: 'A description')) + } + end + + it 'refuses an empty fields list' do + error_for('fields' => []).should raise_error(ArgumentError) + end + + it 'refuses a field it has no value for' do + error_for('fields' => %w[title foobar]).should raise_error(ArgumentError, /foobar/) + end + + it 'refuses fields that are not a list' do + error_for('fields' => 'title').should raise_error(ArgumentError) + end + + it 'refuses a field named twice' do + error_for('fields' => %w[title title]).should raise_error(ArgumentError, /title/) + end + + it 'refuses a Recipe with no db' do + lambda { + Automatic::Plugin::StoreDigest.new({}, []).run + }.should raise_error(ArgumentError) + end + + it 'refuses a db that is an empty name' do + lambda { + Automatic::Plugin::StoreDigest.new({ 'db' => '' }, []).run + }.should raise_error(ArgumentError) + end + end + + describe 'the pipeline it returns' do + it 'skips a nil feed' do + returned = run({}, nil, DigestSpec.feed(title: 'A title', description: 'A description')) + + returned.should have(1).feed + returned.first.items.should have(1).item + end + + it 'returns no feed where every item of one has been seen' do + feed = lambda { DigestSpec.feed(title: 'A title', description: 'A description') } + + run({}, feed.call).should have(1).feed + run({}, feed.call).should have(0).feed + end + + it 'returns a feed of the new items where a feed holds both' do + run({}, DigestSpec.feed(title: 'A title', description: 'A description')) + + returned = run({}, DigestSpec.feed( + { link: 'https://example.com/a', title: 'A title', description: 'A description' }, + { link: 'https://example.com/b', title: 'A new title', + description: 'A new description' } + )) + + returned.should have(1).feed + returned.first.items.should have(1).item + returned.first.items.first.title.should eq 'A new title' + end + + it 'judges each of several feeds on its own' do + returned = run({}, + DigestSpec.feed(title: 'One title', description: 'One description'), + DigestSpec.feed(title: 'Another title', + description: 'Another description')) + + returned.should have(2).feeds + record.count.should eq 2 + end + + it 'returns the shape every plugin returns' do + returned = run({}, DigestSpec.feed(title: 'A title', description: 'A description')) + + returned.should be_an(Array) + returned.each { |feed| feed.should respond_to(:items) } + end + end + + describe 'a database that fails' do + let(:feed) { DigestSpec.feed(title: 'A title', description: 'A description') } + + # The point of this plugin is that what it passed on is recorded. An item + # passed on after a failed write would be published again next run, so the + # failure ends the run instead. + it 'does not swallow a failed write' do + record.stub(:create!) { raise ActiveRecord::StatementInvalid, 'no such table' } + + lambda { run({}, feed) }.should raise_error(ActiveRecord::StatementInvalid) + end + + # Two runs of one Recipe overlapping: both read the digest as absent, and + # the unique index is what stops the second from storing it twice. The + # stubbed read is how one process is made to see what the other had not + # committed when it looked. + it 'reads a rejected duplicate write as an item it has seen' do + run({}, feed) + record.stub(:exists?).and_return(false) + + lambda { + run({}, DigestSpec.feed(title: 'A title', description: 'A description')). + should have(0).feed + }.should change(record, :count).by(0) + end + + it 'does not swallow a failed read' do + record.stub(:exists?) { raise ActiveRecord::StatementInvalid, 'database is locked' } + + lambda { run({}, feed) }.should raise_error(ActiveRecord::StatementInvalid) + end + end + + it 'stores a SHA-256 digest, which is 64 hexadecimal characters' do + run({}, DigestSpec.feed(title: 'A title', description: 'A description')) + + record.first.digest.should match(/\A[0-9a-f]{64}\z/) + end +end