Add CustomFeedWeb plugin to build feeds from HTML index pages

Gemfile

@@ -42,9 +42,10 @@ group :plugins, :store, optional: true do
 end
 
 # An HTML parser, for the plugins that read HTML: FilterFullFeed,
-# FilterImageSource, FilterDescriptionLink, and FeedParser.parse_html for
-# SubscriptionLink and SubscriptionTumblr. PublishMarkdown uses it when it is
-# installed and reduces a body to text without it.
+# FilterImageSource, FilterDescriptionLink, CustomFeedWeb, and
+# FeedParser.parse_html for SubscriptionLink and SubscriptionTumblr.
+# PublishMarkdown uses it when it is installed and reduces a body to text
+# without it.
 group :plugins, :html, optional: true do
   gem 'nokogiri', '>= 1.15', '< 2.0'
 end

README.md

@@ -101,7 +101,7 @@ a plugin set every part of which still has somewhere to talk to. See
 
 - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to
   wire a pipeline together.
-- **34 plugins** across seven categories: subscribe, custom feed, filter,
+- **35 plugins** across seven categories: subscribe, custom feed, filter,
   store, provide, notify, publish — and every one of them has a current use.
 - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain
   Markdown document, to a file or to standard output, with no service and no
@@ -369,13 +369,13 @@ like a shipped plugin replaces it.
 
 ### Which plugins still work
 
-34 plugins ship with the gem. Every one is classified in
+35 plugins ship with the gem. Every one is classified in
 [`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason
 for its status:
 
 | Status | Count | Meaning |
 | --- | --- | --- |
-| **Supported** | 23 | Works on the supported Rubies with current dependencies |
+| **Supported** | 24 | Works on the supported Rubies with current dependencies |
 | **Supported (external)** | 10 | Works, but needs something you provide: a service, a command, a credential, a data file |
 | **Needs rework** | 1 | The service exists; this plugin speaks a replaced interface |
 

doc/DEPLOYMENT.md

@@ -456,7 +456,7 @@ bundle. `plugins` is every group in the first block at once.
 | Plugin | Needs | Installed gem | Checkout group | Status |
 | --- | --- | --- | --- | --- |
 | `StorePermalink`, `StoreFullText` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | Supported |
-| `FilterImageSource`, `FilterDescriptionLink`, `SubscriptionLink`, `SubscriptionTumblr` | `nokogiri` | `gem install nokogiri` | `html` | Supported (`SubscriptionTumblr` external) |
+| `FilterImageSource`, `FilterDescriptionLink`, `SubscriptionLink`, `SubscriptionTumblr`, `CustomFeedWeb` | `nokogiri` | `gem install nokogiri` | `html` | Supported (`SubscriptionTumblr` external) |
 | `PublishMarkdown` | `nokogiri`, for HTML bodies only | `gem install nokogiri` | `html` | Supported; runs without it |
 | `FilterSanitize` | `sanitize` | `gem install sanitize` | `sanitize` | Supported |
 | `autodiscovery` and `inspect` subcommands | `feedbag` | `gem install feedbag` | `autodiscovery` | Supported |

doc/PLUGINS.md

@@ -688,6 +688,130 @@ to follow before putting it in `cron`, and set `interval`.
 
 ### 6.2 CustomFeed
 
+#### CustomFeedWeb — **Supported**
+
+`custom_feed/web.rb`. Fetches HTML index pages and builds one feed per page
+from the article links it lists. For a site that publishes no feed and whose
+list page has more structure than `SubscriptionLink` reads: CSS selectors say
+where an article is and what belongs to it, and the links are resolved,
+filtered and deduplicated on the way into the feed.
+
+```yaml
+  - module: CustomFeedWeb
+    config:
+      retry: 2
+      interval: 1
+      sites:
+        - url: https://example.com/news/
+          name: Example News
+          item_selector: article
+          link_selector: h2 a
+          title_selector: h2
+          description_selector: .summary
+          date_selector: time
+          same_host: true
+          include:
+            - ^https://example\.com/news/
+          exclude:
+            - /category/
+          fetch_items: 50
+```
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `sites` | sequence | Page mappings, fetched in order. Required. |
+| `retry` | integer | Attempts after the first, per page. Default `0`. |
+| `interval` | integer | Seconds between requests. Default `0`. |
+
+Each element of `sites` is a mapping. A bare `- https://example.com/news/` is
+not accepted: a page's settings are what this plugin is for, and one shorthand
+kept working forever is a second format to support.
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `url` | string | The page to fetch. Required. |
+| `name` | string | Channel title. Default the page's `<title>`, then its host. |
+| `item_selector` | string | The node one article occupies. |
+| `link_selector` | string | The permalink, evaluated inside the article where there is one. Default `a[href]`. |
+| `title_selector` | string | The title, inside the article. Default the link's own text. |
+| `description_selector` | string | The summary the page prints, taken as text. |
+| `date_selector` | string | The publication date, inside the article. |
+| `same_host` | boolean | Drop a URL whose host is not the page's. Default `true`. |
+| `include` | sequence | Regular expressions; a URL matching none of them is dropped. |
+| `exclude` | sequence | Regular expressions; a URL matching one of them is dropped. |
+| `fetch_items` | integer | Items per page, from the top. Default `100`; `0`, a negative value and an absent one all mean the default. |
+
+There are three ways a page is read, and which one applies follows from the
+selectors given:
+
+- **Neither `item_selector` nor `link_selector`.** Every `a[href]` on the page
+  is a candidate and its text is the title. This is the mode to start with.
+- **`link_selector` only.** Each node it selects is a candidate, and its text
+  is the title. `main h2 a` is the usual shape of it.
+- **`item_selector`.** Each node it selects is one article, and
+  `link_selector`, `title_selector`, `description_selector` and
+  `date_selector` are evaluated inside that node. Without `link_selector` the
+  article's first `a[href]` is the permalink; without `title_selector` the
+  link's own text is the title.
+
+`title_selector`, `description_selector` and `date_selector` are read inside an
+article, so giving one without `item_selector` names no article to read it in
+and is refused as a settings error.
+
+A candidate URL is resolved against the page it was found on — `/articles/42`,
+`../42` and `//example.com/42` all become the URL a reader would follow — and
+then judged in this order: HTTP or HTTPS, not the page itself, `same_host`,
+`include`, `exclude`, already seen, and finally `fetch_items`. The fragment is
+removed, because two links differing only in their anchor are one article. The
+query string is kept, because `?id=42` is frequently the whole of what
+identifies one; no canonical form is guessed. `same_host` is an exact host
+match, so `blog.example.com` is not `www.example.com`.
+
+The page's own order is kept. A list page's order is the only ordering
+information it carries, and nothing here sorts by date; `FilterSort` is where
+a Recipe asks for that.
+
+`date_selector` prefers the `datetime` attribute of a `<time>` element and
+otherwise parses the node's text. A date that cannot be read is logged and the
+item keeps its place without one — the time the page was fetched is not the
+time the article was published, and is never substituted for it.
+
+The plugin keeps no state: it fetches the page, and what the page lists now is
+what it returns. Whether an item has been published before is the record
+`StorePermalink` keeps, which is what the usual Recipe puts after it:
+
+```yaml
+plugins:
+  - module: CustomFeedWeb
+    config:
+      sites:
+        - url: https://example.com/news/
+          link_selector: main h2 a
+
+  - module: StorePermalink
+    config:
+      db: web-watch.db
+
+  - module: PublishMarkdown
+    config:
+      file: ~/.automatic/markdown/web-watch.md
+      mode: append
+```
+
+A page that could not be fetched is retried, then logged and skipped, and the
+other pages still produce their feeds. Settings that cannot be carried out —
+a site that is not a mapping, a missing or unfetchable `url`, an `include` or
+`exclude` that is not a regular expression, a selector combination that names
+no article — are refused before anything is fetched, because a second attempt
+would fail identically.
+
+Nothing else is fetched: no article body, no next page, no sitemap, no feed
+autodiscovery, and no link found on the page is followed. One run makes one
+request per site. Set `interval` when several sites are on one host.
+
+Needs `nokogiri`, which it reads the page with: `gem install nokogiri`, or the
+`html` group in a checkout.
+
 #### CustomFeedSVNLog — **Supported (external)**
 
 `custom_feed/svn_log.rb`. Runs `svn log --xml` against a repository and makes a
@@ -1310,11 +1434,11 @@ is a claim that the plugin works.
 
 | Status | Count | Plugins |
 | --- | --- | --- |
-| Supported | 23 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `StorePermalink`, `StoreFullText`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` |
+| Supported | 24 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `StorePermalink`, `StoreFullText`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` |
 | Supported (external) | 10 | `SubscriptionTumblr`, `CustomFeedSVNLog`, `FilterFullFeed`, `ProvideFluentd`, `NotifyIkachan`, `PublishEject`, `PublishMemcached`, `PublishFluentd`, `PublishInstapaper`, `PublishAmazonS3` |
 | Needs rework | 1 | `PublishHatenaBookmark` |
 
-Thirty-four plugins. Every one of them either runs, or names the one thing it
+Thirty-five plugins. Every one of them either runs, or names the one thing it
 needs from the operator; the single exception says what is wrong with it and
 what fixing it would take.
 

doc/VERSIONS

@@ -11,6 +11,7 @@ v26.08 (Release Date: TBD)
 - Classify every shipped plugin by its current support status rather than simulating obsolete services in tests.
 - Modernize the bundled plugins for current Ruby, libraries and services, and remove the integrations that are no longer viable.
 - Remove the plugins for Twitter, Pocket, HipChat, Google Calendar, livedoor Weather, So-net G-Guide and Chan-Toru, and Google News link rewriting; their services or APIs no longer exist, and a Recipe naming one now fails at load.
+- Add the plugin CustomFeedWeb, which builds one feed per HTML index page from the article links it lists, selected with CSS selectors and filtered by host, pattern and count.
 - Rebuild the test and CI strategy for current RSpec and Ruby, with deterministic isolation from user data and external services.
 - Add Markdown as the primary service-independent publication format, with a documented and tested first-run workflow.
 - Rebuild the maintained documentation around current usage, architecture, policy, plugins and deployment, and remove superseded historical documents.