From 2eaf8119e138824712570b7b94adfefe3a631b39 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 10:25:06 +0000 Subject: [PATCH 1/2] Document how a checkout gets a Recipe's optional dependencies A Recipe needs the sum of what its plugins need, and in a source checkout that sum is a list of Bundler groups rather than a list of gems. Nothing said so end to end: the table listed each plugin's group, and an operator running a three-plugin Recipe still had to discover, one failed run at a time, that `html` alone gets past CustomFeedWeb and stops at StoreDigest. doc/DEPLOYMENT.md takes one Recipe -- CustomFeedWeb, StoreDigest and PublishMarkdown -- from written to running in seven steps beside the table it reads from, and adds why a checkout resolves through the bundle at all, what the missing-gem message does and does not mean there, which commands answer "what does this bundle hold", and when to select `plugins` rather than the groups a Recipe names. The other three documents gain a pointer each and no second copy. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019fgz9HsShz5W9ytoVms5BH --- README.md | 10 +- doc/DEPLOYMENT.md | 262 ++++++++++++++++++++++++++++++++++++++++++++-- doc/PLUGINS.md | 6 ++ doc/QUICKSTART.md | 19 +++- 4 files changed, 281 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 0851ab5..c7520fd 100644 --- a/README.md +++ b/README.md @@ -233,10 +233,12 @@ for. If the `bundle` command is unavailable, install Bundler first with `gem install bundler`. In a checkout, every `automatic` below becomes `bundle exec bin/automatic`, and -`bundle exec` sees only the bundle — so a plugin's gem is added with a group -rather than with `gem install`. The group names, and which plugin needs which -gem, are in [`doc/DEPLOYMENT.md`](doc/DEPLOYMENT.md); what each plugin does is -in [`doc/PLUGINS.md`](doc/PLUGINS.md). +the checkout resolves its gems through the bundle — so a plugin's gem is added +with a group rather than with `gem install`, and a Recipe using plugins from two +groups selects both at once. The group names, which plugin needs which gem, and +a Recipe taken step by step through choosing its groups are in +[`doc/DEPLOYMENT.md`](doc/DEPLOYMENT.md); what each plugin does is in +[`doc/PLUGINS.md`](doc/PLUGINS.md). ## 6. Quick start diff --git a/doc/DEPLOYMENT.md b/doc/DEPLOYMENT.md index 7b9931d..8b369d2 100644 --- a/doc/DEPLOYMENT.md +++ b/doc/DEPLOYMENT.md @@ -114,13 +114,15 @@ bundle install ``` Several at once are space-separated: `bundle config set --local with "store -html"`. The group names are in the table below. +html"`, and a Recipe using plugins from two groups needs exactly that. The group +names are in the table below, and "Working out what a Recipe needs, in a +checkout" takes one Recipe through choosing them. -In a checkout, `gem install ` on its own is **not** enough: `bundle exec` -puts only the bundle on the load path, so a gem the `Gemfile` does not mention -is not visible to it. Use the group, which is why the groups exist. Outside a -checkout — the installed gem, run as `automatic` — there is no bundle and `gem -install ` is exactly right. +In a checkout, `gem install ` on its own is **not** enough: the checkout +resolves its gems through the bundle, so a gem the `Gemfile` does not mention is +not visible to it however plainly `gem list` shows it. Use the group, which is +why the groups exist. Outside a checkout — the installed gem, run as `automatic` +— there is no bundle and `gem install ` is exactly right. Everything below that says `automatic` becomes `bundle exec bin/automatic` in a checkout. @@ -502,6 +504,247 @@ automatic: The `activerecord` gem is not installed. It is needed by the store plugins. Install it with `gem install activerecord`, ... ``` +### Working out what a Recipe needs, in a checkout + +The table says what a plugin needs. A Recipe needs the **sum** of what its +plugins need, worked out before the first run rather than discovered one failed +run at a time, and in a checkout that sum is a list of groups. This Recipe — +index pages watched for new articles, de-duplicated by content, published as one +Markdown document — is the worked example: + +```yaml +# ~/.automatic/config/web2markdown.yml +plugins: + - module: CustomFeedWeb + config: + retry: 2 + interval: 2 + sites: + # Python Insider. Its articles are + # https://blog.python.org/2026/08/python-3147-31315/ and the like. + - name: Python Insider + url: https://blog.python.org/ + link_selector: 'a[href]' + include: + - '^https://blog\.python\.org/20[0-9]{2}/[0-9]{2}/[^/]+/?$' + same_host: true + fetch_items: 20 + + # The Go Blog. Its articles are https://go.dev/blog/pkgsite-api + # and the like. Further sites are further entries of the same shape. + - name: The Go Blog + url: https://go.dev/blog/ + link_selector: 'a[href]' + include: + - '^https://go\.dev/blog/[^/]+$' + same_host: true + fetch_items: 20 + + - module: StoreDigest + config: + db: web2markdown.db + fields: + - title + - link + + - module: PublishMarkdown + config: + file: ~/.automatic/markdown/web.md + mode: append +``` + +**1. List the plugins the Recipe names.** They are the `module` lines, in order: +`CustomFeedWeb`, `StoreDigest`, `PublishMarkdown`. + +**2. Look each one up in the table above.** What that lookup yields here, and +the whole of what this Recipe's dependencies are: + +| Plugin | Needs | Checkout group | +| --- | --- | --- | +| `CustomFeedWeb` | `nokogiri`, to read the index pages | `html` | +| `StoreDigest` | `activerecord`, `sqlite3` | `store` | +| `PublishMarkdown` | nothing of its own | — | + +`PublishMarkdown` is the row worth reading twice. It needs no gem: it reduces an +HTML body with `nokogiri` where one is installed and with its own substitution +where none is, so it neither adds a group here nor fails without one. A plugin's +row in the table is the answer, not a guess from what the plugin does. + +**3. Add the groups up.** This Recipe needs `html` **and** `store`. Getting this +step half right fails half way through the run, because a plugin is loaded when +the pipeline reaches it and not before: with `html` alone, `CustomFeedWeb` +loads, fetches its pages and hands its feeds on, and the run then stops where +the second plugin is loaded. + +```text +automatic: The `activerecord` gem is not installed. It is needed by the store +plugins. Install it with `gem install activerecord`, or in a source checkout add +its group to the bundle; see the optional plugin dependencies in +doc/DEPLOYMENT.md. (cannot load such file -- active_record) +``` + +The first plugin having worked is not the setup being finished. Read the whole +Recipe, then install once. + +**4. Select the groups in the checkout.** + +```sh +cd ~/automaticruby +bundle config set --local with "html store" +``` + +**5. Install them.** + +```sh +bundle install +``` + +**6. Check what was selected.** This prints the setting Bundler will use, and +where it came from: + +```sh +bundle config get with +``` + +```text +Set for your local app (/home/you/automaticruby/.bundle/config): [:html, :store] +``` + +**7. Run the Recipe.** + +```sh +bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml +``` + +Run it twice. The second run publishes nothing, because `StoreDigest` has the +digest of everything the pages listed — the same check the Quick Start makes +with `StorePermalink`, and what makes the Recipe safe to put in `cron`. + +A store plugin's `db` is a **file name**, not a path: it is resolved under +`~/.automatic/db`, or under the checkout's own `db/` where that directory does +not exist yet, so a checkout run before `scaffold` keeps its database inside the +checkout. The `Using Database:` line of an `info`-level log names the file that +was opened, which is the way to check. `PublishMarkdown`'s `file` is the +opposite — a path, with `~` expanded — and the Recipe above uses each as it is +meant. + +Adding a plugin to a Recipe later starts this over at step 1: a `FilterSanitize` +added to the Recipe above brings `sanitize` with it, and the `with` setting has +to name `sanitize` as well as `html` and `store`. + +### Why `bundle exec`, and why `gem install` is not enough here + +A checkout resolves its gems through Bundler whether or not you ask it to. +`lib/automatic/environment.rb` treats a `Gemfile` beside `lib/` as "this is a +source checkout" and requires `bundler/setup` before anything else loads, so +`bin/automatic` run from a checkout sees the bundle and nothing outside it. That +is what makes `gem install` the wrong tool there: the gem installs, `gem list` +prints it, plain `ruby -rnokogiri -e ''` loads it — and the checkout still +reports it as missing, because it is not in the bundle. + +```sh +gem install nokogiri +gem list nokogiri # prints the gem that was just installed +./bin/automatic -c ~/.automatic/config/web2markdown.yml +# automatic: The `nokogiri` gem is not installed. It is needed by CustomFeedWeb... +``` + +Use the group, and then run through `bundle exec`, from the checkout directory: + +```sh +cd ~/automaticruby +bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml +``` + +`bundle exec` matters for what it does when the bundle is **not** in the state +you think it is. The `require 'bundler/setup'` above is deliberately forgiving: +where Bundler is absent, or where a group has been selected but not yet +installed, it is rescued and the program falls back to whatever RubyGems can +activate — quietly, and against gems the `Gemfile.lock` never resolved. +Running the same command through `bundle exec` says so instead: + +```text +bundler: failed to load command: bin/automatic (bin/automatic) +Could not find activerecord-8.1.3.1, sqlite3-2.9.6-x86_64-linux-gnu ... in +locally installed gems (Bundler::GemNotFound) +``` + +That is `bundle install` not having been run after step 4, stated as such. +`bundle exec` is also what keeps one habit for the whole checkout: the same +prefix runs the specs, the diagnostic subcommands and the Recipe, and typing +`automatic` instead would run the installed gem rather than the checkout. + +**What "is not installed" means in that message.** It is the framework +reporting that **this process could not `require` the library**, which is wider +than "no copy of the gem exists on this machine". A gem installed by `gem +install` but outside the checkout's bundle produces it; so does a group selected +but not installed, and so does an installation under a different Ruby. The gem +name and the plugin name in the message are what to act on; whether to act with +a group or with `gem install` is decided by where you are running from, not by +the message. + +To see what the bundle actually holds, ask the bundle rather than RubyGems: + +```sh +bundle config get with # which optional groups are selected +bundle show nokogiri # where the bundle's copy is, or an error +bundle show activerecord +bundle show sqlite3 +``` + +```sh +bundle exec ruby -rnokogiri -e 'puts Nokogiri::VERSION' +bundle exec ruby -ractive_record -e 'puts ActiveRecord::VERSION::STRING' +bundle exec ruby -rsqlite3 -e 'puts SQLite3::VERSION' +``` + +Those three `require` the libraries the way the plugins do — note +`active_record` for the `activerecord` gem — under the same bundle the Recipe +will run under. `gem list` answers a different question and is the one to +distrust here: it lists what RubyGems has, which in a checkout is neither what +the plugins will load nor what a missing-gem message is about. + +### All of the optional gems, or only the ones a Recipe names + +Two ways to select groups, for two purposes: + +```sh +bundle config set --local with plugins # all of them, at once +bundle config set --local with "html store" # what this Recipe needs +``` + +`plugins` is every gem in the first block of the `Gemfile` — `activerecord`, +`sqlite3`, `nokogiri`, `sanitize`, `feedbag` — and is meant for working **on** +the plugins: it brings their specs into the ordinary `bundle exec rake` run, +which is how those plugins are verified. It is the right setting for plugin +development and for a checkout you are developing in. + +Naming the groups is the right setting for **running** Recipes, and is what the +design of the split asks for: a gem is installed by the operator who uses the +plugin that needs it, and a checkout that runs this Recipe has no reason to +build `sanitize` or to hold `feedbag`. Start there; `plugins` is not a shortcut +for having read the table, and a checkout that has installed everything hides +the group a Recipe of yours will need on the next machine. + +Either way the setting is written to the checkout's own `.bundle/config`, which +is not committed and belongs to that checkout alone. That is what makes it hold: +`bundle install`, `bundle exec bin/automatic`, `bundle exec rake` and the +checkout's `bin/automatic` all read it, so the groups are selected once rather +than remembered at every command. Passing the environment variable instead — +`BUNDLE_WITH="html store" bundle install` — configures that one command and +leaves the next one without it, which produces exactly the confusing case above: +an installed group that the running process does not select. + +To return the checkout to the minimum: + +```sh +bundle config unset --local with +bundle install +``` + +The gems stay on the machine; what changes is that the bundle no longer includes +them, and the plugins that need them report them as missing again. + ## Your own plugins `~/.automatic/plugins` is on the loader's search path, **ahead of the @@ -523,8 +766,11 @@ not touch this directory. `PATH`. `gem environment` prints it as EXECUTABLE DIRECTORY. **`The gem is not installed. It is needed by ...`** — a plugin's optional -dependency is missing. The message names the gem, the plugin and the command to -install it, and the table above says the same thing. +dependency could not be required. The message names the gem, the plugin and the +command to install it, and the table above says the same thing. In a checkout it +also means a gem installed outside the bundle, or a group selected but not +installed; "Why `bundle exec`, and why `gem install` is not enough here" above +tells the cases apart. **`Automatic::NoPluginError: unknown plugin named ...`** — a Recipe names a plugin that does not ship. Check the spelling against diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index 2171e23..cc4343b 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -404,6 +404,12 @@ runtime dependencies; it goes in an optional group of the `Gemfile` and the operator who uses the plugin installs it. See [`POLICY.md`](POLICY.md) section 9. +A Recipe therefore needs the sum of what its plugins need, and each is loaded +when the pipeline reaches it, so a gem missing for the third plugin is reported +after the first two have run. [`DEPLOYMENT.md`](DEPLOYMENT.md) lists which +plugin needs which gem, and "Working out what a Recipe needs, in a checkout" +takes one Recipe through adding those up before running it. + Where a plugin has an optional capability that needs a heavier library — S3 support in `StoreFile`, for instance — the `require` goes inside the branch that uses it, so the plugin loads and its ordinary path works without that gem diff --git a/doc/QUICKSTART.md b/doc/QUICKSTART.md index 6b842bf..7a5aea7 100644 --- a/doc/QUICKSTART.md +++ b/doc/QUICKSTART.md @@ -136,10 +136,21 @@ bundle exec bin/automatic scaffold bundle exec bin/automatic -c ~/.automatic/config/example/feed2markdown.yml ``` -A checkout resolves gems through Bundler rather than through RubyGems, so step 5 -is done differently there: `bundle config set --local with store` and -`bundle install`, instead of `gem install activerecord sqlite3`. See -[`DEPLOYMENT.md`](DEPLOYMENT.md). +A checkout resolves gems through Bundler rather than through RubyGems, and two +things follow from that: + +- **Step 5 is done with a group, not with `gem install`.** `bundle config set + --local with store` and `bundle install`, instead of `gem install activerecord + sqlite3`. A gem installed by `gem install` is not in the bundle, so the + checkout does not see it however plainly `gem list` shows it. +- **A Recipe needs the groups of every plugin it names, added up.** The Recipe + above needs only `store`; one that also reads HTML pages needs `html` as well, + and selecting one of the two fails half way through the run rather than at the + start. + +[`DEPLOYMENT.md`](DEPLOYMENT.md) has the table of which plugin needs which +group, and "Working out what a Recipe needs, in a checkout" takes a three-plugin +Recipe from written to running that way, step by step. The Recipe is ordinary YAML. Change the feed URL, insert a supported Filter, or change the Markdown path without changing the framework. To write a small From e3b06b169d1d5895bb6bc31a8006570e50c3421c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 10:37:16 +0000 Subject: [PATCH 2/2] Make the Quick Start a first run that installs what its Recipe needs The old first run was chosen so that nothing had to be installed for it, which taught the one thing a second Recipe immediately un-teaches: almost every useful Recipe names a plugin with an optional gem, and working out which gems from which plugins is the step that stops a first run half way through. Teaching it in step 4 of the guide, on a Recipe that needs it, is better than leaving it to be discovered. The Recipe is CustomFeedWeb, StoreDigest and PublishMarkdown over four public index pages: no account, no credential and no service, but two optional gems, arrived at by reading the Recipe plugin by plugin against the table in doc/DEPLOYMENT.md. The checkout section is then the same step done with Bundler groups, and doc/DEPLOYMENT.md keeps the reasoning that belongs to a checkout. Consistency, since "the Quick Start needs no optional gem" was an assertion made in several places: README's summary of it, the note in PLUGINS.md 6.7 and in publish/markdown.rb explaining why PublishMarkdown carries its own HTML reduction, the spec that guards the shipped feed2markdown example, and RELEASING's smoke test, which tests an installed gem with no optional dependency and is described as that now. No behaviour changes, so no Updated:: line moves. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019fgz9HsShz5W9ytoVms5BH --- README.md | 23 ++- config/feed2markdown.yml | 2 +- doc/DEPLOYMENT.md | 32 ++-- doc/PLUGINS.md | 4 +- doc/QUICKSTART.md | 253 ++++++++++++++++++-------- doc/RELEASING.md | 6 +- plugins/publish/markdown.rb | 9 +- spec/config/feed2markdown_spec.rb | 15 +- spec/plugins/publish/markdown_spec.rb | 2 +- 9 files changed, 220 insertions(+), 126 deletions(-) diff --git a/README.md b/README.md index c7520fd..0d5ba5e 100644 --- a/README.md +++ b/README.md @@ -34,8 +34,9 @@ prints to the terminal instead, or downloads the images, or forwards them to Fluentd, or writes them to a database. Write your own plugin and it composes with all the others. -**[Follow the Quick Start](doc/QUICKSTART.md)** to install the gem, scaffold the -example, and produce Markdown from the public Ruby news feed. +**[Follow the Quick Start](doc/QUICKSTART.md)** to install the gem, write a +Recipe that watches four public index pages, install what that Recipe needs, and +produce Markdown from the articles they list. --- @@ -242,7 +243,9 @@ a Recipe taken step by step through choosing its groups are in ## 6. Quick start -The complete first-run guide is [`doc/QUICKSTART.md`](doc/QUICKSTART.md). +The complete first-run guide is [`doc/QUICKSTART.md`](doc/QUICKSTART.md): it +writes one Recipe that reads four public index pages, installs what that Recipe +needs, and leaves the new articles in a Markdown document. ```sh automatic scaffold @@ -253,13 +256,13 @@ automatic -c ~/.automatic/config/example/feed2markdown.yml `assets/`, and copies the example Recipes into `~/.automatic/config/example`. It never overwrites anything already there. -That Recipe fetches the public Ruby news feed and appends its items to -`~/.automatic/markdown/feeds.md`, using nothing but the framework and what -`gem install automatic` brought. Read the file, `grep` it, put it in a -repository, or hand it to whatever reads text next. `feed2console.yml` beside -it is the same pipeline printing to the terminal. Adding a store plugin, so -that a second run appends only what is new, is step 5 of the Quick Start and -the point at which the first optional gems are installed. +`feed2markdown.yml` is the shortest shipped Recipe: it fetches the public Ruby +news feed and appends its items to `~/.automatic/markdown/feeds.md`, using +nothing but the framework and what `gem install automatic` brought. +`feed2console.yml` beside it is the same pipeline printing to the terminal. Read +the file, `grep` it, put it in a repository, or hand it to whatever reads text +next. A Recipe that names a plugin with an optional gem installs that gem first, +which is step 4 of the Quick Start and the habit worth learning early. To check the framework without any network, write this instead: diff --git a/config/feed2markdown.yml b/config/feed2markdown.yml index 07246f4..1a58f44 100644 --- a/config/feed2markdown.yml +++ b/config/feed2markdown.yml @@ -22,7 +22,7 @@ # gem install activerecord sqlite3 # # See doc/DEPLOYMENT.md for the optional plugin dependencies, and -# doc/QUICKSTART.md for this Recipe step by step. +# doc/QUICKSTART.md for a first run taken step by step. # # To send the document to standard output instead, drop the PublishMarkdown # config and set the log level to none, so that only the document is written: diff --git a/doc/DEPLOYMENT.md b/doc/DEPLOYMENT.md index 8b369d2..1c34a57 100644 --- a/doc/DEPLOYMENT.md +++ b/doc/DEPLOYMENT.md @@ -34,8 +34,9 @@ nothing to stop. [`REQUIREMENTS.md`](REQUIREMENTS.md) section 20. - Optional gems for particular plugins, listed in the table under "Optional plugin dependencies" below. None is needed to install Automatic - Ruby, to run the Quick Start, or to run a Recipe that does not use the - plugin. + Ruby, or to run a Recipe that does not use the plugin; a Recipe that does + name one installs it as a step of its own, which is what the Quick Start's + step 4 is. - A build environment may be needed for one of those optional gems — `nokogiri` and `sqlite3` build a native extension where no binary package matches your platform. The framework's own dependencies are pure Ruby, so the normal @@ -508,9 +509,10 @@ plugins. Install it with `gem install activerecord`, ... The table says what a plugin needs. A Recipe needs the **sum** of what its plugins need, worked out before the first run rather than discovered one failed -run at a time, and in a checkout that sum is a list of groups. This Recipe — -index pages watched for new articles, de-duplicated by content, published as one -Markdown document — is the worked example: +run at a time, and in a checkout that sum is a list of groups. The Quick Start's +Recipe — index pages watched for new articles, de-duplicated by content, +published as one Markdown document — is the worked example, shortened here to +one page: ```yaml # ~/.automatic/config/web2markdown.yml @@ -520,18 +522,6 @@ plugins: retry: 2 interval: 2 sites: - # Python Insider. Its articles are - # https://blog.python.org/2026/08/python-3147-31315/ and the like. - - name: Python Insider - url: https://blog.python.org/ - link_selector: 'a[href]' - include: - - '^https://blog\.python\.org/20[0-9]{2}/[0-9]{2}/[^/]+/?$' - same_host: true - fetch_items: 20 - - # The Go Blog. Its articles are https://go.dev/blog/pkgsite-api - # and the like. Further sites are further entries of the same shape. - name: The Go Blog url: https://go.dev/blog/ link_selector: 'a[href]' @@ -553,6 +543,10 @@ plugins: mode: append ``` +[`QUICKSTART.md`](QUICKSTART.md) runs that Recipe end to end, with the sites it +watches and what each plugin does. What follows is the part a checkout does +differently: turning those three plugins into a bundle that can run them. + **1. List the plugins the Recipe names.** They are the `module` lines, in order: `CustomFeedWeb`, `StoreDigest`, `PublishMarkdown`. @@ -617,8 +611,8 @@ bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml ``` Run it twice. The second run publishes nothing, because `StoreDigest` has the -digest of everything the pages listed — the same check the Quick Start makes -with `StorePermalink`, and what makes the Recipe safe to put in `cron`. +digest of everything the page listed, which is what makes the Recipe safe to put +in `cron` and is worth checking before scheduling any Recipe with an effect. A store plugin's `db` is a **file name**, not a path: it is resolved under `~/.automatic/db`, or under the checkout's own `db/` where that directory does diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index cc4343b..42d9cb5 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -1380,8 +1380,8 @@ text; the item's own link is in the metadata list, where nothing loses it. **This plugin needs no gem of its own.** It uses `nokogiri` to reduce a body where `nokogiri` is installed, and reduces it with its own substitution where it -is not, so that the Quick Start runs on a plain `gem install automatic` and no -Recipe pays for an HTML parser it did not ask for. The two produce the same +is not, so that a Recipe ending here runs on a plain `gem install automatic` and +no Recipe pays for an HTML parser it did not ask for. The two produce the same document for the bodies a feed carries; a parser is simply better at markup that is badly malformed, which is the reason to install `nokogiri` if you publish from feeds that produce it. The specs hold both to the same output. diff --git a/doc/QUICKSTART.md b/doc/QUICKSTART.md index 7a5aea7..61e6207 100644 --- a/doc/QUICKSTART.md +++ b/doc/QUICKSTART.md @@ -1,9 +1,16 @@ # Quick Start -This guide takes public information through one short Automatic Ruby pipeline -and leaves it as Markdown. It needs no account, credential, paid service or -database server, and no gem beyond the four `gem install automatic` brings — -no HTML parser, no database, nothing to build. +This guide takes four public pages through one short Automatic Ruby pipeline and +leaves what they publish as one Markdown document. It needs no account, no +credential, no paid service and no database server. + +It does need two of the optional gems — because of the plugins the Recipe +names, not because of the framework — and installing exactly those is a step of +this guide rather than a footnote to it. Automatic Ruby installs what the +framework needs and leaves a plugin's gems to the operator who uses that +plugin, so "which plugins does this Recipe name, and what do they need" is a +question every Recipe asks. Skipping it is the usual way a first run stops half +way through. ## 1. Install @@ -15,6 +22,9 @@ gem install automatic automatic --version ``` +Working from a Git checkout instead is this same guide with one step done +differently; "From a source checkout" at the end is that step. + ## 2. Create the user directory ```sh @@ -25,90 +35,164 @@ This creates `~/.automatic` and copies the shipped examples to `~/.automatic/config/example`. Existing files and directories are not overwritten. -## 3. Run the example +## 3. Write the Recipe + +A Recipe is one job: the plugins it runs, in order, with their settings. This +one reads four public index pages as HTML and makes a feed of the articles each +lists — which is what to do for a page whose feed you do not have — keeps a +record of what it has already seen, and appends the rest to a Markdown +document. -The installed example is `config/feed2markdown.yml` in this repository: +Write it to `~/.automatic/config/web2markdown.yml`: ```yaml plugins: - - module: SubscriptionFeed + - module: CustomFeedWeb config: - feeds: - - https://www.ruby-lang.org/en/feeds/news.rss + retry: 2 + interval: 2 + sites: + # Articles: https://blog.python.org/2026/08/python-3147-31315/ + - name: Python Insider + url: https://blog.python.org/ + link_selector: 'a[href]' + include: + - '^https://blog\.python\.org/20[0-9]{2}/[0-9]{2}/[^/]+/?$' + same_host: true + fetch_items: 20 + + # Articles: https://blog.rust-lang.org/2026/08/04/enabling-polonius-alpha-on-nightly/ + - name: Rust Blog + url: https://blog.rust-lang.org/ + link_selector: 'a[href]' + include: + - '^https://blog\.rust-lang\.org/20[0-9]{2}/[0-9]{2}/[0-9]{2}/[^/]+/?$' + same_host: true + fetch_items: 20 + + # Articles: https://go.dev/blog/pkgsite-api + - name: The Go Blog + url: https://go.dev/blog/ + link_selector: 'a[href]' + include: + - '^https://go\.dev/blog/[^/]+$' + same_host: true + fetch_items: 20 + + # Alerts: https://www.jpcert.or.jp/at/2026/at260021.html + - name: JPCERT/CC Alerts + url: https://www.jpcert.or.jp/at/2026.html + link_selector: 'a[href]' + include: + - '^https://www\.jpcert\.or\.jp/at/20[0-9]{2}/at[0-9]+\.html$' + same_host: true + fetch_items: 20 + + - module: StoreDigest + config: + db: web2markdown.db + fields: + - title + - link - module: PublishMarkdown config: - file: ~/.automatic/markdown/feeds.md + file: ~/.automatic/markdown/web.md mode: append ``` -Run the scaffolded copy: +Three plugins, and each hands its result to the next: -```sh -automatic -c ~/.automatic/config/example/feed2markdown.yml -``` +- **`CustomFeedWeb`** fetches each page and makes a feed of the article links it + lists. `include` is what tells an article from a navigation link, `interval` + is the pause between requests, and one run makes one request per site — + nothing here follows a link or reads an article body. +- **`StoreDigest`** records a digest of each item and passes on only the items + whose digest it had not recorded already. It is what makes the Recipe safe to + run repeatedly. `db` is a **file name**, kept under `~/.automatic/db`. +- **`PublishMarkdown`** appends what is left to a plain-text document, creating + the directory if it is missing. Its `file`, unlike `db`, is a path. + +## 4. Install what the Recipe needs -`SubscriptionFeed` acquires the public Ruby news feed. `PublishMarkdown` -appends those items to a plain-text document, reducing the HTML in each item's -body to text as it goes. +Read the Recipe you have just written, plugin by plugin, and look each one up in +the table of optional plugin dependencies in [`DEPLOYMENT.md`](DEPLOYMENT.md). +That table gives, for these three: -## 4. Read the result +- **`CustomFeedWeb`** — `nokogiri`, which it reads the pages with. In a + checkout, the group `html`. +- **`StoreDigest`** — `activerecord` and `sqlite3`, which it keeps its record + in. In a checkout, the group `store`. +- **`PublishMarkdown`** — nothing of its own. ```sh -sed -n '1,80p' ~/.automatic/markdown/feeds.md +gem install nokogiri +gem install activerecord sqlite3 ``` -Each item is a level-2 heading followed by available metadata and a text body: +**Install what the whole Recipe needs, not what its first plugin needs.** A +plugin is loaded when the pipeline reaches it, so installing only `nokogiri` +would let `CustomFeedWeb` fetch its pages, hand its feeds on, and stop the run +where the next plugin is loaded: -```markdown -## Item title +```text +automatic: The `activerecord` gem is not installed. It is needed by the store +plugins. Install it with `gem install activerecord`, or in a source checkout add +its group to the bundle; see the optional plugin dependencies in +doc/DEPLOYMENT.md. (cannot load such file -- active_record) +``` -- Link: -- Date: 2026-08-14 10:00:00 +0000 +`PublishMarkdown` is the entry worth reading twice: it needs no gem of its own, +using an HTML parser where one is installed and its own substitution where none +is. A plugin's row in that table is the answer, not a guess from what the +plugin does. -Item body. -``` +`nokogiri` and `sqlite3` build a native extension where no binary package +matches your platform. Install a build environment only if one of them says it +needs one. -Run the Recipe again, and the same items are appended a second time: nothing in -this Recipe remembers what it has already published. The next step is what -fixes that. +## 5. Run it -## 5. Collect only what is new +```sh +automatic -c ~/.automatic/config/web2markdown.yml +``` -A store plugin records what has been published and passes on only what has not. -`StorePermalink` keeps that record in SQLite through ActiveRecord, and those two -gems are the store plugins' own dependencies rather than the framework's, so -they are installed when they are wanted: +A bare name is resolved inside `~/.automatic/config`, so `automatic -c +web2markdown.yml` is the same command. The log names each page as it is fetched, +each digest as it is saved, and the document as it is written. + +## 6. Read the result ```sh -gem install activerecord sqlite3 +sed -n '1,80p' ~/.automatic/markdown/web.md ``` -Then put the plugin between the two the Recipe already has: +Each item is a level-2 heading followed by the metadata the feed carries: -```yaml -plugins: - - module: SubscriptionFeed - config: - feeds: - - https://www.ruby-lang.org/en/feeds/news.rss +```markdown +## Extending the pkgsite API - - module: StorePermalink - config: - db: feed2markdown.db +- Link: +- Date: 2026-08-17 09:00:00 +0900 +``` - - module: PublishMarkdown - config: - file: ~/.automatic/markdown/feeds.md - mode: append +## 7. Run it again + +```sh +automatic -c ~/.automatic/config/web2markdown.yml ``` -Run it twice. The second run appends nothing, which is what makes the Recipe -safe to run from `cron` — and, in general, what to do before any plugin with an -effect. Other plugins have optional dependencies of their own, all listed in -[`DEPLOYMENT.md`](DEPLOYMENT.md). +The second run appends nothing: the pages still list the same articles, and +`StoreDigest` has the digest of every one of them. An article published between +the two runs is the one thing that would be added — which is what makes this +Recipe safe to put in `cron`, and what to check before any Recipe with an +effect. -## 6. Run it from cron +If instead everything is appended a second time, the store plugin is writing +somewhere other than where you think. The `Using Database:` line of the log +names the file it opened. + +## 8. Run it from cron Create the log directory once, then use the absolute path reported by `command -v automatic`: @@ -119,39 +203,50 @@ command -v automatic ``` ```crontab -0 * * * * /usr/local/bin/automatic -c $HOME/.automatic/config/example/feed2markdown.yml >> $HOME/.automatic/log/feed2markdown.log 2>&1 +0 * * * * /usr/local/bin/automatic -c $HOME/.automatic/config/web2markdown.yml >> $HOME/.automatic/log/web2markdown.log 2>&1 ``` Automatic Ruby runs once and exits; `cron` supplies the schedule. ## From a source checkout -The normal installation above remains the quickest way to use Automatic Ruby. -To try the development version or change the source, first follow -[README's checkout setup](../README.md#from-a-checkout). Then run the same flow -through the checkout's executable: +The normal installation above remains the quickest way to use Automatic Ruby. To +try the development version or change the source, first follow +[README's checkout setup](../README.md#from-a-checkout). Every `automatic` above +then becomes `bundle exec bin/automatic`, run from the checkout directory: ```sh +cd ~/automaticruby bundle exec bin/automatic scaffold -bundle exec bin/automatic -c ~/.automatic/config/example/feed2markdown.yml +bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml +``` + +Step 4 is the step that differs, because a checkout resolves its gems through +Bundler rather than through RubyGems. Each optional gem is in a Bundler group, +and the Recipe's groups — `html` and `store`, from step 4 — are selected +together and installed once: + +```sh +bundle config set --local with "html store" +bundle install ``` -A checkout resolves gems through Bundler rather than through RubyGems, and two -things follow from that: - -- **Step 5 is done with a group, not with `gem install`.** `bundle config set - --local with store` and `bundle install`, instead of `gem install activerecord - sqlite3`. A gem installed by `gem install` is not in the bundle, so the - checkout does not see it however plainly `gem list` shows it. -- **A Recipe needs the groups of every plugin it names, added up.** The Recipe - above needs only `store`; one that also reads HTML pages needs `html` as well, - and selecting one of the two fails half way through the run rather than at the - start. - -[`DEPLOYMENT.md`](DEPLOYMENT.md) has the table of which plugin needs which -group, and "Working out what a Recipe needs, in a checkout" takes a three-plugin -Recipe from written to running that way, step by step. - -The Recipe is ordinary YAML. Change the feed URL, insert a supported Filter, or -change the Markdown path without changing the framework. To write a small -plugin, continue with [`PLUGIN_DEVELOPMENT.md`](PLUGIN_DEVELOPMENT.md). +`gem install nokogiri` does **not** work here: the gem installs, and the +checkout still reports it as missing, because it is not in the bundle. That +difference, the commands that show what the bundle holds, and the same three +plugins taken step by step through choosing their groups are in +[`DEPLOYMENT.md`](DEPLOYMENT.md) under "Working out what a Recipe needs, in a +checkout". + +## Next + +The Recipe is ordinary YAML. Add a site, change the Markdown path, or put +`PublishConsoleLink` at the end to see what the pipeline holds without writing +anything. The shipped `feed2markdown.yml` and `feed2console.yml` in +`~/.automatic/config/example` are the same shape over an ordinary feed, for a +site that publishes one. + +What each plugin does is [`PLUGINS.md`](PLUGINS.md) section 6; installing, +scheduling and operating a Recipe is [`DEPLOYMENT.md`](DEPLOYMENT.md); writing a +small plugin of your own is +[`PLUGIN_DEVELOPMENT.md`](PLUGIN_DEVELOPMENT.md). diff --git a/doc/RELEASING.md b/doc/RELEASING.md index a9ecfe9..87b7fc1 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -231,8 +231,8 @@ env -u RUBYLIB GEM_HOME="$gem_home" GEM_PATH="$gem_home" \ "$gem_home/bin/automatic" --help ``` -Confirm the installed version equals `VERSION`. Then run the offline Quick -Start shape with a temporary home and a minimal Recipe: +Confirm the installed version equals `VERSION`. Then run a Recipe that needs no +network and no optional gem, with a temporary home: ```sh smoke_home=$(mktemp -d) @@ -375,7 +375,7 @@ security behavior can change independently of this repository: - [ ] `bundle exec rake` passes and the gem builds. - [ ] Package contents and metadata have been inspected. - [ ] GPLv3/LGPLv3, Ruby version and dependencies are correct. -- [ ] The isolated local install and Quick Start smoke test pass. +- [ ] The isolated local install and offline smoke test pass. - [ ] The release commit and immutable `vX.Y.Z` tag identify the built source. - [ ] Publication has explicit approval and `gem push` succeeds. - [ ] RubyGems.org metadata and a clean remote install are verified. diff --git a/plugins/publish/markdown.rb b/plugins/publish/markdown.rb index 22562c8..5d5f86c 100644 --- a/plugins/publish/markdown.rb +++ b/plugins/publish/markdown.rb @@ -171,10 +171,11 @@ def url?(text) # the tags themselves are dropped. # # A parser does it where one is installed, and the substitution below does - # it where none is. That is what keeps this plugin -- the one the Quick - # Start publishes with -- runnable on a plain `gem install automatic`: - # nokogiri is an optional dependency, and reducing a feed body to text is - # not a good enough reason to make everyone install a native extension. + # it where none is. That is what keeps this plugin -- the one a Recipe ends + # with when the result is meant to be read -- runnable on a plain `gem + # install automatic`: nokogiri is an optional dependency, and reducing a + # feed body to text is not a good enough reason to make everyone install a + # native extension. # The two agree on what a body is reduced to; a parser is simply better at # markup that is malformed. See doc/PLUGINS.md section 6.7. def html_to_text(html) diff --git a/spec/config/feed2markdown_spec.rb b/spec/config/feed2markdown_spec.rb index 178b4b8..434a700 100644 --- a/spec/config/feed2markdown_spec.rb +++ b/spec/config/feed2markdown_spec.rb @@ -4,19 +4,20 @@ require 'yaml' -RSpec.describe 'the Quick Start Recipe' do +RSpec.describe 'the feed2markdown example Recipe' do let(:path) { File.join(APP_ROOT, 'config', 'feed2markdown.yml') } let(:recipe) { YAML.safe_load(File.read(path)) } - it 'uses the short supported pipeline documented by the Quick Start' do + it 'uses the short supported pipeline its header documents' do modules = recipe.fetch('plugins').map { |plugin| plugin.fetch('module') } expect(modules).to eq %w[SubscriptionFeed PublishMarkdown] end - # The Quick Start is what a plain `gem install automatic` can run, so the - # Recipe it ships names no plugin that needs an optional gem. The store - # plugins, which do, are the documented next step rather than the first one. - # See doc/POLICY.md section 9.1. + # This is the shipped Recipe a plain `gem install automatic` can run with + # nothing added, which is what makes it the one to reach for first, so it + # names no plugin that needs an optional gem. The store plugins, which do, + # are the documented next step rather than the first one. See + # doc/POLICY.md section 9.1. it 'names no plugin that needs an optional dependency' do modules = recipe.fetch('plugins').map { |plugin| plugin.fetch('module') } expect(modules).not_to include('StorePermalink', 'StoreFullText') @@ -36,7 +37,7 @@ end end - it 'writes the Markdown file described by the Quick Start' do + it 'writes the Markdown file its header describes' do publisher = recipe.fetch('plugins').last.fetch('config') expect(publisher).to include( 'file' => '~/.automatic/markdown/feeds.md', diff --git a/spec/plugins/publish/markdown_spec.rb b/spec/plugins/publish/markdown_spec.rb index 4bd2049..de79285 100644 --- a/spec/plugins/publish/markdown_spec.rb +++ b/spec/plugins/publish/markdown_spec.rb @@ -120,7 +120,7 @@ def publish(config, pipeline) # nokogiri is an optional dependency: the plugin uses it where it is # installed and reduces the body itself where it is not, so that a plain - # `gem install automatic` can run the Quick Start. See doc/PLUGINS.md + # `gem install automatic` can run a Recipe ending here. See doc/PLUGINS.md # section 6.7. The default suite runs the substitute, this run of it is # explicit, and where the optional gem is installed the last example holds # the two to the same answer.