diff --git a/README.md b/README.md index 0851ab5..0d5ba5e 100644 --- a/README.md +++ b/README.md @@ -34,8 +34,9 @@ prints to the terminal instead, or downloads the images, or forwards them to Fluentd, or writes them to a database. Write your own plugin and it composes with all the others. -**[Follow the Quick Start](doc/QUICKSTART.md)** to install the gem, scaffold the -example, and produce Markdown from the public Ruby news feed. +**[Follow the Quick Start](doc/QUICKSTART.md)** to install the gem, write a +Recipe that watches four public index pages, install what that Recipe needs, and +produce Markdown from the articles they list. --- @@ -233,14 +234,18 @@ for. If the `bundle` command is unavailable, install Bundler first with `gem install bundler`. In a checkout, every `automatic` below becomes `bundle exec bin/automatic`, and -`bundle exec` sees only the bundle — so a plugin's gem is added with a group -rather than with `gem install`. The group names, and which plugin needs which -gem, are in [`doc/DEPLOYMENT.md`](doc/DEPLOYMENT.md); what each plugin does is -in [`doc/PLUGINS.md`](doc/PLUGINS.md). +the checkout resolves its gems through the bundle — so a plugin's gem is added +with a group rather than with `gem install`, and a Recipe using plugins from two +groups selects both at once. The group names, which plugin needs which gem, and +a Recipe taken step by step through choosing its groups are in +[`doc/DEPLOYMENT.md`](doc/DEPLOYMENT.md); what each plugin does is in +[`doc/PLUGINS.md`](doc/PLUGINS.md). ## 6. Quick start -The complete first-run guide is [`doc/QUICKSTART.md`](doc/QUICKSTART.md). +The complete first-run guide is [`doc/QUICKSTART.md`](doc/QUICKSTART.md): it +writes one Recipe that reads four public index pages, installs what that Recipe +needs, and leaves the new articles in a Markdown document. ```sh automatic scaffold @@ -251,13 +256,13 @@ automatic -c ~/.automatic/config/example/feed2markdown.yml `assets/`, and copies the example Recipes into `~/.automatic/config/example`. It never overwrites anything already there. -That Recipe fetches the public Ruby news feed and appends its items to -`~/.automatic/markdown/feeds.md`, using nothing but the framework and what -`gem install automatic` brought. Read the file, `grep` it, put it in a -repository, or hand it to whatever reads text next. `feed2console.yml` beside -it is the same pipeline printing to the terminal. Adding a store plugin, so -that a second run appends only what is new, is step 5 of the Quick Start and -the point at which the first optional gems are installed. +`feed2markdown.yml` is the shortest shipped Recipe: it fetches the public Ruby +news feed and appends its items to `~/.automatic/markdown/feeds.md`, using +nothing but the framework and what `gem install automatic` brought. +`feed2console.yml` beside it is the same pipeline printing to the terminal. Read +the file, `grep` it, put it in a repository, or hand it to whatever reads text +next. A Recipe that names a plugin with an optional gem installs that gem first, +which is step 4 of the Quick Start and the habit worth learning early. To check the framework without any network, write this instead: diff --git a/config/feed2markdown.yml b/config/feed2markdown.yml index 07246f4..1a58f44 100644 --- a/config/feed2markdown.yml +++ b/config/feed2markdown.yml @@ -22,7 +22,7 @@ # gem install activerecord sqlite3 # # See doc/DEPLOYMENT.md for the optional plugin dependencies, and -# doc/QUICKSTART.md for this Recipe step by step. +# doc/QUICKSTART.md for a first run taken step by step. # # To send the document to standard output instead, drop the PublishMarkdown # config and set the log level to none, so that only the document is written: diff --git a/doc/DEPLOYMENT.md b/doc/DEPLOYMENT.md index 7b9931d..1c34a57 100644 --- a/doc/DEPLOYMENT.md +++ b/doc/DEPLOYMENT.md @@ -34,8 +34,9 @@ nothing to stop. [`REQUIREMENTS.md`](REQUIREMENTS.md) section 20. - Optional gems for particular plugins, listed in the table under "Optional plugin dependencies" below. None is needed to install Automatic - Ruby, to run the Quick Start, or to run a Recipe that does not use the - plugin. + Ruby, or to run a Recipe that does not use the plugin; a Recipe that does + name one installs it as a step of its own, which is what the Quick Start's + step 4 is. - A build environment may be needed for one of those optional gems — `nokogiri` and `sqlite3` build a native extension where no binary package matches your platform. The framework's own dependencies are pure Ruby, so the normal @@ -114,13 +115,15 @@ bundle install ``` Several at once are space-separated: `bundle config set --local with "store -html"`. The group names are in the table below. +html"`, and a Recipe using plugins from two groups needs exactly that. The group +names are in the table below, and "Working out what a Recipe needs, in a +checkout" takes one Recipe through choosing them. -In a checkout, `gem install ` on its own is **not** enough: `bundle exec` -puts only the bundle on the load path, so a gem the `Gemfile` does not mention -is not visible to it. Use the group, which is why the groups exist. Outside a -checkout — the installed gem, run as `automatic` — there is no bundle and `gem -install ` is exactly right. +In a checkout, `gem install ` on its own is **not** enough: the checkout +resolves its gems through the bundle, so a gem the `Gemfile` does not mention is +not visible to it however plainly `gem list` shows it. Use the group, which is +why the groups exist. Outside a checkout — the installed gem, run as `automatic` +— there is no bundle and `gem install ` is exactly right. Everything below that says `automatic` becomes `bundle exec bin/automatic` in a checkout. @@ -502,6 +505,240 @@ automatic: The `activerecord` gem is not installed. It is needed by the store plugins. Install it with `gem install activerecord`, ... ``` +### Working out what a Recipe needs, in a checkout + +The table says what a plugin needs. A Recipe needs the **sum** of what its +plugins need, worked out before the first run rather than discovered one failed +run at a time, and in a checkout that sum is a list of groups. The Quick Start's +Recipe — index pages watched for new articles, de-duplicated by content, +published as one Markdown document — is the worked example, shortened here to +one page: + +```yaml +# ~/.automatic/config/web2markdown.yml +plugins: + - module: CustomFeedWeb + config: + retry: 2 + interval: 2 + sites: + - name: The Go Blog + url: https://go.dev/blog/ + link_selector: 'a[href]' + include: + - '^https://go\.dev/blog/[^/]+$' + same_host: true + fetch_items: 20 + + - module: StoreDigest + config: + db: web2markdown.db + fields: + - title + - link + + - module: PublishMarkdown + config: + file: ~/.automatic/markdown/web.md + mode: append +``` + +[`QUICKSTART.md`](QUICKSTART.md) runs that Recipe end to end, with the sites it +watches and what each plugin does. What follows is the part a checkout does +differently: turning those three plugins into a bundle that can run them. + +**1. List the plugins the Recipe names.** They are the `module` lines, in order: +`CustomFeedWeb`, `StoreDigest`, `PublishMarkdown`. + +**2. Look each one up in the table above.** What that lookup yields here, and +the whole of what this Recipe's dependencies are: + +| Plugin | Needs | Checkout group | +| --- | --- | --- | +| `CustomFeedWeb` | `nokogiri`, to read the index pages | `html` | +| `StoreDigest` | `activerecord`, `sqlite3` | `store` | +| `PublishMarkdown` | nothing of its own | — | + +`PublishMarkdown` is the row worth reading twice. It needs no gem: it reduces an +HTML body with `nokogiri` where one is installed and with its own substitution +where none is, so it neither adds a group here nor fails without one. A plugin's +row in the table is the answer, not a guess from what the plugin does. + +**3. Add the groups up.** This Recipe needs `html` **and** `store`. Getting this +step half right fails half way through the run, because a plugin is loaded when +the pipeline reaches it and not before: with `html` alone, `CustomFeedWeb` +loads, fetches its pages and hands its feeds on, and the run then stops where +the second plugin is loaded. + +```text +automatic: The `activerecord` gem is not installed. It is needed by the store +plugins. Install it with `gem install activerecord`, or in a source checkout add +its group to the bundle; see the optional plugin dependencies in +doc/DEPLOYMENT.md. (cannot load such file -- active_record) +``` + +The first plugin having worked is not the setup being finished. Read the whole +Recipe, then install once. + +**4. Select the groups in the checkout.** + +```sh +cd ~/automaticruby +bundle config set --local with "html store" +``` + +**5. Install them.** + +```sh +bundle install +``` + +**6. Check what was selected.** This prints the setting Bundler will use, and +where it came from: + +```sh +bundle config get with +``` + +```text +Set for your local app (/home/you/automaticruby/.bundle/config): [:html, :store] +``` + +**7. Run the Recipe.** + +```sh +bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml +``` + +Run it twice. The second run publishes nothing, because `StoreDigest` has the +digest of everything the page listed, which is what makes the Recipe safe to put +in `cron` and is worth checking before scheduling any Recipe with an effect. + +A store plugin's `db` is a **file name**, not a path: it is resolved under +`~/.automatic/db`, or under the checkout's own `db/` where that directory does +not exist yet, so a checkout run before `scaffold` keeps its database inside the +checkout. The `Using Database:` line of an `info`-level log names the file that +was opened, which is the way to check. `PublishMarkdown`'s `file` is the +opposite — a path, with `~` expanded — and the Recipe above uses each as it is +meant. + +Adding a plugin to a Recipe later starts this over at step 1: a `FilterSanitize` +added to the Recipe above brings `sanitize` with it, and the `with` setting has +to name `sanitize` as well as `html` and `store`. + +### Why `bundle exec`, and why `gem install` is not enough here + +A checkout resolves its gems through Bundler whether or not you ask it to. +`lib/automatic/environment.rb` treats a `Gemfile` beside `lib/` as "this is a +source checkout" and requires `bundler/setup` before anything else loads, so +`bin/automatic` run from a checkout sees the bundle and nothing outside it. That +is what makes `gem install` the wrong tool there: the gem installs, `gem list` +prints it, plain `ruby -rnokogiri -e ''` loads it — and the checkout still +reports it as missing, because it is not in the bundle. + +```sh +gem install nokogiri +gem list nokogiri # prints the gem that was just installed +./bin/automatic -c ~/.automatic/config/web2markdown.yml +# automatic: The `nokogiri` gem is not installed. It is needed by CustomFeedWeb... +``` + +Use the group, and then run through `bundle exec`, from the checkout directory: + +```sh +cd ~/automaticruby +bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml +``` + +`bundle exec` matters for what it does when the bundle is **not** in the state +you think it is. The `require 'bundler/setup'` above is deliberately forgiving: +where Bundler is absent, or where a group has been selected but not yet +installed, it is rescued and the program falls back to whatever RubyGems can +activate — quietly, and against gems the `Gemfile.lock` never resolved. +Running the same command through `bundle exec` says so instead: + +```text +bundler: failed to load command: bin/automatic (bin/automatic) +Could not find activerecord-8.1.3.1, sqlite3-2.9.6-x86_64-linux-gnu ... in +locally installed gems (Bundler::GemNotFound) +``` + +That is `bundle install` not having been run after step 4, stated as such. +`bundle exec` is also what keeps one habit for the whole checkout: the same +prefix runs the specs, the diagnostic subcommands and the Recipe, and typing +`automatic` instead would run the installed gem rather than the checkout. + +**What "is not installed" means in that message.** It is the framework +reporting that **this process could not `require` the library**, which is wider +than "no copy of the gem exists on this machine". A gem installed by `gem +install` but outside the checkout's bundle produces it; so does a group selected +but not installed, and so does an installation under a different Ruby. The gem +name and the plugin name in the message are what to act on; whether to act with +a group or with `gem install` is decided by where you are running from, not by +the message. + +To see what the bundle actually holds, ask the bundle rather than RubyGems: + +```sh +bundle config get with # which optional groups are selected +bundle show nokogiri # where the bundle's copy is, or an error +bundle show activerecord +bundle show sqlite3 +``` + +```sh +bundle exec ruby -rnokogiri -e 'puts Nokogiri::VERSION' +bundle exec ruby -ractive_record -e 'puts ActiveRecord::VERSION::STRING' +bundle exec ruby -rsqlite3 -e 'puts SQLite3::VERSION' +``` + +Those three `require` the libraries the way the plugins do — note +`active_record` for the `activerecord` gem — under the same bundle the Recipe +will run under. `gem list` answers a different question and is the one to +distrust here: it lists what RubyGems has, which in a checkout is neither what +the plugins will load nor what a missing-gem message is about. + +### All of the optional gems, or only the ones a Recipe names + +Two ways to select groups, for two purposes: + +```sh +bundle config set --local with plugins # all of them, at once +bundle config set --local with "html store" # what this Recipe needs +``` + +`plugins` is every gem in the first block of the `Gemfile` — `activerecord`, +`sqlite3`, `nokogiri`, `sanitize`, `feedbag` — and is meant for working **on** +the plugins: it brings their specs into the ordinary `bundle exec rake` run, +which is how those plugins are verified. It is the right setting for plugin +development and for a checkout you are developing in. + +Naming the groups is the right setting for **running** Recipes, and is what the +design of the split asks for: a gem is installed by the operator who uses the +plugin that needs it, and a checkout that runs this Recipe has no reason to +build `sanitize` or to hold `feedbag`. Start there; `plugins` is not a shortcut +for having read the table, and a checkout that has installed everything hides +the group a Recipe of yours will need on the next machine. + +Either way the setting is written to the checkout's own `.bundle/config`, which +is not committed and belongs to that checkout alone. That is what makes it hold: +`bundle install`, `bundle exec bin/automatic`, `bundle exec rake` and the +checkout's `bin/automatic` all read it, so the groups are selected once rather +than remembered at every command. Passing the environment variable instead — +`BUNDLE_WITH="html store" bundle install` — configures that one command and +leaves the next one without it, which produces exactly the confusing case above: +an installed group that the running process does not select. + +To return the checkout to the minimum: + +```sh +bundle config unset --local with +bundle install +``` + +The gems stay on the machine; what changes is that the bundle no longer includes +them, and the plugins that need them report them as missing again. + ## Your own plugins `~/.automatic/plugins` is on the loader's search path, **ahead of the @@ -523,8 +760,11 @@ not touch this directory. `PATH`. `gem environment` prints it as EXECUTABLE DIRECTORY. **`The gem is not installed. It is needed by ...`** — a plugin's optional -dependency is missing. The message names the gem, the plugin and the command to -install it, and the table above says the same thing. +dependency could not be required. The message names the gem, the plugin and the +command to install it, and the table above says the same thing. In a checkout it +also means a gem installed outside the bundle, or a group selected but not +installed; "Why `bundle exec`, and why `gem install` is not enough here" above +tells the cases apart. **`Automatic::NoPluginError: unknown plugin named ...`** — a Recipe names a plugin that does not ship. Check the spelling against diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index 2171e23..42d9cb5 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -404,6 +404,12 @@ runtime dependencies; it goes in an optional group of the `Gemfile` and the operator who uses the plugin installs it. See [`POLICY.md`](POLICY.md) section 9. +A Recipe therefore needs the sum of what its plugins need, and each is loaded +when the pipeline reaches it, so a gem missing for the third plugin is reported +after the first two have run. [`DEPLOYMENT.md`](DEPLOYMENT.md) lists which +plugin needs which gem, and "Working out what a Recipe needs, in a checkout" +takes one Recipe through adding those up before running it. + Where a plugin has an optional capability that needs a heavier library — S3 support in `StoreFile`, for instance — the `require` goes inside the branch that uses it, so the plugin loads and its ordinary path works without that gem @@ -1374,8 +1380,8 @@ text; the item's own link is in the metadata list, where nothing loses it. **This plugin needs no gem of its own.** It uses `nokogiri` to reduce a body where `nokogiri` is installed, and reduces it with its own substitution where it -is not, so that the Quick Start runs on a plain `gem install automatic` and no -Recipe pays for an HTML parser it did not ask for. The two produce the same +is not, so that a Recipe ending here runs on a plain `gem install automatic` and +no Recipe pays for an HTML parser it did not ask for. The two produce the same document for the bodies a feed carries; a parser is simply better at markup that is badly malformed, which is the reason to install `nokogiri` if you publish from feeds that produce it. The specs hold both to the same output. diff --git a/doc/QUICKSTART.md b/doc/QUICKSTART.md index 6b842bf..61e6207 100644 --- a/doc/QUICKSTART.md +++ b/doc/QUICKSTART.md @@ -1,9 +1,16 @@ # Quick Start -This guide takes public information through one short Automatic Ruby pipeline -and leaves it as Markdown. It needs no account, credential, paid service or -database server, and no gem beyond the four `gem install automatic` brings — -no HTML parser, no database, nothing to build. +This guide takes four public pages through one short Automatic Ruby pipeline and +leaves what they publish as one Markdown document. It needs no account, no +credential, no paid service and no database server. + +It does need two of the optional gems — because of the plugins the Recipe +names, not because of the framework — and installing exactly those is a step of +this guide rather than a footnote to it. Automatic Ruby installs what the +framework needs and leaves a plugin's gems to the operator who uses that +plugin, so "which plugins does this Recipe name, and what do they need" is a +question every Recipe asks. Skipping it is the usual way a first run stops half +way through. ## 1. Install @@ -15,6 +22,9 @@ gem install automatic automatic --version ``` +Working from a Git checkout instead is this same guide with one step done +differently; "From a source checkout" at the end is that step. + ## 2. Create the user directory ```sh @@ -25,90 +35,164 @@ This creates `~/.automatic` and copies the shipped examples to `~/.automatic/config/example`. Existing files and directories are not overwritten. -## 3. Run the example +## 3. Write the Recipe + +A Recipe is one job: the plugins it runs, in order, with their settings. This +one reads four public index pages as HTML and makes a feed of the articles each +lists — which is what to do for a page whose feed you do not have — keeps a +record of what it has already seen, and appends the rest to a Markdown +document. -The installed example is `config/feed2markdown.yml` in this repository: +Write it to `~/.automatic/config/web2markdown.yml`: ```yaml plugins: - - module: SubscriptionFeed + - module: CustomFeedWeb config: - feeds: - - https://www.ruby-lang.org/en/feeds/news.rss + retry: 2 + interval: 2 + sites: + # Articles: https://blog.python.org/2026/08/python-3147-31315/ + - name: Python Insider + url: https://blog.python.org/ + link_selector: 'a[href]' + include: + - '^https://blog\.python\.org/20[0-9]{2}/[0-9]{2}/[^/]+/?$' + same_host: true + fetch_items: 20 + + # Articles: https://blog.rust-lang.org/2026/08/04/enabling-polonius-alpha-on-nightly/ + - name: Rust Blog + url: https://blog.rust-lang.org/ + link_selector: 'a[href]' + include: + - '^https://blog\.rust-lang\.org/20[0-9]{2}/[0-9]{2}/[0-9]{2}/[^/]+/?$' + same_host: true + fetch_items: 20 + + # Articles: https://go.dev/blog/pkgsite-api + - name: The Go Blog + url: https://go.dev/blog/ + link_selector: 'a[href]' + include: + - '^https://go\.dev/blog/[^/]+$' + same_host: true + fetch_items: 20 + + # Alerts: https://www.jpcert.or.jp/at/2026/at260021.html + - name: JPCERT/CC Alerts + url: https://www.jpcert.or.jp/at/2026.html + link_selector: 'a[href]' + include: + - '^https://www\.jpcert\.or\.jp/at/20[0-9]{2}/at[0-9]+\.html$' + same_host: true + fetch_items: 20 + + - module: StoreDigest + config: + db: web2markdown.db + fields: + - title + - link - module: PublishMarkdown config: - file: ~/.automatic/markdown/feeds.md + file: ~/.automatic/markdown/web.md mode: append ``` -Run the scaffolded copy: +Three plugins, and each hands its result to the next: -```sh -automatic -c ~/.automatic/config/example/feed2markdown.yml -``` +- **`CustomFeedWeb`** fetches each page and makes a feed of the article links it + lists. `include` is what tells an article from a navigation link, `interval` + is the pause between requests, and one run makes one request per site — + nothing here follows a link or reads an article body. +- **`StoreDigest`** records a digest of each item and passes on only the items + whose digest it had not recorded already. It is what makes the Recipe safe to + run repeatedly. `db` is a **file name**, kept under `~/.automatic/db`. +- **`PublishMarkdown`** appends what is left to a plain-text document, creating + the directory if it is missing. Its `file`, unlike `db`, is a path. + +## 4. Install what the Recipe needs -`SubscriptionFeed` acquires the public Ruby news feed. `PublishMarkdown` -appends those items to a plain-text document, reducing the HTML in each item's -body to text as it goes. +Read the Recipe you have just written, plugin by plugin, and look each one up in +the table of optional plugin dependencies in [`DEPLOYMENT.md`](DEPLOYMENT.md). +That table gives, for these three: -## 4. Read the result +- **`CustomFeedWeb`** — `nokogiri`, which it reads the pages with. In a + checkout, the group `html`. +- **`StoreDigest`** — `activerecord` and `sqlite3`, which it keeps its record + in. In a checkout, the group `store`. +- **`PublishMarkdown`** — nothing of its own. ```sh -sed -n '1,80p' ~/.automatic/markdown/feeds.md +gem install nokogiri +gem install activerecord sqlite3 ``` -Each item is a level-2 heading followed by available metadata and a text body: +**Install what the whole Recipe needs, not what its first plugin needs.** A +plugin is loaded when the pipeline reaches it, so installing only `nokogiri` +would let `CustomFeedWeb` fetch its pages, hand its feeds on, and stop the run +where the next plugin is loaded: -```markdown -## Item title +```text +automatic: The `activerecord` gem is not installed. It is needed by the store +plugins. Install it with `gem install activerecord`, or in a source checkout add +its group to the bundle; see the optional plugin dependencies in +doc/DEPLOYMENT.md. (cannot load such file -- active_record) +``` -- Link: -- Date: 2026-08-14 10:00:00 +0000 +`PublishMarkdown` is the entry worth reading twice: it needs no gem of its own, +using an HTML parser where one is installed and its own substitution where none +is. A plugin's row in that table is the answer, not a guess from what the +plugin does. -Item body. -``` +`nokogiri` and `sqlite3` build a native extension where no binary package +matches your platform. Install a build environment only if one of them says it +needs one. -Run the Recipe again, and the same items are appended a second time: nothing in -this Recipe remembers what it has already published. The next step is what -fixes that. +## 5. Run it -## 5. Collect only what is new +```sh +automatic -c ~/.automatic/config/web2markdown.yml +``` -A store plugin records what has been published and passes on only what has not. -`StorePermalink` keeps that record in SQLite through ActiveRecord, and those two -gems are the store plugins' own dependencies rather than the framework's, so -they are installed when they are wanted: +A bare name is resolved inside `~/.automatic/config`, so `automatic -c +web2markdown.yml` is the same command. The log names each page as it is fetched, +each digest as it is saved, and the document as it is written. + +## 6. Read the result ```sh -gem install activerecord sqlite3 +sed -n '1,80p' ~/.automatic/markdown/web.md ``` -Then put the plugin between the two the Recipe already has: +Each item is a level-2 heading followed by the metadata the feed carries: -```yaml -plugins: - - module: SubscriptionFeed - config: - feeds: - - https://www.ruby-lang.org/en/feeds/news.rss +```markdown +## Extending the pkgsite API - - module: StorePermalink - config: - db: feed2markdown.db +- Link: +- Date: 2026-08-17 09:00:00 +0900 +``` - - module: PublishMarkdown - config: - file: ~/.automatic/markdown/feeds.md - mode: append +## 7. Run it again + +```sh +automatic -c ~/.automatic/config/web2markdown.yml ``` -Run it twice. The second run appends nothing, which is what makes the Recipe -safe to run from `cron` — and, in general, what to do before any plugin with an -effect. Other plugins have optional dependencies of their own, all listed in -[`DEPLOYMENT.md`](DEPLOYMENT.md). +The second run appends nothing: the pages still list the same articles, and +`StoreDigest` has the digest of every one of them. An article published between +the two runs is the one thing that would be added — which is what makes this +Recipe safe to put in `cron`, and what to check before any Recipe with an +effect. + +If instead everything is appended a second time, the store plugin is writing +somewhere other than where you think. The `Using Database:` line of the log +names the file it opened. -## 6. Run it from cron +## 8. Run it from cron Create the log directory once, then use the absolute path reported by `command -v automatic`: @@ -119,28 +203,50 @@ command -v automatic ``` ```crontab -0 * * * * /usr/local/bin/automatic -c $HOME/.automatic/config/example/feed2markdown.yml >> $HOME/.automatic/log/feed2markdown.log 2>&1 +0 * * * * /usr/local/bin/automatic -c $HOME/.automatic/config/web2markdown.yml >> $HOME/.automatic/log/web2markdown.log 2>&1 ``` Automatic Ruby runs once and exits; `cron` supplies the schedule. ## From a source checkout -The normal installation above remains the quickest way to use Automatic Ruby. -To try the development version or change the source, first follow -[README's checkout setup](../README.md#from-a-checkout). Then run the same flow -through the checkout's executable: +The normal installation above remains the quickest way to use Automatic Ruby. To +try the development version or change the source, first follow +[README's checkout setup](../README.md#from-a-checkout). Every `automatic` above +then becomes `bundle exec bin/automatic`, run from the checkout directory: ```sh +cd ~/automaticruby bundle exec bin/automatic scaffold -bundle exec bin/automatic -c ~/.automatic/config/example/feed2markdown.yml +bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml ``` -A checkout resolves gems through Bundler rather than through RubyGems, so step 5 -is done differently there: `bundle config set --local with store` and -`bundle install`, instead of `gem install activerecord sqlite3`. See -[`DEPLOYMENT.md`](DEPLOYMENT.md). +Step 4 is the step that differs, because a checkout resolves its gems through +Bundler rather than through RubyGems. Each optional gem is in a Bundler group, +and the Recipe's groups — `html` and `store`, from step 4 — are selected +together and installed once: + +```sh +bundle config set --local with "html store" +bundle install +``` -The Recipe is ordinary YAML. Change the feed URL, insert a supported Filter, or -change the Markdown path without changing the framework. To write a small -plugin, continue with [`PLUGIN_DEVELOPMENT.md`](PLUGIN_DEVELOPMENT.md). +`gem install nokogiri` does **not** work here: the gem installs, and the +checkout still reports it as missing, because it is not in the bundle. That +difference, the commands that show what the bundle holds, and the same three +plugins taken step by step through choosing their groups are in +[`DEPLOYMENT.md`](DEPLOYMENT.md) under "Working out what a Recipe needs, in a +checkout". + +## Next + +The Recipe is ordinary YAML. Add a site, change the Markdown path, or put +`PublishConsoleLink` at the end to see what the pipeline holds without writing +anything. The shipped `feed2markdown.yml` and `feed2console.yml` in +`~/.automatic/config/example` are the same shape over an ordinary feed, for a +site that publishes one. + +What each plugin does is [`PLUGINS.md`](PLUGINS.md) section 6; installing, +scheduling and operating a Recipe is [`DEPLOYMENT.md`](DEPLOYMENT.md); writing a +small plugin of your own is +[`PLUGIN_DEVELOPMENT.md`](PLUGIN_DEVELOPMENT.md). diff --git a/doc/RELEASING.md b/doc/RELEASING.md index a9ecfe9..87b7fc1 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -231,8 +231,8 @@ env -u RUBYLIB GEM_HOME="$gem_home" GEM_PATH="$gem_home" \ "$gem_home/bin/automatic" --help ``` -Confirm the installed version equals `VERSION`. Then run the offline Quick -Start shape with a temporary home and a minimal Recipe: +Confirm the installed version equals `VERSION`. Then run a Recipe that needs no +network and no optional gem, with a temporary home: ```sh smoke_home=$(mktemp -d) @@ -375,7 +375,7 @@ security behavior can change independently of this repository: - [ ] `bundle exec rake` passes and the gem builds. - [ ] Package contents and metadata have been inspected. - [ ] GPLv3/LGPLv3, Ruby version and dependencies are correct. -- [ ] The isolated local install and Quick Start smoke test pass. +- [ ] The isolated local install and offline smoke test pass. - [ ] The release commit and immutable `vX.Y.Z` tag identify the built source. - [ ] Publication has explicit approval and `gem push` succeeds. - [ ] RubyGems.org metadata and a clean remote install are verified. diff --git a/plugins/publish/markdown.rb b/plugins/publish/markdown.rb index 22562c8..5d5f86c 100644 --- a/plugins/publish/markdown.rb +++ b/plugins/publish/markdown.rb @@ -171,10 +171,11 @@ def url?(text) # the tags themselves are dropped. # # A parser does it where one is installed, and the substitution below does - # it where none is. That is what keeps this plugin -- the one the Quick - # Start publishes with -- runnable on a plain `gem install automatic`: - # nokogiri is an optional dependency, and reducing a feed body to text is - # not a good enough reason to make everyone install a native extension. + # it where none is. That is what keeps this plugin -- the one a Recipe ends + # with when the result is meant to be read -- runnable on a plain `gem + # install automatic`: nokogiri is an optional dependency, and reducing a + # feed body to text is not a good enough reason to make everyone install a + # native extension. # The two agree on what a body is reduced to; a parser is simply better at # markup that is malformed. See doc/PLUGINS.md section 6.7. def html_to_text(html) diff --git a/spec/config/feed2markdown_spec.rb b/spec/config/feed2markdown_spec.rb index 178b4b8..434a700 100644 --- a/spec/config/feed2markdown_spec.rb +++ b/spec/config/feed2markdown_spec.rb @@ -4,19 +4,20 @@ require 'yaml' -RSpec.describe 'the Quick Start Recipe' do +RSpec.describe 'the feed2markdown example Recipe' do let(:path) { File.join(APP_ROOT, 'config', 'feed2markdown.yml') } let(:recipe) { YAML.safe_load(File.read(path)) } - it 'uses the short supported pipeline documented by the Quick Start' do + it 'uses the short supported pipeline its header documents' do modules = recipe.fetch('plugins').map { |plugin| plugin.fetch('module') } expect(modules).to eq %w[SubscriptionFeed PublishMarkdown] end - # The Quick Start is what a plain `gem install automatic` can run, so the - # Recipe it ships names no plugin that needs an optional gem. The store - # plugins, which do, are the documented next step rather than the first one. - # See doc/POLICY.md section 9.1. + # This is the shipped Recipe a plain `gem install automatic` can run with + # nothing added, which is what makes it the one to reach for first, so it + # names no plugin that needs an optional gem. The store plugins, which do, + # are the documented next step rather than the first one. See + # doc/POLICY.md section 9.1. it 'names no plugin that needs an optional dependency' do modules = recipe.fetch('plugins').map { |plugin| plugin.fetch('module') } expect(modules).not_to include('StorePermalink', 'StoreFullText') @@ -36,7 +37,7 @@ end end - it 'writes the Markdown file described by the Quick Start' do + it 'writes the Markdown file its header describes' do publisher = recipe.fetch('plugins').last.fetch('config') expect(publisher).to include( 'file' => '~/.automatic/markdown/feeds.md', diff --git a/spec/plugins/publish/markdown_spec.rb b/spec/plugins/publish/markdown_spec.rb index 4bd2049..de79285 100644 --- a/spec/plugins/publish/markdown_spec.rb +++ b/spec/plugins/publish/markdown_spec.rb @@ -120,7 +120,7 @@ def publish(config, pipeline) # nokogiri is an optional dependency: the plugin uses it where it is # installed and reduces the body itself where it is not, so that a plain - # `gem install automatic` can run the Quick Start. See doc/PLUGINS.md + # `gem install automatic` can run a Recipe ending here. See doc/PLUGINS.md # section 6.7. The default suite runs the substitute, this run of it is # explicit, and where the optional gem is installed the last example holds # the two to the same answer.