feat: add support for workers

CHANGELOG.md

@@ -6,6 +6,15 @@
 - `Ferrum::Frame#idle?` whether frame was loaded [#583]
 - `page.accessibility` API and `Node#axnode` for reading the CDP accessibility tree
 - `--no-crashpad` is now included in the default Chrome flags, fully disabling the crashpad handler process [#610]
+- Support for dedicated/shared Workers and Service Workers [#391], [#388]:
+  - `Ferrum::Worker` -- a lightweight connection to a Worker's single execution context, with its own `#network`,
+    `#evaluate`/`#evaluate_async`/`#execute`, and `#on(:request)`/`#on(:auth)` for interception.
+  - Dedicated and shared workers spawned by a page are discovered and connected to automatically and are reachable
+    through `Browser#workers`/ `Context#workers`.
+  - Service workers are discovered too, through `Browser#service_workers`/`Context#service_workers`, but aren't
+    connected to by default. Attaching to one keeps it alive forever, so that's opt-in via `Context#attach_target`,
+    then `target.worker`.
+  - `Ferrum::Target#worker?`, `#shared_worker?`, `#service_worker?`, and `#parent_id` for telling targets apart.
 
 ### Fixed
 - `Ferrum::Client` command id generation and `Ferrum::Client::WebSocket`'s driver interactions were not thread-safe, allowing concurrent commands to collide on the same id or corrupt the frame stream; both are now serialized under a mutex [#602]

docs/21-workers.md

@@ -0,0 +1,130 @@
+---
+sidebar_position: 21
+---
+
+# Workers
+
+Ferrum discovers dedicated workers (`new Worker`), shared workers (`new SharedWorker`) and service workers
+(`navigator.serviceWorker.register`) spawned anywhere in a context.
+
+Dedicated and shared workers are connected to automatically as soon as they're discovered, there's no need to wait
+for them. Service workers are treated differently: attaching to a service worker's session prevents Chrome from ever
+terminating it while the connection stays open, so Ferrum only discovers them and leaves the connection alone unless
+you explicitly ask for it with `attach_target`.
+
+#### workers : `Array<Worker>`
+
+Returns all dedicated and shared workers spawned by pages in this context, already connected.
+
+```ruby
+page.execute <<~JS
+  new Worker(URL.createObjectURL(new Blob(
+    ["self.onmessage = () => self.postMessage(1 + 1)"],
+    { type: "application/javascript" }
+  )))
+JS
+
+sleep 0.1 # give Ferrum a moment to discover and connect to it
+
+worker = browser.workers.first # => #<Ferrum::Worker @target_id="..." @url="blob:...">
+worker.evaluate("1 + 1") # => 2
+```
+
+#### service_workers : `Array<Target>`
+
+Returns service worker targets registered in this context. Unlike `workers`, these are plain `Target` objects and
+are not connected to -- call `attach_target` first if you want to interact with one.
+
+```ruby
+page.evaluate_async(%(navigator.serviceWorker.register("/sw.js").then(arguments[0])), 5)
+
+sleep 0.1
+
+target = browser.service_workers.first # => #<Ferrum::Target @id="..." @type="service_worker">
+target.connected? # => false
+```
+
+#### attach_target(target_id) : `Boolean`
+
+Manually attaches to a target discovered via `service_workers`. Once attached, `target.worker` returns a connected
+`Worker` for it, same as for a dedicated or shared worker.
+
+```ruby
+target = browser.service_workers.first
+browser.attach_target(target.id)
+target.worker.evaluate("1 + 1") # => 2
+```
+
+Keep in mind this keeps the service worker alive for as long as the connection is open -- only attach to the ones
+you actually intend to talk to.
+
+## Worker
+
+A dedicated or shared Worker spawned by a page. Unlike `Page` it has no DOM, frames, mouse/keyboard, or navigation
+history -- just a single global execution context and its own network activity, reachable through `worker.network`.
+
+#### target_id : `String`
+
+The worker's CDP target id.
+
+#### url : `String`
+
+The worker's script URL, e.g. a `blob:` URL for workers created from an inline `Blob`, as is common for
+dedicated/shared workers.
+
+#### network : `Network`
+
+Same API as `page.network` -- see [Network](/docs/ferrum/network). Useful for asserting on requests a worker made on
+its own, independently of the page that spawned it.
+
+```ruby
+worker.network.wait_for_idle
+worker.network.traffic # => [#<Ferrum::Network::Exchange, ...]
+```
+
+#### evaluate / execute / evaluate_async
+
+Same API as on `Page`/`Frame` -- see [JavaScript](/docs/ferrum/javascript). A worker has a single execution context,
+so there's no frame to target.
+
+```ruby
+worker.evaluate("1 + 1") # => 2
+```
+
+#### on(:request) / on(:auth)
+
+Subscribe to the worker's own network interception events, same as `page.on(:request)` -- see the `intercept` method
+in [Network](/docs/ferrum/network).
+
+#### close : `Boolean`
+
+Closes the underlying CDP target and the worker's own connection.
+
+```ruby
+worker.close
+```
+
+## Example
+
+Spawn a page's worker, wait for Ferrum to discover it, and read back a result it computes and posts to the page:
+
+```ruby
+page.go_to
+page.execute <<~JS
+  window.worker = new Worker(URL.createObjectURL(new Blob(
+    ["self.onmessage = (e) => self.postMessage(e.data * 2)"],
+    { type: "application/javascript" }
+  )))
+  window.worker.onmessage = (e) => { window.result = e.data }
+JS
+
+worker = nil
+until worker
+  worker = browser.workers.first
+  sleep 0.05
+end
+
+page.execute("window.worker.postMessage(21)")
+sleep 0.1
+page.evaluate("window.result") # => 42
+```

lib/ferrum/browser.rb

@@ -3,6 +3,7 @@
 require "base64"
 require "forwardable"
 require "ferrum/page"
+require "ferrum/worker"
 require "ferrum/client"
 require "ferrum/contexts"
 require "ferrum/browser/xvfb"
@@ -16,7 +17,7 @@ class Browser
     extend Forwardable
 
     delegate %i[default_context] => :contexts
-    delegate %i[targets create_target page pages windows] => :default_context
+    delegate %i[targets create_target page pages windows workers service_workers attach_target] => :default_context
     delegate %i[go_to goto go back forward refresh reload stop wait_for_reload
                 at_css at_xpath css xpath current_url current_title url title
                 body doctype content=

lib/ferrum/context.rb

@@ -25,7 +25,24 @@ def page
     end
 
     def pages
-      @targets.values.reject(&:iframe?).map(&:page)
+      @targets.values.select(&:page?).map(&:page)
+    end
+
+    # Dedicated and shared workers spawned by any page in this context.
+    #
+    # @return [Array<Worker>]
+    def workers
+      @targets.values.select { |t| t.worker? || t.shared_worker? }.map(&:worker)
+    end
+
+    # Service worker targets registered in this context. Unlike {#workers},
+    # these are plain {Target}s and are not connected to. Attaching to a
+    # service worker's session keeps it alive indefinitely, so we only do
+    # that on demand, via `target.worker`.
+    #
+    # @return [Array<Target>]
+    def service_workers
+      @targets.values.select(&:service_worker?)
     end
 
     # When we call `page` method on target it triggers ruby to connect to given
@@ -68,7 +85,7 @@ def add_target(params:, session_id: nil)
       new_pending = Concurrent::IVar.new
       pending = @pendings.put_if_absent(target.id, new_pending) || new_pending
       pending.try_set(true)
-      true
+      target
     end
 
     def update_target(target_id, params)
@@ -79,10 +96,17 @@ def delete_target(target_id)
       @targets.delete(target_id)
     end
 
+    # Manually attaches to a target, e.g. a service worker discovered via
+    # {#service_workers}. Once attached, `target.worker`/`target.page`
+    # returns a connected {Worker}/{Page} for it.
+    #
+    # Note: attaching to a service worker's session prevents Chrome from
+    # ever terminating it while the connection is open.
     def attach_target(target_id)
       target = @targets[target_id]
       raise NoSuchTargetError unless target
 
+      @contexts.manually_attached(target_id)
       session = @client.command("Target.attachToTarget", targetId: target_id, flatten: true)
       target.session_id = session["sessionId"]
       true
@@ -98,7 +122,7 @@ def close_targets_connection
       @targets.each_value do |target|
         next unless target.connected?
 
-        target.page.close_connection
+        target.close_connection
       end
     end
 

lib/ferrum/contexts.rb

@@ -4,20 +4,28 @@
 
 module Ferrum
   class Contexts
-    ALLOWED_TARGET_TYPES = %w[page iframe].freeze
+    ALLOWED_TARGET_TYPES = %w[page iframe worker shared_worker service_worker].freeze
+    RECURSIVE_AUTO_ATTACH_TYPES = %w[page iframe worker shared_worker].freeze
 
     include Enumerable
 
     attr_reader :contexts
 
     def initialize(client)
-      @contexts = Concurrent::Map.new
       @client = client
+      @contexts = Concurrent::Map.new
+      @manually_attached = Concurrent::Map.new
       subscribe
       auto_attach
       discover
     end
 
+    # Marks a target, so the next time we see it attached, we leave its session alone instead of {#detach}ing it.
+    # Used by {Context#attach_target} right before it manually attaches to a service worker on the caller's behalf.
+    def manually_attached(target_id)
+      @manually_attached[target_id] = true
+    end
+
     def default_context
       @default_context ||= create
     end
@@ -72,20 +80,28 @@ def size
 
     private
 
-    def subscribe # rubocop:disable Metrics/PerceivedComplexity, Metrics/CyclomaticComplexity
-      @client.on("Target.attachedToTarget") do |params|
+    def subscribe
+      subscribe_attached_target(@client)
+      subscribe_target_created
+    end
+
+    # Registered once on the top-level client, and again on every page's/
+    # worker's own session once we re-arm auto-attach on it.
+    def subscribe_attached_target(client)
+      client.on("Target.attachedToTarget") do |params|
         info, session_id = params.values_at("targetInfo", "sessionId")
         next unless ALLOWED_TARGET_TYPES.include?(info["type"])
 
         context_id = info["browserContextId"]
         add_context(context_id)
+        target = @contexts[context_id]&.add_target(session_id: session_id, params: info)
 
-        @contexts[context_id]&.add_target(session_id: session_id, params: info)
-        if params["waitingForDebugger"]
-          @client.session(session_id).command("Runtime.runIfWaitingForDebugger", async: true)
-        end
+        rearm_auto_attach(session_id, info["type"])
+        handle_attach(target, session_id, params)
       end
+    end
 
+    def subscribe_target_created
       @client.on("Target.targetCreated") do |params|
         info = params["targetInfo"]
         next unless ALLOWED_TARGET_TYPES.include?(info["type"])
@@ -120,6 +136,55 @@ def subscribe # rubocop:disable Metrics/PerceivedComplexity, Metrics/CyclomaticC
       end
     end
 
+    def rearm_auto_attach(session_id, type)
+      return unless RECURSIVE_AUTO_ATTACH_TYPES.include?(type)
+
+      client = @client.session(session_id)
+      client.command("Target.setAutoAttach", autoAttach: true, waitForDebuggerOnStart: true, flatten: true, async: true)
+      subscribe_attached_target(client)
+    end
+
+    def handle_attach(target, session_id, params)
+      return unless target
+
+      if target.service_worker?
+        detach_unless_manually_attached(target, session_id)
+      elsif target.worker? || target.shared_worker?
+        connect_worker(target)
+      elsif params["waitingForDebugger"]
+        resume(session_id)
+      end
+    end
+
+    # Attaching keeps a service worker alive forever, so unless the caller
+    # explicitly asked to connect to it (via Context#attach_target), we
+    # just resume it and let go.
+    def detach_unless_manually_attached(target, session_id)
+      return if @manually_attached.delete(target.id)
+
+      detach(session_id)
+    rescue BrowserError
+      nil
+    end
+
+    # Workers have no events to notify us when they're ready, so we
+    # connect right away. Worker#prepare enables the Network domain and
+    # only then resumes the debugger itself.
+    def connect_worker(target)
+      target.worker
+    rescue BrowserError
+      nil
+    end
+
+    def resume(session_id)
+      @client.session(session_id).command("Runtime.runIfWaitingForDebugger", async: true)
+    end
+
+    def detach(session_id)
+      resume(session_id)
+      @client.command("Target.detachFromTarget", sessionId: session_id)
+    end
+
     def discover
       @client.command("Target.setDiscoverTargets", discover: true)
     end