feat: opt-in block_internal_urls egress filter for the library layer (Refs #2146)

[joysinleung]

Summary

Implements the opt-in library-side egress filter discussed in #2146. It started life as a responsible-disclosure report (the library fetches any caller-supplied URL, including internal/cloud-metadata addresses), and the maintainers classified that as intended behavior for a user-agent invoked by a trusted caller — the SSRF trust boundary correctly lives at the Docker API server (egress_broker.py). The one scenario that justifies exposing the same primitives to library callers is agentic / LLM-chosen-URL pipelines, where a prompt-injection or malicious page can steer the crawler at internal hosts before the embedding app validates the URL.

This PR adds an opt-in block_internal_urls flag so those callers get server-grade destination filtering without re-implementing it.

Design

New module crawl4ai/url_safety.py (dependency-free — no FastAPI, no egress proxy, so it imports cleanly from the library):

• Reuses the same blocked address ranges as deploy/docker/utils.py (_BLOCKED_NETWORKS: loopback, private, link-local incl. 169.254.0.0/16 cloud metadata, CGN 100.64/10, 0.0.0.0/8, IPv6 ULA/link-local/loopback) plus a _BLOCKED_HOSTNAMES set (localhost, metadata, metadata.google.internal, kubernetes.default*).
• BlockedURLError(ValueError) with an opaque message that never echoes the resolved IP/hostname (no DNS-oracle leak).
• Expands IPv6-embedded IPv4 forms (::ffff:127.0.0.1, ::127.0.0.1) before checking, mirroring the Docker server.
• Honors a CRAWL4AI_ALLOW_INTERNAL_URLS escape hatch (parity with the server).
• raw:/raw:// URLs (inline HTML, no network fetch) are never treated as internal.

Config: CrawlerRunConfig.block_internal_urls (default False) — preserves the user-agent contract; existing behavior is unchanged. Because from_kwargs/to_dict/clone are signature-driven, the field flows through serialization automatically.

Chokepoint: inserted in both crawl entry points right after the scheme allow-list check:

• AsyncCrawlerStrategy.crawl() → covers the HTTP path (AsyncHTTPCrawlerStrategy._handle_http).
• AsyncPlaywrightCrawlerStrategy.crawl() → covers the browser path (page.goto).

Only http(s):// URLs are checked; file:// and raw: are local/inline content and are skipped.

Tests

tests/async/test_url_safety.py (23 tests, all passing). Deterministic (IP literals, no DNS dependency):

• is_internal_url / check_url_destination classification for internal vs global addresses (incl. IPv6, CGN, metadata).
• BlockedURLError raised on internal, passes on global.
• CRAWL4AI_ALLOW_INTERNAL_URLS override.
• CrawlerRunConfig default / set / to_dict roundtrip / clone.
• Integration: a local HTTP server + AsyncHTTPCrawlerStrategy proves the chokepoint actually fires — block_internal_urls=True raises before any connection, False (default) fetches and returns content.

Answers to the open questions from #2146

1. Flag name — went with block_internal_urls (matches the existing check_robots_txt / prefetch style on CrawlerRunConfig). Happy to rename if you prefer deny_private_destinations / egress_filter.
2. Chokepoint — shared logic lives in url_safety.py; it's called from both crawl() entry points so HTTP and browser paths stay in sync (no duplicated trust logic).
3. Browser redirects — the check covers the initial URL (pre-connection, single resolution). Playwright goto follows redirects internally and the library connects directly (no pinning proxy), so redirect hops are not revalidated in this opt-in layer. That gap is exactly why the authoritative trust boundary for untrusted multi-tenant input remains the Docker server's egress_broker (DNS pinning + per-hop revalidation). I noted this as a deliberate limitation rather than silently claiming full coverage — if you'd like, a follow-up could add an opt-in "disable browser redirects + manual revalidate" mode.

Scope note

This is not a default-behavior change and does not close the original SSRF report's premise — it's the opt-in escape hatch the maintainers offered. Default arun(url) behavior is unchanged.

Refs #2146.