feat(baoyu-fetch): add URL reader CLI with Chrome CDP and site adapters

2026-08-06 00:43:03 +08:00 · 2026-03-27 14:11:00 -05:00
parent 2c14872e88
commit d0764c2739
65 changed files with 10235 additions and 0 deletions
@@ -0,0 +1,124 @@
+# baoyu-fetch
+
+English | [简体中文](./README.zh-CN.md) | [Changelog](./CHANGELOG.md) | [中文更新日志](./CHANGELOG.zh-CN.md)
+
+`baoyu-fetch` is a Bun CLI built on Chrome CDP. Give it a URL and it returns
+high-quality `markdown` or `json`. When a site adapter matches, it prefers API
+responses or structured page data; otherwise it falls back to generic HTML
+extraction.
+
+## Features
+
+- Capture rendered page content through Chrome CDP
+- Observe network requests and responses, and fetch bodies when needed
+- Adapter registry that auto-selects a handler from the URL
+- Built-in adapters for `x`, `youtube`, and `hn`
+- Generic fallback: Defuddle first, then Readability + HTML-to-Markdown; when `--format markdown` is requested, it can also fall back to `defuddle.md`
+- Print `markdown` / `json` to stdout or save with `--output`
+- Optionally download extracted images or videos and rewrite Markdown links
+- Optional wait modes for login and verification flows
+- Chrome profile defaults to `baoyu-skills/chrome-profile`
+
+## Installation
+
+```bash
+bun install
+```
+
+For package usage, the quickest option is:
+
+```bash
+bunx baoyu-fetch https://example.com
+```
+
+You can also install it globally:
+
+```bash
+npm install -g baoyu-fetch
+```
+
+The npm package ships TypeScript source entrypoints instead of a prebuilt
+`dist`, so Bun is required at runtime.
+
+## Usage
+
+```bash
+bun run src/cli.ts https://example.com
+bunx baoyu-fetch https://example.com
+baoyu-fetch https://example.com
+baoyu-fetch https://example.com --format markdown --output article.md
+baoyu-fetch https://example.com --format markdown --output article.md --download-media
+baoyu-fetch https://x.com/jack/status/20 --format json --output article.json
+baoyu-fetch https://x.com/jack/status/20 --json
+baoyu-fetch https://x.com/jack/status/20 --wait-for interaction
+baoyu-fetch https://x.com/jack/status/20 --wait-for force
+baoyu-fetch https://x.com/jack/status/20 --chrome-profile-dir ~/Library/Application\\ Support/baoyu-skills/chrome-profile
+```
+
+## Options
+
+```bash
+baoyu-fetch <url> [options]
+
+Options:
+  --output <file>       Save output to file
+  --format <type>       Output format: markdown | json
+  --json                Alias for --format json
+  --adapter <name>      Force an adapter (for example x / hn / generic)
+  --download-media      Download adapter-reported media into ./imgs and ./videos, then rewrite markdown links
+  --media-dir <dir>     Base directory for downloaded media. Defaults to the output directory
+  --debug-dir <dir>     Write debug artifacts (html, document.json, network.json)
+  --cdp-url <url>       Reuse an existing Chrome DevTools endpoint
+  --browser-path <path> Explicit Chrome binary path
+  --chrome-profile-dir <path>
+                        Chrome user data dir. Defaults to BAOYU_CHROME_PROFILE_DIR
+                        or baoyu-skills/chrome-profile
+  --headless            Launch a temporary headless Chrome if needed
+  --wait-for <mode>     Wait mode: interaction | force
+  --wait-for-interaction
+                        Alias for --wait-for interaction
+  --wait-for-login      Alias for --wait-for interaction
+  --interaction-timeout <ms>
+                        Manual interaction timeout. Default: 600000
+  --interaction-poll-interval <ms>
+                        Poll interval while waiting. Default: 1500
+  --login-timeout <ms>  Alias for --interaction-timeout
+  --login-poll-interval <ms>
+                        Alias for --interaction-poll-interval
+  --timeout <ms>        Page load timeout. Default: 30000
+  --help                Show help
+```
+
+## How It Works
+
+1. The CLI parses the target URL and options.
+2. It opens or connects to a Chrome CDP session and creates a controlled tab.
+3. `NetworkJournal` records requests and responses.
+4. The adapter registry resolves a site-specific adapter when possible.
+5. The adapter returns a structured `ExtractedDocument`.
+6. If nothing matches, generic HTML extraction runs instead.
+7. The result is rendered as Markdown, or returned as JSON with both
+   `document` and `markdown`.
+
+## Development
+
+```bash
+bun run check
+bun run test
+bun run build
+```
+
+## Release
+
+When you make a user-visible change, add a changeset first:
+
+```bash
+bunx changeset
+```
+
+After the generated `.changeset/*.md` file lands on `main`, GitHub Actions will
+open or update the release PR. Merging that release PR publishes the package to
+npm.
+
+The publish flow does not build `dist`; it publishes `src/*.ts` for Bun
+execution directly.