Compare commits

...

27 Commits

Author SHA1 Message Date
Jim Liu 宝玉 aa1a967a9f chore: release v1.114.1 2026-05-08 17:45:03 -05:00
Jim Liu 宝玉 d643bad53c test(baoyu-danger-gemini-web): cover generated image response fallback 2026-05-08 17:43:31 -05:00
Jim Liu 宝玉 0d977787b5 Merge pull request #146 from evilstar2016/fix/gemini-generated-image-detection
fix(gemini-webapi): add fallback scan for generated images
2026-05-08 17:40:45 -05:00
evilstar2016 516803feb4 fix(gemini-webapi): add fallback scan for generated images when wants_generated fails
The `wants_generated` detection checks `candidate[12][7][0]` and an old
`googleusercontent.com/image_generation_content/` URL pattern in the response
text. Both are no longer present in the current Gemini Web API response format,
causing the entire generated-image extraction block to be skipped even when
Gemini successfully generates images — resulting in "No image returned in
response" errors.

Generated image URLs now appear as `https://lh3.googleusercontent.com/gg-dl/`
somewhere in the response parts. This commit adds an unconditional fallback that
scans all response parts for those URLs when `generated_images` is still empty
after the existing `wants_generated` block, reusing the already-present
`collect_strings` helper and `GeneratedImage` constructor.

The existing code path is untouched — the fallback only runs when no images
were found through the original logic, so old response formats continue to work.
2026-05-06 18:48:08 +09:00
Jim Liu 宝玉 4af0506fa3 chore: release v1.114.0 2026-05-05 11:17:55 -05:00
Jim Liu 宝玉 80a1c2970a docs(release-skills): add GitHub release publishing workflow 2026-05-05 11:17:45 -05:00
Jim Liu 宝玉 cdfa0dbff9 feat(baoyu-infographic): add retro popup pop style 2026-05-05 11:17:42 -05:00
Jim Liu 宝玉 505a7e10ce chore: release v1.113.0 2026-04-25 15:03:22 -05:00
Jim Liu 宝玉 6d063734ae feat(baoyu-imagine): add DashScope Wan 2.7 image model support (#141)
* feat(baoyu-imagine): add DashScope Wan 2.7 image model support

Closes #139.

Adds the new `wan2.7-image-pro` and `wan2.7-image` model family to the
DashScope provider so users can call Wan 2.7 directly through the
official Aliyun (Bailian) API instead of going through Replicate.

- Register `wan2.7-image-pro` and `wan2.7-image` as a new `wan27` family
  in the DashScope provider with their own size resolution rules:
  pixel range `[768*768, 4096*4096]` for `wan2.7-image-pro` text-to-image,
  `[768*768, 2048*2048]` for `wan2.7-image-pro` with refs and for the
  base `wan2.7-image` model in any mode, with aspect ratios validated
  against the documented `[1:8, 8:1]` band.
- Allow up to 9 reference images per request (image editing /
  multi-image fusion). Local files are inlined as base64 data URLs;
  `http(s)://` paths are forwarded as-is. Other DashScope models still
  reject `--ref` with a hint to switch to a wan2.7 model or another
  provider.
- Drop `prompt_extend` from the request body for the Wan 2.7 family
  (not part of the Wan 2.7 API surface) and skip the Qwen-only negative
  prompt for this family.
- Allow `--provider dashscope --ref ...` in `detectProvider` so users
  can opt into Wan 2.7 reference workflows, while keeping Wan 2.7 out
  of the auto-detect ref priority list.
- Add provider, reference, and usage-example documentation, plus
  unit tests covering family routing, size derivation across the
  three pixel-budget modes, ratio rejection, explicit-size validation,
  and the new `--provider dashscope` ref opt-in path.

Made-with: Cursor

* fix(baoyu-imagine): force n=1 for DashScope wan2.7 to avoid silent multi-image billing

Cross-checked the implementation against the official Wan 2.7 image
generation & editing API reference and found that the API defaults
`parameters.n` to 4 in non-collage mode (1-4 range, billed per image).
baoyu-imagine has single-image save semantics — only the first image
in the response is kept — so without an explicit `n: 1` users would
silently pay for 3 discarded images per request.

- Always send `parameters.n: 1` in the wan2.7 request body
- Reject `--n > 1` for wan2.7 with a clear error pointing at the
  single-image save semantics
- Add tests asserting the request body shape (n=1, no prompt_extend,
  no negative_prompt) and the --n>1 rejection
- Document the defaults-vs-skill mismatch in the dashscope reference

Made-with: Cursor

* Fix DashScope Wan 2.7 review feedback
2026-04-25 14:54:08 -05:00
Jim Liu 宝玉 31d728b505 chore: release v1.112.0 2026-04-24 02:15:57 -05:00
Jim Liu 宝玉 f6d5df0594 fix(baoyu-post-to-x): add entry point guard to md-to-html.ts for module import compatibility
Wrap main() in an import.meta.url check so that importing parseMarkdown
from x-article.ts no longer triggers the CLI entry point. Mirrors the
same fix applied to baoyu-post-to-weibo.
2026-04-24 02:15:30 -05:00
Jim Liu 宝玉 4bd5fe573e feat(baoyu-article-illustrator): default to sketch-notes educational infographic style
Make `hand-drawn-edu` (infographic + sketch-notes + macaron) the universal
fallback preset when content analysis surfaces no strong signal. Rework
sketch-notes style spec around warm cream paper + black hand-drawn lines +
soft pastel section blocks.

- Change `hand-drawn-edu` preset type from flowchart to infographic; add
  `hand-drawn-edu-flow` (flowchart) and `hand-drawn-edu-compare` (comparison)
  as variants for users who need those layouts in the same warm style
- Elevate `sketch-notes` to primary style across infographic / flowchart /
  comparison / framework auto-selection; add sketch-notes column to
  Type x Style compatibility matrix
- Rewrite sketch-notes.md: macaron pastel palette, canonical single-page
  layout (title / sectioned boxes / takeaway), diagram-only rule
- Add infographic + sketch-notes + macaron prompt block to
  prompt-construction.md
- Update workflow, style-presets, and first-time-setup defaults to match
2026-04-24 02:15:25 -05:00
Jim Liu 宝玉 8c17d77209 chore: release v1.111.1 2026-04-21 16:59:35 -05:00
Jim Liu 宝玉 fb749aae36 docs(baoyu-article-illustrator): add Confirmation Policy
State that defaults / signals / EXTEND.md preferences are recommendation
inputs only — Step 3 is mandatory unless the current request opts out
explicitly.
2026-04-21 16:59:11 -05:00
Jim Liu 宝玉 a23f2e6a4b docs(baoyu-xhs-images): add Confirmation Policy
Sync Confirmation Policy with baoyu-image-cards (deprecated skill kept
functional per CLAUDE.md).
2026-04-21 16:59:11 -05:00
Jim Liu 宝玉 ae32c56a3c docs(baoyu-image-cards): add Confirmation Policy
State that defaults / signals / EXTEND.md preferences are recommendation
inputs only — Step 2 is mandatory unless the current request opts out via
`--yes` / equivalent wording.
2026-04-21 16:59:07 -05:00
Jim Liu 宝玉 4b2ad3ad18 docs(baoyu-slide-deck): add Confirmation Policy
State that defaults / signals / EXTEND.md preferences are recommendation
inputs only — Step 2 is mandatory unless the current request opts out
explicitly.
2026-04-21 16:59:07 -05:00
Jim Liu 宝玉 512b6d2e15 docs(baoyu-cover-image): add Confirmation Policy
State that defaults / keywords / EXTEND.md preferences are recommendation
inputs only — Step 2 is mandatory unless the current request opts out via
`--quick` / `quick_mode: true` / equivalent wording.
2026-04-21 16:59:03 -05:00
Jim Liu 宝玉 60d0b9c95b docs(baoyu-infographic): add Confirmation Policy as single source of truth
Consolidate the confirmation rule into one section so defaults, keywords,
and EXTEND.md preferences are treated as recommendation inputs only — never
as authorization to skip Step 4. Remove the repeated reminders that were
scattered across Step 5, Step 6, Default combination, Keyword Shortcuts,
and the preferences docs.
2026-04-21 16:58:58 -05:00
Jim Liu 宝玉 acf1d42a64 chore: release v1.111.0 2026-04-21 16:44:01 -05:00
Jim Liu 宝玉 adb783ae6d refactor: unify image-backend resolution with preferred_image_backend preference
Replaces the stateless ask-once backend rule with a 4-step resolution
(current-request override > saved preference > auto-select > ask) and
adds a single `preferred_image_backend` field (auto | ask | <backend-id>)
to every image-consuming skill's EXTEND.md schema. Runtime-native tools
(Codex `imagegen`, Hermes `image_generate`, ...) win by default; absent
field equals `auto` so existing user EXTEND.md files stay valid with no
schema version bump.

Each image-consuming skill also gains a top-level `## Changing
Preferences` section as a first-class surface for pinning a backend and
editing common one-line preferences.

Applies across:
  - baoyu-infographic (reverts in-progress two-field image_backend_mode design)
  - baoyu-comic
  - baoyu-cover-image
  - baoyu-image-cards
  - baoyu-article-illustrator
  - baoyu-slide-deck
  - baoyu-xhs-images
  - docs/image-generation-tools.md (canonical author-side doc)
2026-04-21 16:43:58 -05:00
Jim Liu 宝玉 a4bd37c390 chore: release v1.110.0 2026-04-21 15:15:19 -05:00
Jim Liu 宝玉 f8e434d2a0 chore: release v1.109.0 2026-04-21 13:39:42 -05:00
Jim Liu 宝玉 bb66585c00 fix(project): include vendored skill files in releases 2026-04-21 13:39:13 -05:00
Jim Liu 宝玉 d6442e1ab4 feat(baoyu-url-to-markdown): vendor baoyu-fetch runtime 2026-04-21 13:39:09 -05:00
Jim Liu 宝玉 2f09f48726 chore(deps): upgrade Defuddle extraction stack 2026-04-21 13:39:05 -05:00
Jim Liu 宝玉 fe26c2b639 fix(baoyu-fetch): extract X video variants 2026-04-21 13:39:01 -05:00
112 changed files with 10312 additions and 618 deletions
+1 -1
View File
@@ -6,7 +6,7 @@
},
"metadata": {
"description": "Skills shared by Baoyu for improving daily work efficiency",
"version": "1.108.0"
"version": "1.114.1"
},
"plugins": [
{
+74 -7
View File
@@ -1,6 +1,6 @@
---
name: release-skills
description: Universal release workflow. Auto-detects version files and changelogs. Supports Node.js, Python, Rust, Claude Plugin, and generic projects. Use when user says "release", "发布", "new version", "bump version", "push", "推送".
description: Universal release workflow. Auto-detects version files and changelogs. Supports Node.js, Python, Rust, Claude Plugin, GitHub Releases, annotated tags, historical release backfill, and generic projects. Use when user says "release", "发布", "new version", "bump version", "push", "推送", "release notes", "GitHub Release", or "回填 Release".
---
# Release Skills
@@ -39,6 +39,7 @@ Just run `/release-skills` - auto-detects your project configuration.
| `--major` | Force major version bump |
| `--minor` | Force minor version bump |
| `--patch` | Force patch version bump |
| `--backfill-releases` | Create missing GitHub Releases for existing tags from changelog sections |
## Workflow
@@ -57,7 +58,11 @@ Just run `/release-skills` - auto-detects your project configuration.
- `HISTORY*.md`
- `CHANGES*.md`
4. Identify language of each changelog by filename suffix
5. Display detected configuration
5. Detect GitHub release support:
- Check whether `origin` points to GitHub
- Check whether `gh` is installed and authenticated
- Check existing releases with `gh release list --limit 5` when available
6. Display detected configuration
**Project Hook Contract**:
@@ -310,6 +315,14 @@ git commit -m "docs(project): update architecture documentation"
- Read version file (JSON/TOML/text)
- Update version number
- Write back (preserve formatting)
3. **Create release notes file**:
- Prefer the new version section from `CHANGELOG.md`
- If no English/default changelog exists, use the first detected changelog
- Extract only the exact `## {VERSION} - {YYYY-MM-DD}` section through the next `##`
- Match both plain version and tag-prefixed headings when needed, e.g. `1.2.3` and `v1.2.3`
- Keep breaking changes near the top; if needed, add a short highlight before other sections
- Write notes to a UTF-8 temp file and reuse it for annotated tag messages, GitHub Releases, and `publish_artifact`
- In normal mode, stop rather than creating an empty tag or GitHub Release when notes cannot be found
**Version Paths by File Type**:
@@ -325,7 +338,7 @@ git commit -m "docs(project): update architecture documentation"
Before creating the release commit, ask user to confirm:
**Use AskUserQuestion with two questions**:
**Use AskUserQuestion with three questions**:
1. **Version bump** (single select):
- Show recommended version based on Step 3 analysis
@@ -335,6 +348,11 @@ Before creating the release commit, ask user to confirm:
2. **Push to remote** (single select):
- Options: "Yes, push after commit", "No, keep local only"
3. **Publish GitHub Release** (single select):
- Offer this only when GitHub release support is available
- Default to "Yes, publish after tag push" when the user also chose push
- If the user keeps the release local, do not create or edit a GitHub Release
**Example Output Before Confirmation**:
```
Commits created:
@@ -349,10 +367,11 @@ Changelog preview (en):
### Fixes
- Improve panel layout for long dialogues in comic
Ready to create release commit and tag.
Release notes source: CHANGELOG.md#1.3.0
Ready to create release commit, annotated tag, and GitHub Release.
```
### Step 9: Create Release Commit and Tag
### Step 9: Create Release Commit and Annotated Tag
After user confirmation:
@@ -367,10 +386,11 @@ After user confirmation:
git commit -m "chore: release v{VERSION}"
```
3. **Create tag**:
3. **Create annotated tag**:
```bash
git tag v{VERSION}
git tag -a v{VERSION} -F <release-notes-file>
```
If `.releaserc.yml` sets `tag.sign: true`, use `git tag -s` with the same notes file.
4. **Push if user confirmed** (Step 8):
```bash
@@ -380,6 +400,28 @@ After user confirmation:
**Note**: Do NOT add Co-Authored-By line. This is a release commit, not a code contribution.
### Step 10: Publish Release Artifacts and GitHub Release
Project artifact publishing and GitHub Releases are separate outputs:
1. **Project artifacts**:
- If `release.hooks.publish_artifact` exists, run it once per prepared target
- Pass the same `{release_notes_file}` used for the tag and GitHub Release
- In dry-run mode, pass `{dry_run}=true` and report what would be published
2. **GitHub Release**:
- Run only if the user confirmed remote publishing and GitHub support is available
- Ensure the tag exists on the remote before creating the release
- Create or update using the extracted notes:
```bash
if gh release view v{VERSION} >/dev/null 2>&1; then
gh release edit v{VERSION} --title "v{VERSION}" --notes-file <release-notes-file>
else
gh release create v{VERSION} --title "v{VERSION}" --notes-file <release-notes-file> --verify-tag
fi
```
- Never inline multiline release notes into shell commands
**Post-Release Output**:
```
Release v1.3.0 created.
@@ -391,9 +433,32 @@ Commits:
4. chore: release v1.3.0
Tag: v1.3.0
Tag type: annotated
GitHub Release: published # or "skipped/local only"
Status: Pushed to origin # or "Local only - run git push when ready"
```
## Backfill Existing GitHub Releases
Use this mode when the user asks to backfill historical releases or passes `--backfill-releases`.
1. Do not bump versions, edit changelogs, or create release commits.
2. List existing tags in version order and detect missing releases:
```bash
git tag --sort=v:refname
gh release view <tag>
```
3. For each tag without a GitHub Release:
- Normalize the changelog lookup by stripping the configured tag prefix, e.g. `v1.2.3` -> `1.2.3`
- Extract the matching section from `CHANGELOG.md`; fall back to the first matching changelog file
- Skip or ask before publishing if no matching changelog section exists
- Create the release with:
```bash
gh release create <tag> --title "<tag>" --notes-file <release-notes-file> --verify-tag
```
4. Detect lightweight tags with `git cat-file -t <tag>` (`commit` means lightweight, `tag` means annotated).
5. Do not rewrite public lightweight tags by default. Converting an existing remote tag to an annotated tag requires explicit user confirmation because it rewrites a published reference.
## Configuration (.releaserc.yml)
Optional config file in project root to override defaults:
@@ -498,6 +563,7 @@ No changes made. Run without --dry-run to execute.
/release-skills --minor # Force minor bump
/release-skills --patch # Force patch bump
/release-skills --major # Force major bump (with confirmation)
/release-skills --backfill-releases # Create missing GitHub Releases for existing tags
```
## When to Use
@@ -506,6 +572,7 @@ Trigger this skill when user requests:
- "release", "发布", "create release", "new version", "新版本"
- "bump version", "update version", "更新版本"
- "prepare release"
- "release notes", "GitHub Release", "回填 Release"
- "push to remote" (with uncommitted changes)
**Important**: If user says "just push" or "直接 push" with uncommitted changes, STILL follow all steps above first.
+61 -1
View File
@@ -2,6 +2,66 @@
English | [中文](./CHANGELOG.zh.md)
## 1.114.1 - 2026-05-08
### Fixes
- `baoyu-danger-gemini-web`: restore generated-image extraction for current Gemini Web responses where image URLs appear as `https://lh3.googleusercontent.com/gg-dl/` without the legacy generated-image markers. Adds regression coverage for the fallback response shape. (by @evilstar2016)
## 1.114.0 - 2026-05-05
### Features
- `baoyu-infographic`: add `retro-popup-pop` style — retro pixel popup × pop-art collage. Renders content as a stack of 80/90s desktop popup windows (title bars, close buttons, ERROR / ALERT dialogs, file windows like `PROBLEMS.EXE`, progress bars, OK / CANCEL / FIX IT buttons) with thick black outlines, flat color fills, and bright cyan (#12B8DE) or vintage cream (#F5F0E6) backgrounds. Pairs especially well with the `dense-modules` layout; promoted as a recommended style for the `高密度信息大图` keyword shortcut and the `Product/Buying Guide` content type. Style Gallery count goes from 21 to 22.
Credit to AJ@WaytoAGI.
### Documentation
- `release-skills`: document GitHub Release publishing in the release workflow, including release-notes extraction from changelog sections, annotated tag creation, `gh release create/edit`, and historical release backfill for existing tags.
## 1.113.0 - 2026-04-25
### Features
- `baoyu-imagine`: add DashScope Wan 2.7 image model support (`wan2.7-image-pro` and `wan2.7-image`) directly through the official Aliyun (Bailian) API. Supports text-to-image, image editing, and multi-image fusion with up to 9 reference images, with documented `[1:8, 8:1]` aspect ratio validation and per-mode pixel-budget rules. Forces `parameters.n: 1` to match baoyu-imagine's single-image save semantics and explicitly rejects `--n > 1` to prevent silent multi-image billing (the API defaults to `n=4` in non-collage mode). Allows `--provider dashscope --ref ...` opt-in for Wan 2.7 reference workflows.
## 1.112.0 - 2026-04-24
### Features
- `baoyu-article-illustrator`: make `hand-drawn-edu` (infographic + sketch-notes + macaron) the universal fallback preset when content analysis surfaces no strong signal — warm cream paper, black hand-drawn lines, soft pastel section blocks. Elevate `sketch-notes` to primary style across infographic / flowchart / comparison / framework auto-selection; rewrite the sketch-notes style spec (macaron palette, canonical single-page layout, diagram-only rule); add matching prompt block and workflow defaults.
- `baoyu-article-illustrator`: add `hand-drawn-edu-flow` (flowchart) and `hand-drawn-edu-compare` (comparison) presets for the same warm educational style.
### Breaking Changes
- `baoyu-article-illustrator`: `hand-drawn-edu` preset now maps to `infographic` instead of `flowchart`. Users relying on the previous flowchart behavior should switch to the new `hand-drawn-edu-flow` preset.
### Fixes
- `baoyu-post-to-x`: add entry point guard to `scripts/md-to-html.ts` so that importing `parseMarkdown` from `x-article.ts` no longer triggers the CLI entry point. Mirrors the same fix applied to `baoyu-post-to-weibo`.
## 1.111.1 - 2026-04-21
### Documentation
- Add a top-level `## Confirmation Policy` section to every image-generating skill (`baoyu-infographic`, `baoyu-cover-image`, `baoyu-slide-deck`, `baoyu-image-cards`, `baoyu-xhs-images`, `baoyu-article-illustrator`) as a single source of truth: explicit skill invocation, keyword shortcuts, EXTEND.md defaults, and auto-recommendations are recommendation inputs only — they never authorize skipping the confirmation step. Opt-out requires an explicit current-request signal (`--no-confirm` / `--quick` / `--yes` / "直接生成" / equivalent).
- `baoyu-infographic`: consolidate the scattered reminders (previously repeated across Step 5, Step 6, Default combination, Keyword Shortcuts, and the preferences docs) into a single policy section referenced from Step 4's hard gate.
## 1.111.0 - 2026-04-21
### Refactor
- Unify image-backend resolution across all image-consuming skills (`baoyu-infographic`, `baoyu-comic`, `baoyu-cover-image`, `baoyu-image-cards`, `baoyu-article-illustrator`, `baoyu-slide-deck`, `baoyu-xhs-images`): add a single `preferred_image_backend` preference field (`auto | ask | <backend-id>`) and replace the stateless ask-once rule with a 4-step resolution (current-request override → saved preference → auto-select → ask). Runtime-native tools (Codex `imagegen`, Hermes `image_generate`) are preferred by default; existing `EXTEND.md` files without the field are treated as `auto` with no schema bump.
- Add a top-level `## Changing Preferences` section to each image-consuming skill as a first-class surface for pinning the backend and editing common one-line preferences.
## 1.110.0 - 2026-04-21
### Features
- `baoyu-imagine`: add `gpt-image-2` support for OpenAI image generation and edits, make it the default OpenAI model, and document the official size/quality mapping, custom-size constraints, and Azure deployment guidance
## 1.109.0 - 2026-04-21
### Features
- `baoyu-url-to-markdown`: vendor the `baoyu-fetch` runtime into `scripts/lib` and run it through a local `scripts/baoyu-fetch` CLI so published skill installs are self-contained
### Fixes
- `baoyu-fetch`: extract playable X/Twitter video MP4 variants for single posts and X Articles, choosing the highest-bitrate MP4 and rendering article videos as `[video](...)`
- `sync-clawhub`: publish from the shared release file list so extensionless CLI entrypoints, `bun.lock`, and vendored `scripts/lib` files are uploaded
### Maintenance
- Upgrade `defuddle` to 0.17.0 and `jsdom` to 29.0.2; override `@xmldom/xmldom` to 0.8.13 to keep the Defuddle dependency chain vulnerability-free
## 1.108.0 - 2026-04-19
### Refactor
@@ -636,7 +696,7 @@ English | [中文](./CHANGELOG.zh.md)
- `baoyu-format-markdown`: add reader-perspective content analysis phase — analyzes highlights, structure, and formatting issues before applying formatting
- `baoyu-format-markdown`: restructure workflow from 8 steps to 7 with explicit do/don't formatting principles and completion report
- `baoyu-translate`: extract Step 2 workflow mechanics to separate reference file for cleaner SKILL.md
- `baoyu-translate`: expand trigger keywords (改成中文, 快翻, 本地化, etc.) for better skill activation
- `baoyu-translate`: expand trigger keywords (改成中文,快翻,本地化etc.) for better skill activation
- `baoyu-translate`: add proactive warning for long content in quick mode
- `baoyu-translate`: save frontmatter to `chunks/frontmatter.md` during chunking
+63 -3
View File
@@ -2,6 +2,66 @@
[English](./CHANGELOG.md) | 中文
## 1.114.1 - 2026-05-08
### 修复
- `baoyu-danger-gemini-web`:修复当前 Gemini Web 响应中生成图 URL 以 `https://lh3.googleusercontent.com/gg-dl/` 形式出现、但不再包含旧版生成图 marker 时的图片提取失败问题。补充该响应形态的回归测试。 (by @evilstar2016)
## 1.114.0 - 2026-05-05
### 新功能
- `baoyu-infographic`:新增 `retro-popup-pop` 风格 —— 复古像素弹窗 × 波普信息图。画面由多个 80/90 年代桌面弹窗叠加而成(标题栏、关闭按钮、ERROR / ALERT 报错对话框、`PROBLEMS.EXE` 等复古文件窗、进度条、OK / CANCEL / FIX IT 按钮),统一粗黑描边、平涂色块,背景使用亮青蓝(#12B8DE)或复古奶油色(#F5F0E6)。与 `dense-modules` 布局尤其契合;同时升级为 `高密度信息大图` 关键词快捷方式与 `Product/Buying Guide` 内容类型的推荐风格。风格库从 21 个扩展至 22 个。
Credit to AJ@WaytoAGI.
### 文档
- `release-skills`:补充 GitHub Release 发布流程,包括从 changelog 段落提取 release notes、创建 annotated tag、执行 `gh release create/edit`,以及为已有 tag 回填历史 GitHub Releases。
## 1.113.0 - 2026-04-25
### 新功能
- `baoyu-imagine`:新增 DashScope Wan 2.7 图像模型支持(`wan2.7-image-pro``wan2.7-image`),通过阿里云百炼官方 API 直接调用,无需经 Replicate 转发。支持文生图、图像编辑、多图融合(最多 9 张参考图),按官方文档校验 `[1:8, 8:1]` 宽高比范围,并按模式应用不同的像素预算规则。强制 `parameters.n: 1` 以匹配 baoyu-imagine 的单图保存语义,显式拒绝 `--n > 1`,避免在用户不知情的情况下产生多图计费(API 在非拼图模式下默认 `n=4`)。允许通过 `--provider dashscope --ref ...` 显式启用 Wan 2.7 参考图工作流。
## 1.112.0 - 2026-04-24
### 新功能
- `baoyu-article-illustrator`:当内容分析未检测到明确信号时,将 `hand-drawn-edu`infographic + sketch-notes + macaron)作为通用默认预设 —— 暖奶油色纸面背景、黑色手绘线条、柔和马卡龙色块。`sketch-notes` 升级为 infographic / flowchart / comparison / framework 自动选择的首选风格;重写 sketch-notes 风格规范(马卡龙调色板、标准单页布局、仅限示意图的规则);新增对应的 prompt 模板块和默认工作流规则。
- `baoyu-article-illustrator`:新增 `hand-drawn-edu-flow`flowchart)和 `hand-drawn-edu-compare`(comparison)两个预设,保持相同的温暖教育风格。
### 破坏性变更
- `baoyu-article-illustrator``hand-drawn-edu` 预设的类型由 `flowchart` 改为 `infographic`。依赖原有流程图行为的用户请改用新增的 `hand-drawn-edu-flow` 预设。
### 修复
- `baoyu-post-to-x`:为 `scripts/md-to-html.ts` 添加入口守卫,确保 `x-article.ts` 导入 `parseMarkdown` 时不再触发 CLI 入口逻辑。与 `baoyu-post-to-weibo` 此前的修复保持一致。
## 1.111.1 - 2026-04-21
### 文档
- 为每个图片生成类技能(`baoyu-infographic``baoyu-cover-image``baoyu-slide-deck``baoyu-image-cards``baoyu-xhs-images``baoyu-article-illustrator`)新增顶级 `## Confirmation Policy` 章节作为单一事实源:显式调用技能、关键词快捷方式、EXTEND.md 偏好、自动推荐都只是"推荐输入",不授权跳过确认步骤。跳过确认必须由当前请求中的明确信号触发(`--no-confirm` / `--quick` / `--yes` / "直接生成" / 同义表达)。
- `baoyu-infographic`:将原先散落在 Step 5、Step 6、Default combination、Keyword Shortcuts 及 preferences 文档中的重复提醒合并为单一策略章节,由 Step 4 的 hard gate 引用。
## 1.111.0 - 2026-04-21
### 重构
- 统一所有图片生成类技能(`baoyu-infographic``baoyu-comic``baoyu-cover-image``baoyu-image-cards``baoyu-article-illustrator``baoyu-slide-deck``baoyu-xhs-images`)的后端选择规则:新增单一 `preferred_image_backend` 偏好字段(`auto | ask | <backend-id>`),用 4 步解析规则(当前请求覆盖 → 已保存偏好 → 自动选择 → 询问用户)替换原有的无状态询问规则。默认优先使用运行时原生工具(如 Codex `imagegen`、Hermes `image_generate`);未设置该字段的现有 `EXTEND.md` 文件视为 `auto`,无需升级 schema 版本。
- 在每个图片技能中新增顶级 `## Changing Preferences` 章节,作为固定后端和修改常用偏好的一级入口。
## 1.110.0 - 2026-04-21
### 新功能
- `baoyu-imagine`:新增 `gpt-image-2` 支持,用于 OpenAI 图像生成与编辑;将其设为默认 OpenAI 模型,并补齐官方尺寸/质量映射、自定义尺寸约束与 Azure 部署说明
## 1.109.0 - 2026-04-21
### 新功能
- `baoyu-url-to-markdown`:将 `baoyu-fetch` 运行时代码 vendored 到 `scripts/lib`,并通过本地 `scripts/baoyu-fetch` CLI 调用,使发布后的技能安装不再依赖 `baoyu-fetch` npm 包
### 修复
- `baoyu-fetch`:修复 X/Twitter 单条内容与 X Article 的视频解析,提取可播放的最高码率 MP4,并将文章视频渲染为 `[video](...)`
- `sync-clawhub`:改用共享 release 文件清单发布,确保无扩展名 CLI 入口、`bun.lock` 和 vendored `scripts/lib` 文件都会被上传
### 维护
-`defuddle` 升级到 0.17.0、`jsdom` 升级到 29.0.2,并通过 override 将 `@xmldom/xmldom` 固定到 0.8.13,清除 Defuddle 依赖链上的漏洞提示
## 1.108.0 - 2026-04-19
### 重构
@@ -592,7 +652,7 @@
## 1.52.0 - 2026-03-06
### 新功能
- `baoyu-post-to-weibo`:新增 `--video` 视频上传支持(图片+视频最多 18 个文件)
- `baoyu-post-to-weibo`:新增 `--video` 视频上传支持(图片 + 视频最多 18 个文件)
- `baoyu-post-to-weibo`:上传方式从剪贴板粘贴改为 `DOM.setFileInputFiles`,提升上传可靠性
### 修复
@@ -1055,7 +1115,7 @@
## 1.20.0 - 2026-01-24
### 新功能
- `baoyu-cover-image`:从类型 × 风格二维系统升级为**四维系统**——新增 `--text` 维度(none 无文字、title-only 仅标题、title-subtitle 标题+副标题、text-rich 丰富文字)控制文字密度,新增 `--mood` 维度(subtle 低调、balanced 平衡、bold 醒目)控制情感强度。新增 `--quick` 标志跳过确认,直接使用自动选择。
- `baoyu-cover-image`:从类型 × 风格二维系统升级为**四维系统**——新增 `--text` 维度(none 无文字、title-only 仅标题、title-subtitle 标题 + 副标题、text-rich 丰富文字)控制文字密度,新增 `--mood` 维度(subtle 低调、balanced 平衡、bold 醒目)控制情感强度。新增 `--quick` 标志跳过确认,直接使用自动选择。
### 文档
- `baoyu-cover-image`:新增维度参考文件——`references/dimensions/text.md`(文字密度级别)和 `references/dimensions/mood.md`(氛围强度级别)。
@@ -1075,7 +1135,7 @@
- `baoyu-image-gen`:代码模块化——类型定义提取至 `types.ts`provider 实现提取至 `providers/google.ts``providers/openai.ts`
### 文档
- `baoyu-comic`:改进 ohmsha 预设文档,明确默认哆啦A梦角色定义和视觉描述。
- `baoyu-comic`:改进 ohmsha 预设文档,明确默认哆啦 A 梦角色定义和视觉描述。
## 1.18.3 - 2026-01-23
+8 -8
View File
@@ -714,7 +714,7 @@ AI-powered generation backends.
#### baoyu-imagine
AI SDK-based image generation using OpenAI, Azure OpenAI, Google, OpenRouter, DashScope (Aliyun Tongyi Wanxiang), MiniMax, Jimeng (即梦), Seedream (豆包), and Replicate APIs. Supports text-to-image, reference images, aspect ratios, custom sizes, batch generation, and quality presets.
AI SDK-based image generation using OpenAI GPT Image 2, Azure OpenAI, Google, OpenRouter, DashScope (Aliyun Tongyi Wanxiang), MiniMax, Jimeng (即梦), Seedream (豆包), and Replicate APIs. Supports text-to-image, reference images, aspect ratios, custom sizes, batch generation, and quality presets.
```bash
# Basic generation (auto-detect provider)
@@ -727,10 +727,10 @@ AI SDK-based image generation using OpenAI, Azure OpenAI, Google, OpenRouter, Da
/baoyu-imagine --prompt "A banner" --image banner.png --quality 2k
# Specific provider
/baoyu-imagine --prompt "A cat" --image cat.png --provider openai
/baoyu-imagine --prompt "A cat" --image cat.png --provider openai --model gpt-image-2
# Azure OpenAI (model = deployment name)
/baoyu-imagine --prompt "A cat" --image cat.png --provider azure --model gpt-image-1.5
/baoyu-imagine --prompt "A cat" --image cat.png --provider azure --model gpt-image-2
# OpenRouter
/baoyu-imagine --prompt "A cat" --image cat.png --provider openrouter
@@ -786,7 +786,7 @@ AI SDK-based image generation using OpenAI, Azure OpenAI, Google, OpenRouter, Da
| `--provider` | `google`, `openai`, `azure`, `openrouter`, `dashscope`, `zai`, `minimax`, `jimeng`, `seedream`, or `replicate` |
| `--model`, `-m` | Model ID or deployment name. Azure uses deployment name; OpenRouter uses full model IDs; Z.AI uses `glm-image`; MiniMax uses `image-01` / `image-01-live` |
| `--ar` | Aspect ratio (e.g., `16:9`, `1:1`, `4:3`) |
| `--size` | Size (e.g., `1024x1024`) |
| `--size` | Size (e.g., `1024x1024`; `gpt-image-2` accepts valid custom sizes up to 3840px max edge) |
| `--quality` | `normal` or `2k` (default: `2k`) |
| `--imageSize` | `1K`, `2K`, or `4K` for Google/OpenRouter |
| `--imageApiDialect` | `openai-native` or `ratio-metadata` for OpenAI-compatible gateways |
@@ -810,9 +810,9 @@ AI SDK-based image generation using OpenAI, Azure OpenAI, Google, OpenRouter, Da
| `JIMENG_ACCESS_KEY_ID` | Jimeng Volcengine access key | - |
| `JIMENG_SECRET_ACCESS_KEY` | Jimeng Volcengine secret key | - |
| `ARK_API_KEY` | Seedream Volcengine ARK API key | - |
| `OPENAI_IMAGE_MODEL` | OpenAI model | `gpt-image-1.5` |
| `OPENAI_IMAGE_MODEL` | OpenAI model | `gpt-image-2` |
| `AZURE_OPENAI_DEPLOYMENT` | Azure default deployment name | - |
| `AZURE_OPENAI_IMAGE_MODEL` | Backward-compatible Azure deployment/model alias | `gpt-image-1.5` |
| `AZURE_OPENAI_IMAGE_MODEL` | Backward-compatible Azure deployment/model alias | `gpt-image-2` |
| `OPENROUTER_IMAGE_MODEL` | OpenRouter model | `google/gemini-3.1-flash-image-preview` |
| `GOOGLE_IMAGE_MODEL` | Google model | `gemini-3-pro-image-preview` |
| `DASHSCOPE_IMAGE_MODEL` | DashScope model | `qwen-image-2.0-pro` |
@@ -1132,14 +1132,14 @@ mkdir -p ~/.baoyu-skills
cat > ~/.baoyu-skills/.env << 'EOF'
# OpenAI
OPENAI_API_KEY=sk-xxx
OPENAI_IMAGE_MODEL=gpt-image-1.5
OPENAI_IMAGE_MODEL=gpt-image-2
# OPENAI_BASE_URL=https://api.openai.com/v1
# OPENAI_IMAGE_USE_CHAT=false
# Azure OpenAI
AZURE_OPENAI_API_KEY=xxx
AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-image-1.5
AZURE_OPENAI_DEPLOYMENT=gpt-image-2
# AZURE_API_VERSION=2025-04-01-preview
# OpenRouter
+8 -8
View File
@@ -714,7 +714,7 @@ AI 驱动的生成后端。
#### baoyu-imagine
基于 AI SDK 的图像生成,支持 OpenAI、Azure OpenAI、Google、OpenRouter、DashScope(阿里通义万相)、MiniMax、即梦(Jimeng)、豆包(Seedream)和 Replicate API。支持文生图、参考图、宽高比、自定义尺寸、批量生成和质量预设。
基于 AI SDK 的图像生成,支持 OpenAI GPT Image 2、Azure OpenAI、Google、OpenRouter、DashScope(阿里通义万相)、MiniMax、即梦(Jimeng)、豆包(Seedream)和 Replicate API。支持文生图、参考图、宽高比、自定义尺寸、批量生成和质量预设。
```bash
# 基础生成(自动检测服务商)
@@ -727,10 +727,10 @@ AI 驱动的生成后端。
/baoyu-imagine --prompt "横幅图" --image banner.png --quality 2k
# 指定服务商
/baoyu-imagine --prompt "一只猫" --image cat.png --provider openai
/baoyu-imagine --prompt "一只猫" --image cat.png --provider openai --model gpt-image-2
# Azure OpenAImodel 为部署名称)
/baoyu-imagine --prompt "一只猫" --image cat.png --provider azure --model gpt-image-1.5
/baoyu-imagine --prompt "一只猫" --image cat.png --provider azure --model gpt-image-2
# OpenRouter
/baoyu-imagine --prompt "一只猫" --image cat.png --provider openrouter
@@ -786,7 +786,7 @@ AI 驱动的生成后端。
| `--provider` | `google``openai``azure``openrouter``dashscope``zai``minimax``jimeng``seedream``replicate` |
| `--model`, `-m` | 模型 ID 或部署名。Azure 使用部署名;OpenRouter 使用完整模型 IDZ.AI 使用 `glm-image`MiniMax 使用 `image-01` / `image-01-live` |
| `--ar` | 宽高比(如 `16:9``1:1``4:3` |
| `--size` | 尺寸(如 `1024x1024` |
| `--size` | 尺寸(如 `1024x1024``gpt-image-2` 支持最长边不超过 3840px 的有效自定义尺寸 |
| `--quality` | `normal``2k`(默认:`2k` |
| `--imageSize` | Google/OpenRouter 使用的 `1K``2K``4K` |
| `--imageApiDialect` | OpenAI 兼容网关的图像 API 方言(`openai-native``ratio-metadata` |
@@ -810,9 +810,9 @@ AI 驱动的生成后端。
| `JIMENG_ACCESS_KEY_ID` | 即梦火山引擎 Access Key | - |
| `JIMENG_SECRET_ACCESS_KEY` | 即梦火山引擎 Secret Key | - |
| `ARK_API_KEY` | 豆包火山引擎 ARK API 密钥 | - |
| `OPENAI_IMAGE_MODEL` | OpenAI 模型 | `gpt-image-1.5` |
| `OPENAI_IMAGE_MODEL` | OpenAI 模型 | `gpt-image-2` |
| `AZURE_OPENAI_DEPLOYMENT` | Azure 默认部署名 | - |
| `AZURE_OPENAI_IMAGE_MODEL` | 兼容旧配置的 Azure 部署/模型别名 | `gpt-image-1.5` |
| `AZURE_OPENAI_IMAGE_MODEL` | 兼容旧配置的 Azure 部署/模型别名 | `gpt-image-2` |
| `OPENROUTER_IMAGE_MODEL` | OpenRouter 模型 | `google/gemini-3.1-flash-image-preview` |
| `GOOGLE_IMAGE_MODEL` | Google 模型 | `gemini-3-pro-image-preview` |
| `DASHSCOPE_IMAGE_MODEL` | DashScope 模型 | `qwen-image-2.0-pro` |
@@ -1132,14 +1132,14 @@ mkdir -p ~/.baoyu-skills
cat > ~/.baoyu-skills/.env << 'EOF'
# OpenAI
OPENAI_API_KEY=sk-xxx
OPENAI_IMAGE_MODEL=gpt-image-1.5
OPENAI_IMAGE_MODEL=gpt-image-2
# OPENAI_BASE_URL=https://api.openai.com/v1
# OPENAI_IMAGE_USE_CHAT=false
# Azure OpenAI
AZURE_OPENAI_API_KEY=xxx
AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-image-1.5
AZURE_OPENAI_DEPLOYMENT=gpt-image-2
# AZURE_API_VERSION=2025-04-01-preview
# OpenRouter
BIN
View File
Binary file not shown.
+23 -5
View File
@@ -4,13 +4,29 @@ Skills in this repo are loaded by multiple agent runtimes (Claude Code, Codex, H
## The Rule
When a skill needs to render an image:
When a skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if the skill's `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
No explicit priority between runtime-native tools and repo skills — treat them equivalently and let the user decide when there's a choice. No persisted preference mechanism; the question is cheap, and the rule is stateless.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends.
## The Preference Field
Each image-consuming skill's `EXTEND.md` carries a single `preferred_image_backend` field:
| Value | Meaning |
|---|---|
| `auto` (default) | Apply the auto-select rule — runtime-native preferred, fall back to only installed backend, ask if multiple non-native. |
| `ask` | Always confirm the backend on every run, even when a runtime-native tool exists. |
| `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) | Pin this backend when available; fall back to `auto` if it isn't. |
The field is **absent-equals-auto**: older `EXTEND.md` files without this field behave exactly as if `preferred_image_backend: auto` were set. No schema version bump is needed to introduce it.
## Prompt File Requirement (hard)
@@ -20,6 +36,8 @@ Regardless of which backend is chosen, every skill that renders images MUST writ
Each `SKILL.md` that renders images includes **exactly one** `## Image Generation Tools` section (near the top, after `## User Input Tools` and before the main workflow) that **inlines** this rule. Skills are self-contained and cannot link to `docs/` — each skill folder must ship the rule inside its own `SKILL.md`. See [CLAUDE.md → Skill Self-Containment](../CLAUDE.md).
Each skill's `references/config/preferences-schema.md` (and its `EXTEND.md` template in `first-time-setup.md`) lists `preferred_image_backend` alongside other preference fields. First-time setup does NOT ask the user about the backend — `auto` is set silently. Users who want to pin a specific backend edit `EXTEND.md` later, and each skill's `## Changing Preferences` section documents the common one-line edits.
Concrete tool names (`imagegen`, `image_generate`, `baoyu-imagine`) in this document and in SKILL.md are **examples** — agents in other runtimes apply the rule above and substitute the local equivalent. Skill-specific parameters for these backends are illustrative; runtimes without those knobs can omit them.
## Backend Skills Are Exempt
+4 -4
View File
@@ -21,18 +21,18 @@ bash scripts/sync-clawhub.sh # sync all skills
bash scripts/sync-clawhub.sh <skill> # sync one skill
```
Release hooks are configured via `.releaserc.yml`. This repo does not stage a separate release directory: release prep verifies that skills depend on published npm package versions, and publish reads the skill directory directly.
Release hooks are configured via `.releaserc.yml`. This repo does not stage a separate release directory: publish reads the skill directory directly and validates that local package references and CLI bin targets are self-contained.
## Shared Workspace Packages
`packages/` is the **only** source of truth for shared runtime code. Publish shared packages to npm and reference them from skill script `package.json` files with semver ranges. Do not vendor shared packages into `skills/*/scripts/vendor/`.
`packages/` is the source of truth for shared runtime code. Most skills consume shared packages from npm with semver ranges. `baoyu-url-to-markdown` is the exception: it vendors the `baoyu-fetch` runtime into `skills/baoyu-url-to-markdown/scripts/lib/` so the published skill is self-contained and does not depend on the `baoyu-fetch` npm package.
Current packages:
- `baoyu-chrome-cdp` (Chrome CDP utilities), consumed by 5 skills (`baoyu-danger-gemini-web`, `baoyu-danger-x-to-markdown`, `baoyu-post-to-wechat`, `baoyu-post-to-weibo`, `baoyu-post-to-x`)
- `baoyu-md` (shared Markdown rendering and placeholder pipeline), consumed by 3 skills (`baoyu-markdown-to-html`, `baoyu-post-to-wechat`, `baoyu-post-to-weibo`)
- `baoyu-fetch` (URL-to-Markdown CLI), consumed by 1 skill (`baoyu-url-to-markdown`)
- `baoyu-fetch` (URL-to-Markdown CLI), vendored into 1 skill (`baoyu-url-to-markdown`)
**How it works**: npm packages are built from `packages/` and published to the public npm registry. Skills depend on those packages with `^<version>` specs. Release prep runs `node scripts/verify-shared-package-deps.mjs` so `file:` dependencies and vendored workspace packages cannot slip back in.
**How it works**: npm packages are built from `packages/` and published to the public npm registry. Skills normally depend on those packages with `^<version>` specs. Release prep runs `node scripts/verify-shared-package-deps.mjs` so accidental `file:` dependencies cannot slip back in. For vendored skill runtimes, keep the copied code under the skill directory and run `node scripts/publish-skill.mjs --skill-dir <skill> --version <version> --dry-run` before publishing.
**Update workflow**:
1. Edit package under `packages/`
+287 -158
View File
@@ -22,18 +22,52 @@
}
},
"node_modules/@asamuzakjp/css-color": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-3.2.0.tgz",
"integrity": "sha512-K1A6z8tS3XsmCMM86xoWdn7Fkdn9m6RSVtocUrJYIwZnFVkng/PvkEoWtOWmP+Scc6saYWHWZYbndEEXxl24jw==",
"version": "5.1.11",
"resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-5.1.11.tgz",
"integrity": "sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg==",
"license": "MIT",
"dependencies": {
"@csstools/css-calc": "^2.1.3",
"@csstools/css-color-parser": "^3.0.9",
"@csstools/css-parser-algorithms": "^3.0.4",
"@csstools/css-tokenizer": "^3.0.3",
"lru-cache": "^10.4.3"
"@asamuzakjp/generational-cache": "^1.0.1",
"@csstools/css-calc": "^3.2.0",
"@csstools/css-color-parser": "^4.1.0",
"@csstools/css-parser-algorithms": "^4.0.0",
"@csstools/css-tokenizer": "^4.0.0"
},
"engines": {
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/@asamuzakjp/dom-selector": {
"version": "7.1.1",
"resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-7.1.1.tgz",
"integrity": "sha512-67RZDnYRc8H/8MLDgQCDE//zoqVFwajkepHZgmXrbwybzXOEwOWGPYGmALYl9J2DOLfFPPs6kKCqmbzV895hTQ==",
"license": "MIT",
"dependencies": {
"@asamuzakjp/generational-cache": "^1.0.1",
"@asamuzakjp/nwsapi": "^2.3.9",
"bidi-js": "^1.0.3",
"css-tree": "^3.2.1",
"is-potential-custom-element-name": "^1.0.1"
},
"engines": {
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/@asamuzakjp/generational-cache": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@asamuzakjp/generational-cache/-/generational-cache-1.0.1.tgz",
"integrity": "sha512-wajfB8KqzMCN2KGNFdLkReeHncd0AslUSrvHVvvYWuU8ghncRJoA50kT3zP9MVL0+9g4/67H+cdvBskj9THPzg==",
"license": "MIT",
"engines": {
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/@asamuzakjp/nwsapi": {
"version": "2.3.9",
"resolved": "https://registry.npmjs.org/@asamuzakjp/nwsapi/-/nwsapi-2.3.9.tgz",
"integrity": "sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q==",
"license": "MIT"
},
"node_modules/@babel/runtime": {
"version": "7.29.2",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz",
@@ -44,6 +78,18 @@
"node": ">=6.9.0"
}
},
"node_modules/@bramus/specificity": {
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/@bramus/specificity/-/specificity-2.4.2.tgz",
"integrity": "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw==",
"license": "MIT",
"dependencies": {
"css-tree": "^3.0.0"
},
"bin": {
"specificity": "bin/cli.js"
}
},
"node_modules/@changesets/apply-release-plan": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/@changesets/apply-release-plan/-/apply-release-plan-7.1.0.tgz",
@@ -307,9 +353,9 @@
}
},
"node_modules/@csstools/color-helpers": {
"version": "5.1.0",
"resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-5.1.0.tgz",
"integrity": "sha512-S11EXWJyy0Mz5SYvRmY8nJYTFFd1LCNV+7cXyAgQtOOuzb4EsgfqDufL+9esx72/eLhsRdGZwaldu/h+E4t4BA==",
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-6.0.2.tgz",
"integrity": "sha512-LMGQLS9EuADloEFkcTBR3BwV/CGHV7zyDxVRtVDTwdI2Ca4it0CCVTT9wCkxSgokjE5Ho41hEPgb8OEUwoXr6Q==",
"funding": [
{
"type": "github",
@@ -322,13 +368,13 @@
],
"license": "MIT-0",
"engines": {
"node": ">=18"
"node": ">=20.19.0"
}
},
"node_modules/@csstools/css-calc": {
"version": "2.1.4",
"resolved": "https://registry.npmjs.org/@csstools/css-calc/-/css-calc-2.1.4.tgz",
"integrity": "sha512-3N8oaj+0juUw/1H3YwmDDJXCgTB1gKU6Hc/bB502u9zR0q2vd786XJH9QfrKIEgFlZmhZiq6epXl4rHqhzsIgQ==",
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/@csstools/css-calc/-/css-calc-3.2.0.tgz",
"integrity": "sha512-bR9e6o2BDB12jzN/gIbjHa5wLJ4UjD1CB9pM7ehlc0ddk6EBz+yYS1EV2MF55/HUxrHcB/hehAyt5vhsA3hx7w==",
"funding": [
{
"type": "github",
@@ -341,17 +387,17 @@
],
"license": "MIT",
"engines": {
"node": ">=18"
"node": ">=20.19.0"
},
"peerDependencies": {
"@csstools/css-parser-algorithms": "^3.0.5",
"@csstools/css-tokenizer": "^3.0.4"
"@csstools/css-parser-algorithms": "^4.0.0",
"@csstools/css-tokenizer": "^4.0.0"
}
},
"node_modules/@csstools/css-color-parser": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/@csstools/css-color-parser/-/css-color-parser-3.1.0.tgz",
"integrity": "sha512-nbtKwh3a6xNVIp/VRuXV64yTKnb1IjTAEEh3irzS+HkKjAOYLTGNb9pmVNntZ8iVBHcWDA2Dof0QtPgFI1BaTA==",
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/@csstools/css-color-parser/-/css-color-parser-4.1.0.tgz",
"integrity": "sha512-U0KhLYmy2GVj6q4T3WaAe6NPuFYCPQoE3b0dRGxejWDgcPp8TP7S5rVdM5ZrFaqu4N67X8YaPBw14dQSYx3IyQ==",
"funding": [
{
"type": "github",
@@ -364,21 +410,21 @@
],
"license": "MIT",
"dependencies": {
"@csstools/color-helpers": "^5.1.0",
"@csstools/css-calc": "^2.1.4"
"@csstools/color-helpers": "^6.0.2",
"@csstools/css-calc": "^3.2.0"
},
"engines": {
"node": ">=18"
"node": ">=20.19.0"
},
"peerDependencies": {
"@csstools/css-parser-algorithms": "^3.0.5",
"@csstools/css-tokenizer": "^3.0.4"
"@csstools/css-parser-algorithms": "^4.0.0",
"@csstools/css-tokenizer": "^4.0.0"
}
},
"node_modules/@csstools/css-parser-algorithms": {
"version": "3.0.5",
"resolved": "https://registry.npmjs.org/@csstools/css-parser-algorithms/-/css-parser-algorithms-3.0.5.tgz",
"integrity": "sha512-DaDeUkXZKjdGhgYaHNJTV9pV7Y9B3b644jCLs9Upc3VeNGg6LWARAT6O+Q+/COo+2gg/bM5rhpMAtf70WqfBdQ==",
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@csstools/css-parser-algorithms/-/css-parser-algorithms-4.0.0.tgz",
"integrity": "sha512-+B87qS7fIG3L5h3qwJ/IFbjoVoOe/bpOdh9hAjXbvx0o8ImEmUsGXN0inFOnk2ChCFgqkkGFQ+TpM5rbhkKe4w==",
"funding": [
{
"type": "github",
@@ -391,16 +437,40 @@
],
"license": "MIT",
"engines": {
"node": ">=18"
"node": ">=20.19.0"
},
"peerDependencies": {
"@csstools/css-tokenizer": "^3.0.4"
"@csstools/css-tokenizer": "^4.0.0"
}
},
"node_modules/@csstools/css-syntax-patches-for-csstree": {
"version": "1.1.3",
"resolved": "https://registry.npmjs.org/@csstools/css-syntax-patches-for-csstree/-/css-syntax-patches-for-csstree-1.1.3.tgz",
"integrity": "sha512-SH60bMfrRCJF3morcdk57WklujF4Jr/EsQUzqkarfHXEFcAR1gg7fS/chAE922Sehgzc1/+Tz5H3Ypa1HiEKrg==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/csstools"
},
{
"type": "opencollective",
"url": "https://opencollective.com/csstools"
}
],
"license": "MIT-0",
"peerDependencies": {
"css-tree": "^3.2.1"
},
"peerDependenciesMeta": {
"css-tree": {
"optional": true
}
}
},
"node_modules/@csstools/css-tokenizer": {
"version": "3.0.4",
"resolved": "https://registry.npmjs.org/@csstools/css-tokenizer/-/css-tokenizer-3.0.4.tgz",
"integrity": "sha512-Vd/9EVDiu6PPJt9yAh6roZP6El1xHrdvIVGjyBsHR0RYwNHgL7FJPyIIW4fANJNG6FtyZfvlRPpFI4ZM/lubvw==",
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@csstools/css-tokenizer/-/css-tokenizer-4.0.0.tgz",
"integrity": "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA==",
"funding": [
{
"type": "github",
@@ -413,7 +483,7 @@
],
"license": "MIT",
"engines": {
"node": ">=18"
"node": ">=20.19.0"
}
},
"node_modules/@emnapi/runtime": {
@@ -875,6 +945,23 @@
"node": ">=18"
}
},
"node_modules/@exodus/bytes": {
"version": "1.15.0",
"resolved": "https://registry.npmjs.org/@exodus/bytes/-/bytes-1.15.0.tgz",
"integrity": "sha512-UY0nlA+feH81UGSHv92sLEPLCeZFjXOuHhrIo0HQydScuQc8s0A7kL/UdgwgDq8g8ilksmuoF35YVTNphV2aBQ==",
"license": "MIT",
"engines": {
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
},
"peerDependencies": {
"@noble/hashes": "^1.8.0 || ^2.0.0"
},
"peerDependenciesMeta": {
"@noble/hashes": {
"optional": true
}
}
},
"node_modules/@img/colour": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz",
@@ -1639,24 +1726,15 @@
}
},
"node_modules/@xmldom/xmldom": {
"version": "0.8.11",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.11.tgz",
"integrity": "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw==",
"version": "0.8.13",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.13.tgz",
"integrity": "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw==",
"license": "MIT",
"optional": true,
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/agent-base": {
"version": "7.1.4",
"resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz",
"integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==",
"license": "MIT",
"engines": {
"node": ">= 14"
}
},
"node_modules/ansi-colors": {
"version": "4.1.3",
"resolved": "https://registry.npmjs.org/ansi-colors/-/ansi-colors-4.1.3.tgz",
@@ -1730,6 +1808,15 @@
"node": ">=4"
}
},
"node_modules/bidi-js": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz",
"integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==",
"license": "MIT",
"dependencies": {
"require-from-string": "^2.0.2"
}
},
"node_modules/boolbase": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz",
@@ -1892,6 +1979,19 @@
"url": "https://github.com/sponsors/fb55"
}
},
"node_modules/css-tree": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz",
"integrity": "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA==",
"license": "MIT",
"dependencies": {
"mdn-data": "2.27.1",
"source-map-js": "^1.2.1"
},
"engines": {
"node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0"
}
},
"node_modules/css-what": {
"version": "6.2.2",
"resolved": "https://registry.npmjs.org/css-what/-/css-what-6.2.2.tgz",
@@ -1911,30 +2011,26 @@
"devOptional": true,
"license": "MIT"
},
"node_modules/cssstyle": {
"version": "4.6.0",
"resolved": "https://registry.npmjs.org/cssstyle/-/cssstyle-4.6.0.tgz",
"integrity": "sha512-2z+rWdzbbSZv6/rhtvzvqeZQHrBaqgogqt85sqFNbabZOuFbCVFb8kPeEtZjiKkbrm395irpNKiYeFeLiQnFPg==",
"node_modules/data-urls": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz",
"integrity": "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA==",
"license": "MIT",
"dependencies": {
"@asamuzakjp/css-color": "^3.2.0",
"rrweb-cssom": "^0.8.0"
"whatwg-mimetype": "^5.0.0",
"whatwg-url": "^16.0.0"
},
"engines": {
"node": ">=18"
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/data-urls": {
"node_modules/data-urls/node_modules/whatwg-mimetype": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/data-urls/-/data-urls-5.0.0.tgz",
"integrity": "sha512-ZYP5VBHshaDAiVZxjbRVcFJpc+4xGgT0bK3vzy1HLN8jTO975HEbuYzZJcHoQEY5K1a0z8YayJkyVETa08eNTg==",
"resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz",
"integrity": "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==",
"license": "MIT",
"dependencies": {
"whatwg-mimetype": "^4.0.0",
"whatwg-url": "^14.0.0"
},
"engines": {
"node": ">=18"
"node": ">=20"
}
},
"node_modules/debug": {
@@ -1974,9 +2070,9 @@
}
},
"node_modules/defuddle": {
"version": "0.14.0",
"resolved": "https://registry.npmjs.org/defuddle/-/defuddle-0.14.0.tgz",
"integrity": "sha512-btavZGd1WgiVqrVM62WGRXMUi/aU7ckTZiq0xXWLZMHvzIqNZjwIFQEDRx8MarD7fIgsB90NXZ9xHJkKtapt2Q==",
"version": "0.17.0",
"resolved": "https://registry.npmjs.org/defuddle/-/defuddle-0.17.0.tgz",
"integrity": "sha512-rjRb9kxgrJLFhFJjXTSAitbmmSyvjEqGCi5JiyM6ImOf4leC/O+f7TPVPAymk/gnENFTrn9j7T/LTfW9IVwiLw==",
"license": "MIT",
"dependencies": {
"commander": "^12.1.0"
@@ -2418,15 +2514,15 @@
}
},
"node_modules/html-encoding-sniffer": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-4.0.0.tgz",
"integrity": "sha512-Y22oTqIU4uuPgEemfz7NDJz6OeKf12Lsu+QC+s3BVpda64lTiMYCyGwg5ki4vFxkMwQdeZDl2adZoqUgdFuTgQ==",
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-6.0.0.tgz",
"integrity": "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg==",
"license": "MIT",
"dependencies": {
"whatwg-encoding": "^3.1.1"
"@exodus/bytes": "^1.6.0"
},
"engines": {
"node": ">=18"
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/html-escaper": {
@@ -2467,38 +2563,12 @@
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/http-proxy-agent": {
"version": "7.0.2",
"resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz",
"integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==",
"license": "MIT",
"dependencies": {
"agent-base": "^7.1.0",
"debug": "^4.3.4"
},
"engines": {
"node": ">= 14"
}
},
"node_modules/https": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/https/-/https-1.0.0.tgz",
"integrity": "sha512-4EC57ddXrkaF0x83Oj8sM6SLQHAWXw90Skqu2M4AEWENZ3F02dFJE/GARA8igO79tcgYqGrD7ae4f5L3um2lgg==",
"license": "ISC"
},
"node_modules/https-proxy-agent": {
"version": "7.0.6",
"resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz",
"integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==",
"license": "MIT",
"dependencies": {
"agent-base": "^7.1.2",
"debug": "4"
},
"engines": {
"node": ">= 14"
}
},
"node_modules/human-id": {
"version": "4.1.3",
"resolved": "https://registry.npmjs.org/human-id/-/human-id-4.1.3.tgz",
@@ -2686,34 +2756,35 @@
}
},
"node_modules/jsdom": {
"version": "26.1.0",
"resolved": "https://registry.npmjs.org/jsdom/-/jsdom-26.1.0.tgz",
"integrity": "sha512-Cvc9WUhxSMEo4McES3P7oK3QaXldCfNWp7pl2NNeiIFlCoLr3kfq9kb1fxftiwk1FLV7CvpvDfonxtzUDeSOPg==",
"version": "29.0.2",
"resolved": "https://registry.npmjs.org/jsdom/-/jsdom-29.0.2.tgz",
"integrity": "sha512-9VnGEBosc/ZpwyOsJBCQ/3I5p7Q5ngOY14a9bf5btenAORmZfDse1ZEheMiWcJ3h81+Fv7HmJFdS0szo/waF2w==",
"license": "MIT",
"dependencies": {
"cssstyle": "^4.2.1",
"data-urls": "^5.0.0",
"decimal.js": "^10.5.0",
"html-encoding-sniffer": "^4.0.0",
"http-proxy-agent": "^7.0.2",
"https-proxy-agent": "^7.0.6",
"@asamuzakjp/css-color": "^5.1.5",
"@asamuzakjp/dom-selector": "^7.0.6",
"@bramus/specificity": "^2.4.2",
"@csstools/css-syntax-patches-for-csstree": "^1.1.1",
"@exodus/bytes": "^1.15.0",
"css-tree": "^3.2.1",
"data-urls": "^7.0.0",
"decimal.js": "^10.6.0",
"html-encoding-sniffer": "^6.0.0",
"is-potential-custom-element-name": "^1.0.1",
"nwsapi": "^2.2.16",
"parse5": "^7.2.1",
"rrweb-cssom": "^0.8.0",
"lru-cache": "^11.2.7",
"parse5": "^8.0.0",
"saxes": "^6.0.0",
"symbol-tree": "^3.2.4",
"tough-cookie": "^5.1.1",
"tough-cookie": "^6.0.1",
"undici": "^7.24.5",
"w3c-xmlserializer": "^5.0.0",
"webidl-conversions": "^7.0.0",
"whatwg-encoding": "^3.1.1",
"whatwg-mimetype": "^4.0.0",
"whatwg-url": "^14.1.1",
"ws": "^8.18.0",
"webidl-conversions": "^8.0.1",
"whatwg-mimetype": "^5.0.0",
"whatwg-url": "^16.0.1",
"xml-name-validator": "^5.0.0"
},
"engines": {
"node": ">=18"
"node": "^20.19.0 || ^22.13.0 || >=24.0.0"
},
"peerDependencies": {
"canvas": "^3.0.0"
@@ -2724,6 +2795,48 @@
}
}
},
"node_modules/jsdom/node_modules/entities": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/entities/-/entities-8.0.0.tgz",
"integrity": "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=20.19.0"
},
"funding": {
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/jsdom/node_modules/parse5": {
"version": "8.0.1",
"resolved": "https://registry.npmjs.org/parse5/-/parse5-8.0.1.tgz",
"integrity": "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw==",
"license": "MIT",
"dependencies": {
"entities": "^8.0.0"
},
"funding": {
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/jsdom/node_modules/undici": {
"version": "7.25.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.25.0.tgz",
"integrity": "sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ==",
"license": "MIT",
"engines": {
"node": ">=20.18.1"
}
},
"node_modules/jsdom/node_modules/whatwg-mimetype": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz",
"integrity": "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==",
"license": "MIT",
"engines": {
"node": ">=20"
}
},
"node_modules/jsonfile": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-4.0.0.tgz",
@@ -2861,10 +2974,13 @@
}
},
"node_modules/lru-cache": {
"version": "10.4.3",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz",
"integrity": "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==",
"license": "ISC"
"version": "11.3.5",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.3.5.tgz",
"integrity": "sha512-NxVFwLAnrd9i7KUBxC4DrUhmgjzOs+1Qm50D3oF1/oL+r1NpZ4gA7xvG0/zJ8evR7zIKn4vLf7qTNduWFtCrRw==",
"license": "BlueOak-1.0.0",
"engines": {
"node": "20 || >=22"
}
},
"node_modules/markdown-table": {
"version": "3.0.4",
@@ -3105,6 +3221,12 @@
"url": "https://opencollective.com/unified"
}
},
"node_modules/mdn-data": {
"version": "2.27.1",
"resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz",
"integrity": "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ==",
"license": "CC0-1.0"
},
"node_modules/mensch": {
"version": "0.3.4",
"resolved": "https://registry.npmjs.org/mensch/-/mensch-0.3.4.tgz",
@@ -3782,12 +3904,6 @@
"url": "https://github.com/fb55/nth-check?sponsor=1"
}
},
"node_modules/nwsapi": {
"version": "2.2.23",
"resolved": "https://registry.npmjs.org/nwsapi/-/nwsapi-2.2.23.tgz",
"integrity": "sha512-7wfH4sLbt4M0gCDzGE6vzQBo0bfTKjU7Sfpqy/7gs1qBfYz2vEJH6vXcBKpO3+6Yu1telwd0t9HpyOoLEQQbIQ==",
"license": "MIT"
},
"node_modules/outdent": {
"version": "0.5.0",
"resolved": "https://registry.npmjs.org/outdent/-/outdent-0.5.0.tgz",
@@ -4200,6 +4316,15 @@
"url": "https://opencollective.com/unified"
}
},
"node_modules/require-from-string": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
"integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/resolve-from": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-5.0.0.tgz",
@@ -4231,12 +4356,6 @@
"node": ">=0.10.0"
}
},
"node_modules/rrweb-cssom": {
"version": "0.8.0",
"resolved": "https://registry.npmjs.org/rrweb-cssom/-/rrweb-cssom-0.8.0.tgz",
"integrity": "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==",
"license": "MIT"
},
"node_modules/run-parallel": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/run-parallel/-/run-parallel-1.2.0.tgz",
@@ -4402,6 +4521,15 @@
"node": "*"
}
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
"integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
"license": "BSD-3-Clause",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/spawndamnit": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/spawndamnit/-/spawndamnit-3.0.1.tgz",
@@ -4481,21 +4609,21 @@
}
},
"node_modules/tldts": {
"version": "6.1.86",
"resolved": "https://registry.npmjs.org/tldts/-/tldts-6.1.86.tgz",
"integrity": "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ==",
"version": "7.0.28",
"resolved": "https://registry.npmjs.org/tldts/-/tldts-7.0.28.tgz",
"integrity": "sha512-+Zg3vWhRUv8B1maGSTFdev9mjoo8Etn2Ayfs4cnjlD3CsGkxXX4QyW3j2WJ0wdjYcYmy7Lx2RDsZMhgCWafKIw==",
"license": "MIT",
"dependencies": {
"tldts-core": "^6.1.86"
"tldts-core": "^7.0.28"
},
"bin": {
"tldts": "bin/cli.js"
}
},
"node_modules/tldts-core": {
"version": "6.1.86",
"resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-6.1.86.tgz",
"integrity": "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA==",
"version": "7.0.28",
"resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-7.0.28.tgz",
"integrity": "sha512-7W5Efjhsc3chVdFhqtaU0KtK32J37Zcr9RKtID54nG+tIpcY79CQK/veYPODxtD/LJ4Lue66jvrQzIX2Z2/pUQ==",
"license": "MIT"
},
"node_modules/to-regex-range": {
@@ -4512,27 +4640,27 @@
}
},
"node_modules/tough-cookie": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz",
"integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==",
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-6.0.1.tgz",
"integrity": "sha512-LktZQb3IeoUWB9lqR5EWTHgW/VTITCXg4D21M+lvybRVdylLrRMnqaIONLVb5mav8vM19m44HIcGq4qASeu2Qw==",
"license": "BSD-3-Clause",
"dependencies": {
"tldts": "^6.1.32"
"tldts": "^7.0.5"
},
"engines": {
"node": ">=16"
}
},
"node_modules/tr46": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/tr46/-/tr46-5.1.1.tgz",
"integrity": "sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw==",
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/tr46/-/tr46-6.0.0.tgz",
"integrity": "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw==",
"license": "MIT",
"dependencies": {
"punycode": "^2.3.1"
},
"engines": {
"node": ">=18"
"node": ">=20"
}
},
"node_modules/trough": {
@@ -4778,12 +4906,12 @@
}
},
"node_modules/webidl-conversions": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-7.0.0.tgz",
"integrity": "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g==",
"version": "8.0.1",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz",
"integrity": "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=12"
"node": ">=20"
}
},
"node_modules/whatwg-encoding": {
@@ -4809,16 +4937,17 @@
}
},
"node_modules/whatwg-url": {
"version": "14.2.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-14.2.0.tgz",
"integrity": "sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw==",
"version": "16.0.1",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-16.0.1.tgz",
"integrity": "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw==",
"license": "MIT",
"dependencies": {
"tr46": "^5.1.0",
"webidl-conversions": "^7.0.0"
"@exodus/bytes": "^1.11.0",
"tr46": "^6.0.0",
"webidl-conversions": "^8.0.1"
},
"engines": {
"node": ">=18"
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/which": {
@@ -4894,8 +5023,8 @@
"dependencies": {
"@mozilla/readability": "^0.6.0",
"chrome-launcher": "^1.2.1",
"defuddle": "^0.14.0",
"jsdom": "^26.0.0",
"defuddle": "^0.17.0",
"jsdom": "^29.0.2",
"remark-gfm": "^4.0.1",
"remark-parse": "^11.0.0",
"turndown": "^7.2.0",
+3
View File
@@ -20,5 +20,8 @@
"pdf-lib": "^1.17.1",
"pptxgenjs": "^4.0.1",
"sharp": "^0.34.5"
},
"overrides": {
"@xmldom/xmldom": "0.8.13"
}
}
+15
View File
@@ -5,6 +5,21 @@ English | [简体中文](./CHANGELOG.zh-CN.md)
The format is inspired by Keep a Changelog, and the project follows Semantic
Versioning.
## [0.1.2] - 2026-04-21
#### Changed
- Upgraded Defuddle to 0.17.0 and jsdom to 29.0.2 for generic extraction.
- Added an `@xmldom/xmldom` override to keep the optional Defuddle MathML
dependency chain on a non-vulnerable release.
#### Fixed
- Fixed X/Twitter video extraction for single posts and X Articles by selecting
the highest-bitrate MP4 variant instead of the preview image URL.
- Fixed X Article media rendering so video entities are emitted as
`[video](...)` links instead of image embeds.
## [0.1.1] - 2026-03-27
#### Added
+12
View File
@@ -4,6 +4,18 @@
格式参考 Keep a Changelog,版本号遵循 Semantic Versioning。
## [0.1.2] - 2026-04-21
### 变更
- 将 Defuddle 升级到 0.17.0、jsdom 升级到 29.0.2,用于通用页面提取。
- 新增 `@xmldom/xmldom` override,使 Defuddle 的可选 MathML 依赖链保持在无漏洞版本。
### 修复
- 修复 X/Twitter 单条内容和 X Article 的视频提取逻辑,改为选择最高码率 MP4 变体,而不是预览图 URL。
- 修复 X Article 媒体渲染,视频实体现在输出为 `[video](...)` 链接,而不是图片嵌入。
## [0.1.1] - 2026-03-27
### 新增
+52 -36
View File
@@ -1,13 +1,14 @@
{
"lockfileVersion": 1,
"configVersion": 0,
"workspaces": {
"": {
"name": "baoyu-fetch",
"dependencies": {
"@mozilla/readability": "^0.6.0",
"chrome-launcher": "^1.2.1",
"defuddle": "^0.14.0",
"jsdom": "^26.0.0",
"defuddle": "^0.17.0",
"jsdom": "^29.0.2",
"remark-gfm": "^4.0.1",
"remark-parse": "^11.0.0",
"turndown": "^7.2.0",
@@ -24,11 +25,22 @@
},
},
},
"overrides": {
"@xmldom/xmldom": "0.8.13",
},
"packages": {
"@asamuzakjp/css-color": ["@asamuzakjp/css-color@3.2.0", "", { "dependencies": { "@csstools/css-calc": "^2.1.3", "@csstools/css-color-parser": "^3.0.9", "@csstools/css-parser-algorithms": "^3.0.4", "@csstools/css-tokenizer": "^3.0.3", "lru-cache": "^10.4.3" } }, "sha512-K1A6z8tS3XsmCMM86xoWdn7Fkdn9m6RSVtocUrJYIwZnFVkng/PvkEoWtOWmP+Scc6saYWHWZYbndEEXxl24jw=="],
"@asamuzakjp/css-color": ["@asamuzakjp/css-color@5.1.11", "", { "dependencies": { "@asamuzakjp/generational-cache": "^1.0.1", "@csstools/css-calc": "^3.2.0", "@csstools/css-color-parser": "^4.1.0", "@csstools/css-parser-algorithms": "^4.0.0", "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg=="],
"@asamuzakjp/dom-selector": ["@asamuzakjp/dom-selector@7.1.1", "", { "dependencies": { "@asamuzakjp/generational-cache": "^1.0.1", "@asamuzakjp/nwsapi": "^2.3.9", "bidi-js": "^1.0.3", "css-tree": "^3.2.1", "is-potential-custom-element-name": "^1.0.1" } }, "sha512-67RZDnYRc8H/8MLDgQCDE//zoqVFwajkepHZgmXrbwybzXOEwOWGPYGmALYl9J2DOLfFPPs6kKCqmbzV895hTQ=="],
"@asamuzakjp/generational-cache": ["@asamuzakjp/generational-cache@1.0.1", "", {}, "sha512-wajfB8KqzMCN2KGNFdLkReeHncd0AslUSrvHVvvYWuU8ghncRJoA50kT3zP9MVL0+9g4/67H+cdvBskj9THPzg=="],
"@asamuzakjp/nwsapi": ["@asamuzakjp/nwsapi@2.3.9", "", {}, "sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q=="],
"@babel/runtime": ["@babel/runtime@7.29.2", "", {}, "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g=="],
"@bramus/specificity": ["@bramus/specificity@2.4.2", "", { "dependencies": { "css-tree": "^3.0.0" }, "bin": { "specificity": "bin/cli.js" } }, "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw=="],
"@changesets/apply-release-plan": ["@changesets/apply-release-plan@7.1.0", "", { "dependencies": { "@changesets/config": "^3.1.3", "@changesets/get-version-range-type": "^0.4.0", "@changesets/git": "^3.0.4", "@changesets/should-skip-package": "^0.1.2", "@changesets/types": "^6.1.0", "@manypkg/get-packages": "^1.1.3", "detect-indent": "^6.0.0", "fs-extra": "^7.0.1", "lodash.startcase": "^4.4.0", "outdent": "^0.5.0", "prettier": "^2.7.1", "resolve-from": "^5.0.0", "semver": "^7.5.3" } }, "sha512-yq8ML3YS7koKQ/9bk1PqO0HMzApIFNwjlwCnwFEXMzNe8NpzeeYYKCmnhWJGkN8g7E51MnWaSbqRcTcdIxUgnQ=="],
"@changesets/assemble-release-plan": ["@changesets/assemble-release-plan@6.0.9", "", { "dependencies": { "@changesets/errors": "^0.2.0", "@changesets/get-dependents-graph": "^2.1.3", "@changesets/should-skip-package": "^0.1.2", "@changesets/types": "^6.1.0", "@manypkg/get-packages": "^1.1.3", "semver": "^7.5.3" } }, "sha512-tPgeeqCHIwNo8sypKlS3gOPmsS3wP0zHt67JDuL20P4QcXiw/O4Hl7oXiuLnP9yg+rXLQ2sScdV1Kkzde61iSQ=="],
@@ -63,15 +75,19 @@
"@changesets/write": ["@changesets/write@0.4.0", "", { "dependencies": { "@changesets/types": "^6.1.0", "fs-extra": "^7.0.1", "human-id": "^4.1.1", "prettier": "^2.7.1" } }, "sha512-CdTLvIOPiCNuH71pyDu3rA+Q0n65cmAbXnwWH84rKGiFumFzkmHNT8KHTMEchcxN+Kl8I54xGUhJ7l3E7X396Q=="],
"@csstools/color-helpers": ["@csstools/color-helpers@5.1.0", "", {}, "sha512-S11EXWJyy0Mz5SYvRmY8nJYTFFd1LCNV+7cXyAgQtOOuzb4EsgfqDufL+9esx72/eLhsRdGZwaldu/h+E4t4BA=="],
"@csstools/color-helpers": ["@csstools/color-helpers@6.0.2", "", {}, "sha512-LMGQLS9EuADloEFkcTBR3BwV/CGHV7zyDxVRtVDTwdI2Ca4it0CCVTT9wCkxSgokjE5Ho41hEPgb8OEUwoXr6Q=="],
"@csstools/css-calc": ["@csstools/css-calc@2.1.4", "", { "peerDependencies": { "@csstools/css-parser-algorithms": "^3.0.5", "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-3N8oaj+0juUw/1H3YwmDDJXCgTB1gKU6Hc/bB502u9zR0q2vd786XJH9QfrKIEgFlZmhZiq6epXl4rHqhzsIgQ=="],
"@csstools/css-calc": ["@csstools/css-calc@3.2.0", "", { "peerDependencies": { "@csstools/css-parser-algorithms": "^4.0.0", "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-bR9e6o2BDB12jzN/gIbjHa5wLJ4UjD1CB9pM7ehlc0ddk6EBz+yYS1EV2MF55/HUxrHcB/hehAyt5vhsA3hx7w=="],
"@csstools/css-color-parser": ["@csstools/css-color-parser@3.1.0", "", { "dependencies": { "@csstools/color-helpers": "^5.1.0", "@csstools/css-calc": "^2.1.4" }, "peerDependencies": { "@csstools/css-parser-algorithms": "^3.0.5", "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-nbtKwh3a6xNVIp/VRuXV64yTKnb1IjTAEEh3irzS+HkKjAOYLTGNb9pmVNntZ8iVBHcWDA2Dof0QtPgFI1BaTA=="],
"@csstools/css-color-parser": ["@csstools/css-color-parser@4.1.0", "", { "dependencies": { "@csstools/color-helpers": "^6.0.2", "@csstools/css-calc": "^3.2.0" }, "peerDependencies": { "@csstools/css-parser-algorithms": "^4.0.0", "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-U0KhLYmy2GVj6q4T3WaAe6NPuFYCPQoE3b0dRGxejWDgcPp8TP7S5rVdM5ZrFaqu4N67X8YaPBw14dQSYx3IyQ=="],
"@csstools/css-parser-algorithms": ["@csstools/css-parser-algorithms@3.0.5", "", { "peerDependencies": { "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-DaDeUkXZKjdGhgYaHNJTV9pV7Y9B3b644jCLs9Upc3VeNGg6LWARAT6O+Q+/COo+2gg/bM5rhpMAtf70WqfBdQ=="],
"@csstools/css-parser-algorithms": ["@csstools/css-parser-algorithms@4.0.0", "", { "peerDependencies": { "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-+B87qS7fIG3L5h3qwJ/IFbjoVoOe/bpOdh9hAjXbvx0o8ImEmUsGXN0inFOnk2ChCFgqkkGFQ+TpM5rbhkKe4w=="],
"@csstools/css-tokenizer": ["@csstools/css-tokenizer@3.0.4", "", {}, "sha512-Vd/9EVDiu6PPJt9yAh6roZP6El1xHrdvIVGjyBsHR0RYwNHgL7FJPyIIW4fANJNG6FtyZfvlRPpFI4ZM/lubvw=="],
"@csstools/css-syntax-patches-for-csstree": ["@csstools/css-syntax-patches-for-csstree@1.1.3", "", { "peerDependencies": { "css-tree": "^3.2.1" }, "optionalPeers": ["css-tree"] }, "sha512-SH60bMfrRCJF3morcdk57WklujF4Jr/EsQUzqkarfHXEFcAR1gg7fS/chAE922Sehgzc1/+Tz5H3Ypa1HiEKrg=="],
"@csstools/css-tokenizer": ["@csstools/css-tokenizer@4.0.0", "", {}, "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA=="],
"@exodus/bytes": ["@exodus/bytes@1.15.0", "", { "peerDependencies": { "@noble/hashes": "^1.8.0 || ^2.0.0" }, "optionalPeers": ["@noble/hashes"] }, "sha512-UY0nlA+feH81UGSHv92sLEPLCeZFjXOuHhrIo0HQydScuQc8s0A7kL/UdgwgDq8g8ilksmuoF35YVTNphV2aBQ=="],
"@inquirer/external-editor": ["@inquirer/external-editor@1.0.3", "", { "dependencies": { "chardet": "^2.1.1", "iconv-lite": "^0.7.0" }, "peerDependencies": { "@types/node": ">=18" }, "optionalPeers": ["@types/node"] }, "sha512-RWbSrDiYmO4LbejWY7ttpxczuwQyZLBUyygsA9Nsv95hpzUWwnNTVQmAq3xuh7vNwCp07UTmE5i11XAEExx4RA=="],
@@ -107,9 +123,7 @@
"@types/ws": ["@types/ws@8.18.1", "", { "dependencies": { "@types/node": "*" } }, "sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg=="],
"@xmldom/xmldom": ["@xmldom/xmldom@0.8.11", "", {}, "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw=="],
"agent-base": ["agent-base@7.1.4", "", {}, "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ=="],
"@xmldom/xmldom": ["@xmldom/xmldom@0.8.13", "", {}, "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw=="],
"ansi-colors": ["ansi-colors@4.1.3", "", {}, "sha512-/6w/C21Pm1A7aZitlI5Ni/2J6FFQN8i1Cvz3kHABAAbw93v/NlvKdVOqz7CCWz/3iv/JplRSEEZ83XION15ovw=="],
@@ -123,6 +137,8 @@
"better-path-resolve": ["better-path-resolve@1.0.0", "", { "dependencies": { "is-windows": "^1.0.0" } }, "sha512-pbnl5XzGBdrFU/wT4jqmJVPn2B6UHPBOhzMQkY/SPUPB6QtUXtmBHBIwCbXJol93mOpGMnQyP/+BB19q04xj7g=="],
"bidi-js": ["bidi-js@1.0.3", "", { "dependencies": { "require-from-string": "^2.0.2" } }, "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw=="],
"boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="],
"braces": ["braces@3.0.3", "", { "dependencies": { "fill-range": "^7.1.1" } }, "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA=="],
@@ -143,13 +159,13 @@
"css-select": ["css-select@5.2.2", "", { "dependencies": { "boolbase": "^1.0.0", "css-what": "^6.1.0", "domhandler": "^5.0.2", "domutils": "^3.0.1", "nth-check": "^2.0.1" } }, "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw=="],
"css-tree": ["css-tree@3.2.1", "", { "dependencies": { "mdn-data": "2.27.1", "source-map-js": "^1.2.1" } }, "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA=="],
"css-what": ["css-what@6.2.2", "", {}, "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA=="],
"cssom": ["cssom@0.5.0", "", {}, "sha512-iKuQcq+NdHqlAcwUY0o/HL69XQrUaQdMjmStJ8JFmUaiiQErlhrmuigkg/CU4E2J0IyUKUrMAgl36TvN67MqTw=="],
"cssstyle": ["cssstyle@4.6.0", "", { "dependencies": { "@asamuzakjp/css-color": "^3.2.0", "rrweb-cssom": "^0.8.0" } }, "sha512-2z+rWdzbbSZv6/rhtvzvqeZQHrBaqgogqt85sqFNbabZOuFbCVFb8kPeEtZjiKkbrm395irpNKiYeFeLiQnFPg=="],
"data-urls": ["data-urls@5.0.0", "", { "dependencies": { "whatwg-mimetype": "^4.0.0", "whatwg-url": "^14.0.0" } }, "sha512-ZYP5VBHshaDAiVZxjbRVcFJpc+4xGgT0bK3vzy1HLN8jTO975HEbuYzZJcHoQEY5K1a0z8YayJkyVETa08eNTg=="],
"data-urls": ["data-urls@7.0.0", "", { "dependencies": { "whatwg-mimetype": "^5.0.0", "whatwg-url": "^16.0.0" } }, "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA=="],
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
@@ -157,7 +173,7 @@
"decode-named-character-reference": ["decode-named-character-reference@1.3.0", "", { "dependencies": { "character-entities": "^2.0.0" } }, "sha512-GtpQYB283KrPp6nRw50q3U9/VfOutZOe103qlN7BPP6Ad27xYnOIWv4lPzo8HCAL+mMZofJ9KEy30fq6MfaK6Q=="],
"defuddle": ["defuddle@0.14.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "linkedom": "^0.18.12", "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-btavZGd1WgiVqrVM62WGRXMUi/aU7ckTZiq0xXWLZMHvzIqNZjwIFQEDRx8MarD7fIgsB90NXZ9xHJkKtapt2Q=="],
"defuddle": ["defuddle@0.17.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "linkedom": "^0.18.12", "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-rjRb9kxgrJLFhFJjXTSAitbmmSyvjEqGCi5JiyM6ImOf4leC/O+f7TPVPAymk/gnENFTrn9j7T/LTfW9IVwiLw=="],
"dequal": ["dequal@2.0.3", "", {}, "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA=="],
@@ -203,16 +219,12 @@
"graceful-fs": ["graceful-fs@4.2.11", "", {}, "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="],
"html-encoding-sniffer": ["html-encoding-sniffer@4.0.0", "", { "dependencies": { "whatwg-encoding": "^3.1.1" } }, "sha512-Y22oTqIU4uuPgEemfz7NDJz6OeKf12Lsu+QC+s3BVpda64lTiMYCyGwg5ki4vFxkMwQdeZDl2adZoqUgdFuTgQ=="],
"html-encoding-sniffer": ["html-encoding-sniffer@6.0.0", "", { "dependencies": { "@exodus/bytes": "^1.6.0" } }, "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg=="],
"html-escaper": ["html-escaper@3.0.3", "", {}, "sha512-RuMffC89BOWQoY0WKGpIhn5gX3iI54O6nRA0yC124NYVtzjmFWBIiFd8M0x+ZdX0P9R4lADg1mgP8C7PxGOWuQ=="],
"htmlparser2": ["htmlparser2@10.1.0", "", { "dependencies": { "domelementtype": "^2.3.0", "domhandler": "^5.0.3", "domutils": "^3.2.2", "entities": "^7.0.1" } }, "sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ=="],
"http-proxy-agent": ["http-proxy-agent@7.0.2", "", { "dependencies": { "agent-base": "^7.1.0", "debug": "^4.3.4" } }, "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig=="],
"https-proxy-agent": ["https-proxy-agent@7.0.6", "", { "dependencies": { "agent-base": "^7.1.2", "debug": "4" } }, "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw=="],
"human-id": ["human-id@4.1.3", "", { "bin": { "human-id": "dist/cli.js" } }, "sha512-tsYlhAYpjCKa//8rXZ9DqKEawhPoSytweBC2eNvcaDK+57RZLHGqNs3PZTQO6yekLFSuvA6AlnAfrw1uBvtb+Q=="],
"iconv-lite": ["iconv-lite@0.7.2", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw=="],
@@ -241,7 +253,7 @@
"js-yaml": ["js-yaml@4.1.1", "", { "dependencies": { "argparse": "^2.0.1" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA=="],
"jsdom": ["jsdom@26.1.0", "", { "dependencies": { "cssstyle": "^4.2.1", "data-urls": "^5.0.0", "decimal.js": "^10.5.0", "html-encoding-sniffer": "^4.0.0", "http-proxy-agent": "^7.0.2", "https-proxy-agent": "^7.0.6", "is-potential-custom-element-name": "^1.0.1", "nwsapi": "^2.2.16", "parse5": "^7.2.1", "rrweb-cssom": "^0.8.0", "saxes": "^6.0.0", "symbol-tree": "^3.2.4", "tough-cookie": "^5.1.1", "w3c-xmlserializer": "^5.0.0", "webidl-conversions": "^7.0.0", "whatwg-encoding": "^3.1.1", "whatwg-mimetype": "^4.0.0", "whatwg-url": "^14.1.1", "ws": "^8.18.0", "xml-name-validator": "^5.0.0" }, "peerDependencies": { "canvas": "^3.0.0" }, "optionalPeers": ["canvas"] }, "sha512-Cvc9WUhxSMEo4McES3P7oK3QaXldCfNWp7pl2NNeiIFlCoLr3kfq9kb1fxftiwk1FLV7CvpvDfonxtzUDeSOPg=="],
"jsdom": ["jsdom@29.0.2", "", { "dependencies": { "@asamuzakjp/css-color": "^5.1.5", "@asamuzakjp/dom-selector": "^7.0.6", "@bramus/specificity": "^2.4.2", "@csstools/css-syntax-patches-for-csstree": "^1.1.1", "@exodus/bytes": "^1.15.0", "css-tree": "^3.2.1", "data-urls": "^7.0.0", "decimal.js": "^10.6.0", "html-encoding-sniffer": "^6.0.0", "is-potential-custom-element-name": "^1.0.1", "lru-cache": "^11.2.7", "parse5": "^8.0.0", "saxes": "^6.0.0", "symbol-tree": "^3.2.4", "tough-cookie": "^6.0.1", "undici": "^7.24.5", "w3c-xmlserializer": "^5.0.0", "webidl-conversions": "^8.0.1", "whatwg-mimetype": "^5.0.0", "whatwg-url": "^16.0.1", "xml-name-validator": "^5.0.0" }, "peerDependencies": { "canvas": "^3.0.0" }, "optionalPeers": ["canvas"] }, "sha512-9VnGEBosc/ZpwyOsJBCQ/3I5p7Q5ngOY14a9bf5btenAORmZfDse1ZEheMiWcJ3h81+Fv7HmJFdS0szo/waF2w=="],
"jsonfile": ["jsonfile@4.0.0", "", { "optionalDependencies": { "graceful-fs": "^4.1.6" } }, "sha512-m6F1R3z8jjlf2imQHS2Qez5sjKWQzbuuhuJ/FKYFRZvPE3PuHcSMVZzfsLhGVOkfd20obL5SWEBew5ShlquNxg=="],
@@ -255,7 +267,7 @@
"longest-streak": ["longest-streak@3.1.0", "", {}, "sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g=="],
"lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="],
"lru-cache": ["lru-cache@11.3.5", "", {}, "sha512-NxVFwLAnrd9i7KUBxC4DrUhmgjzOs+1Qm50D3oF1/oL+r1NpZ4gA7xvG0/zJ8evR7zIKn4vLf7qTNduWFtCrRw=="],
"markdown-table": ["markdown-table@3.0.4", "", {}, "sha512-wiYz4+JrLyb/DqW2hkFJxP7Vd7JuTDm77fvbM8VfEQdmSMqcImWeeRbHwZjBjIFki/VaMK2BhFi7oUUZeM5bqw=="],
@@ -285,6 +297,8 @@
"mdast-util-to-string": ["mdast-util-to-string@4.0.0", "", { "dependencies": { "@types/mdast": "^4.0.0" } }, "sha512-0H44vDimn51F0YwvxSJSm0eCDOJTRlmN0R1yBh4HLj9wiV1Dn0QoXGbvFAWj2hSItVTlCmBF1hqKlIyUBVFLPg=="],
"mdn-data": ["mdn-data@2.27.1", "", {}, "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ=="],
"merge2": ["merge2@1.4.1", "", {}, "sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg=="],
"micromark": ["micromark@4.0.2", "", { "dependencies": { "@types/debug": "^4.0.0", "debug": "^4.0.0", "decode-named-character-reference": "^1.0.0", "devlop": "^1.0.0", "micromark-core-commonmark": "^2.0.0", "micromark-factory-space": "^2.0.0", "micromark-util-character": "^2.0.0", "micromark-util-chunked": "^2.0.0", "micromark-util-combine-extensions": "^2.0.0", "micromark-util-decode-numeric-character-reference": "^2.0.0", "micromark-util-encode": "^2.0.0", "micromark-util-normalize-identifier": "^2.0.0", "micromark-util-resolve-all": "^2.0.0", "micromark-util-sanitize-uri": "^2.0.0", "micromark-util-subtokenize": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-zpe98Q6kvavpCr1NPVSCMebCKfD7CA2NqZ+rykeNhONIJBpc1tFKt9hucLGwha3jNTNI8lHpctWJWoimVF4PfA=="],
@@ -351,8 +365,6 @@
"nth-check": ["nth-check@2.1.1", "", { "dependencies": { "boolbase": "^1.0.0" } }, "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w=="],
"nwsapi": ["nwsapi@2.2.23", "", {}, "sha512-7wfH4sLbt4M0gCDzGE6vzQBo0bfTKjU7Sfpqy/7gs1qBfYz2vEJH6vXcBKpO3+6Yu1telwd0t9HpyOoLEQQbIQ=="],
"outdent": ["outdent@0.5.0", "", {}, "sha512-/jHxFIzoMXdqPzTaCpFzAAWhpkSjZPF4Vsn6jAfNpmbH/ymsmd7Qc6VE9BGn0L6YMj6uwpQLxCECpus4ukKS9Q=="],
"p-filter": ["p-filter@2.1.0", "", { "dependencies": { "p-map": "^2.0.0" } }, "sha512-ZBxxZ5sL2HghephhpGAQdoskxplTwr7ICaehZwLIlfL6acuVgZPm8yBNuRAFBGEqtD/hmUeq9eqLg2ys9Xr/yw=="],
@@ -397,12 +409,12 @@
"remark-stringify": ["remark-stringify@11.0.0", "", { "dependencies": { "@types/mdast": "^4.0.0", "mdast-util-to-markdown": "^2.0.0", "unified": "^11.0.0" } }, "sha512-1OSmLd3awB/t8qdoEOMazZkNsfVTeY4fTsgzcQFdXNq8ToTN4ZGwrMnlda4K6smTFKD+GRV6O48i6Z4iKgPPpw=="],
"require-from-string": ["require-from-string@2.0.2", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="],
"resolve-from": ["resolve-from@5.0.0", "", {}, "sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw=="],
"reusify": ["reusify@1.1.0", "", {}, "sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw=="],
"rrweb-cssom": ["rrweb-cssom@0.8.0", "", {}, "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw=="],
"run-parallel": ["run-parallel@1.2.0", "", { "dependencies": { "queue-microtask": "^1.2.2" } }, "sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA=="],
"safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="],
@@ -419,6 +431,8 @@
"slash": ["slash@3.0.0", "", {}, "sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q=="],
"source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
"spawndamnit": ["spawndamnit@3.0.1", "", { "dependencies": { "cross-spawn": "^7.0.5", "signal-exit": "^4.0.1" } }, "sha512-MmnduQUuHCoFckZoWnXsTg7JaiLBJrKFj9UI2MbRPGaJeVpsLcVBu6P/IGZovziM/YBsellCmsprgNA+w0CzVg=="],
"sprintf-js": ["sprintf-js@1.0.3", "", {}, "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g=="],
@@ -433,15 +447,15 @@
"term-size": ["term-size@2.2.1", "", {}, "sha512-wK0Ri4fOGjv/XPy8SBHZChl8CM7uMc5VML7SqiQ0zG7+J5Vr+RMQDoHa2CNT6KHUnTGIXH34UDMkPzAUyapBZg=="],
"tldts": ["tldts@6.1.86", "", { "dependencies": { "tldts-core": "^6.1.86" }, "bin": { "tldts": "bin/cli.js" } }, "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ=="],
"tldts": ["tldts@7.0.28", "", { "dependencies": { "tldts-core": "^7.0.28" }, "bin": { "tldts": "bin/cli.js" } }, "sha512-+Zg3vWhRUv8B1maGSTFdev9mjoo8Etn2Ayfs4cnjlD3CsGkxXX4QyW3j2WJ0wdjYcYmy7Lx2RDsZMhgCWafKIw=="],
"tldts-core": ["tldts-core@6.1.86", "", {}, "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA=="],
"tldts-core": ["tldts-core@7.0.28", "", {}, "sha512-7W5Efjhsc3chVdFhqtaU0KtK32J37Zcr9RKtID54nG+tIpcY79CQK/veYPODxtD/LJ4Lue66jvrQzIX2Z2/pUQ=="],
"to-regex-range": ["to-regex-range@5.0.1", "", { "dependencies": { "is-number": "^7.0.0" } }, "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ=="],
"tough-cookie": ["tough-cookie@5.1.2", "", { "dependencies": { "tldts": "^6.1.32" } }, "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A=="],
"tough-cookie": ["tough-cookie@6.0.1", "", { "dependencies": { "tldts": "^7.0.5" } }, "sha512-LktZQb3IeoUWB9lqR5EWTHgW/VTITCXg4D21M+lvybRVdylLrRMnqaIONLVb5mav8vM19m44HIcGq4qASeu2Qw=="],
"tr46": ["tr46@5.1.1", "", { "dependencies": { "punycode": "^2.3.1" } }, "sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw=="],
"tr46": ["tr46@6.0.0", "", { "dependencies": { "punycode": "^2.3.1" } }, "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw=="],
"trough": ["trough@2.2.0", "", {}, "sha512-tmMpK00BjZiUyVyvrBK7knerNgmgvcV/KLVyuma/SC+TQN167GrMRciANTz09+k3zW8L8t60jWO1GpfkZdjTaw=="],
@@ -453,6 +467,8 @@
"uhyphen": ["uhyphen@0.2.0", "", {}, "sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA=="],
"undici": ["undici@7.25.0", "", {}, "sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ=="],
"undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"unified": ["unified@11.0.5", "", { "dependencies": { "@types/unist": "^3.0.0", "bail": "^2.0.0", "devlop": "^1.0.0", "extend": "^3.0.0", "is-plain-obj": "^4.0.0", "trough": "^2.0.0", "vfile": "^6.0.0" } }, "sha512-xKvGhPWw3k84Qjh8bI3ZeJjqnyadK+GEFtazSfZv/rKeTkTjOJho6mFqh2SM96iIcZokxiOpg78GazTSg8+KHA=="],
@@ -473,13 +489,11 @@
"w3c-xmlserializer": ["w3c-xmlserializer@5.0.0", "", { "dependencies": { "xml-name-validator": "^5.0.0" } }, "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA=="],
"webidl-conversions": ["webidl-conversions@7.0.0", "", {}, "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g=="],
"webidl-conversions": ["webidl-conversions@8.0.1", "", {}, "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ=="],
"whatwg-encoding": ["whatwg-encoding@3.1.1", "", { "dependencies": { "iconv-lite": "0.6.3" } }, "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ=="],
"whatwg-mimetype": ["whatwg-mimetype@5.0.0", "", {}, "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw=="],
"whatwg-mimetype": ["whatwg-mimetype@4.0.0", "", {}, "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg=="],
"whatwg-url": ["whatwg-url@14.2.0", "", { "dependencies": { "tr46": "^5.1.0", "webidl-conversions": "^7.0.0" } }, "sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw=="],
"whatwg-url": ["whatwg-url@16.0.1", "", { "dependencies": { "@exodus/bytes": "^1.11.0", "tr46": "^6.0.0", "webidl-conversions": "^8.0.1" } }, "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw=="],
"which": ["which@2.0.2", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "./bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="],
@@ -503,11 +517,13 @@
"htmlparser2/entities": ["entities@7.0.1", "", {}, "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA=="],
"jsdom/parse5": ["parse5@8.0.1", "", { "dependencies": { "entities": "^8.0.0" } }, "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw=="],
"mdast-util-find-and-replace/escape-string-regexp": ["escape-string-regexp@5.0.0", "", {}, "sha512-/veY75JbMK4j1yjvuUxuVsiS/hr/4iHs9FTT6cgTexxdE0Ly/glccBAkloH/DofkjRbZU3bnoj38mOmhkZ0lHw=="],
"read-yaml-file/js-yaml": ["js-yaml@3.14.2", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg=="],
"whatwg-encoding/iconv-lite": ["iconv-lite@0.6.3", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw=="],
"jsdom/parse5/entities": ["entities@8.0.0", "", {}, "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA=="],
"read-yaml-file/js-yaml/argparse": ["argparse@1.0.10", "", { "dependencies": { "sprintf-js": "~1.0.2" } }, "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg=="],
}
+5 -2
View File
@@ -46,8 +46,8 @@
"dependencies": {
"@mozilla/readability": "^0.6.0",
"chrome-launcher": "^1.2.1",
"defuddle": "^0.14.0",
"jsdom": "^26.0.0",
"defuddle": "^0.17.0",
"jsdom": "^29.0.2",
"remark-gfm": "^4.0.1",
"remark-parse": "^11.0.0",
"turndown": "^7.2.0",
@@ -61,5 +61,8 @@
"@types/jsdom": "^21.1.7",
"@types/ws": "^8.18.1",
"typescript": "^5.9.2"
},
"overrides": {
"@xmldom/xmldom": "0.8.13"
}
}
@@ -339,4 +339,105 @@ describe("x article extraction", () => {
expect(content.markdown).toContain("https://example.com/report");
expect(content.markdown).not.toContain("https://t.co/example");
});
test("renders article video media as the highest bitrate mp4 link", () => {
const payload = {
data: {
tweetResult: {
result: {
rest_id: "2046628728210350366",
legacy: {
full_text: "Fallback text",
favorite_count: 12,
retweet_count: 3,
reply_count: 1,
created_at: "Tue Apr 21 16:34:47 +0000 2026",
},
core: {
user_results: {
result: {
legacy: {
name: "Google AI Studio",
screen_name: "GoogleAIStudio",
},
},
},
},
article: {
article_results: {
result: {
title: "Article with video",
media_entities: [
{
media_id: "2046627051822530560",
media_info: {
__typename: "ApiVideo",
variants: [
{
bit_rate: 2176000,
content_type: "video/mp4",
url: "https://video.twimg.com/amplify_video/2046627051822530560/vid/avc1/1280x720/medium.mp4",
},
{
content_type: "application/x-mpegURL",
url: "https://video.twimg.com/amplify_video/2046627051822530560/pl/playlist.m3u8",
},
{
bit_rate: 10368000,
content_type: "video/mp4",
url: "https://video.twimg.com/amplify_video/2046627051822530560/vid/avc1/1920x1080/high.mp4",
},
],
},
},
],
content_state: {
blocks: [
{
type: "atomic",
text: " ",
data: {},
entityRanges: [{ key: 0, length: 1, offset: 0 }],
inlineStyleRanges: [],
},
],
entityMap: [
{
key: "0",
value: {
type: "MEDIA",
mutability: "Immutable",
data: {
mediaItems: [{ mediaId: "2046627051822530560" }],
},
},
},
],
},
},
},
},
},
},
},
};
const document = extractArticleDocumentFromPayload(
payload,
"2046628728210350366",
"https://x.com/GoogleAIStudio/status/2046628728210350366",
);
expect(document).not.toBeNull();
const content = document?.content[0];
expect(content?.type).toBe("markdown");
if (!content || content.type !== "markdown") {
throw new Error("Expected markdown content");
}
expect(content.markdown).toBe(
"[video](https://video.twimg.com/amplify_video/2046627051822530560/vid/avc1/1920x1080/high.mp4)",
);
});
});
@@ -184,4 +184,77 @@ describe("x single tweet extraction", () => {
"Quoted Author (@quoted_author)\n\nQuoted tweet text\n\nphoto: https://pbs.twimg.com/media/quoted?format=jpg&name=4096x4096",
});
});
test("uses the highest bitrate mp4 variant for tweet video media", () => {
const payload = {
data: {
tweetResult: {
result: {
rest_id: "2046628728210350366",
legacy: {
full_text: "Video post https://t.co/video",
favorite_count: 12,
retweet_count: 3,
reply_count: 1,
created_at: "Tue Apr 21 16:34:47 +0000 2026",
extended_entities: {
media: [
{
type: "video",
media_url_https: "https://pbs.twimg.com/amplify_video_thumb/2046627051822530560/img/poster.jpg",
url: "https://t.co/video",
video_info: {
variants: [
{
content_type: "application/x-mpegURL",
url: "https://video.twimg.com/amplify_video/2046627051822530560/pl/playlist.m3u8",
},
{
bitrate: 256000,
content_type: "video/mp4",
url: "https://video.twimg.com/amplify_video/2046627051822530560/vid/avc1/480x270/low.mp4",
},
{
bitrate: 10368000,
content_type: "video/mp4",
url: "https://video.twimg.com/amplify_video/2046627051822530560/vid/avc1/1920x1080/high.mp4",
},
],
},
},
],
},
},
core: {
user_results: {
result: {
legacy: {
name: "Google AI Studio",
screen_name: "GoogleAIStudio",
},
},
},
},
},
},
},
};
const document = extractSingleTweetDocumentFromPayload(
payload,
"2046628728210350366",
"https://x.com/GoogleAIStudio/status/2046628728210350366",
);
expect(document).not.toBeNull();
const listBlock = document?.content.find((block) => block.type === "list");
expect(listBlock).toEqual({
type: "list",
ordered: false,
items: [
"video: https://video.twimg.com/amplify_video/2046627051822530560/vid/avc1/1920x1080/high.mp4",
],
});
});
});
+48 -20
View File
@@ -9,18 +9,43 @@ import {
getUser,
isRecord,
normalizeTitle,
resolveBestXVideoVariantUrl,
toHighResXImageUrl,
toXTweet,
} from "./shared";
import type { JsonObject } from "./types";
function resolveArticleMediaUrl(mediaInfo: JsonObject): string {
interface ArticleMedia {
kind: "image" | "video";
url: string;
}
function resolveArticleMedia(mediaInfo: JsonObject): ArticleMedia | null {
const videoUrl = resolveBestXVideoVariantUrl(mediaInfo);
if (videoUrl) {
return {
kind: "video",
url: videoUrl,
};
}
const rawUrl =
(typeof mediaInfo.original_img_url === "string" && mediaInfo.original_img_url) ||
(typeof mediaInfo.url === "string" && mediaInfo.url) ||
"";
return rawUrl ? toHighResXImageUrl(rawUrl) : "";
if (!rawUrl) {
return null;
}
return {
kind: "image",
url: toHighResXImageUrl(rawUrl),
};
}
function resolveArticleMediaUrl(mediaInfo: JsonObject): string {
return resolveArticleMedia(mediaInfo)?.url ?? "";
}
function normalizeEntityMap(entityMap: unknown): Map<string, JsonObject> {
@@ -139,8 +164,8 @@ function getTweetId(entityMap: Map<string, JsonObject>, entityKey: unknown): str
return data.tweetId;
}
function buildMediaUrlMap(articleResult: JsonObject): Map<string, string> {
const mediaMap = new Map<string, string>();
function buildMediaMap(articleResult: JsonObject): Map<string, ArticleMedia> {
const mediaMap = new Map<string, ArticleMedia>();
const mediaEntities = Array.isArray(articleResult.media_entities) ? articleResult.media_entities : [];
for (const entity of mediaEntities) {
@@ -148,25 +173,28 @@ function buildMediaUrlMap(articleResult: JsonObject): Map<string, string> {
continue;
}
const mediaInfo = entity.media_info;
const url = resolveArticleMediaUrl(mediaInfo);
if (url) {
mediaMap.set(entity.media_id, url);
const media = resolveArticleMedia(entity.media_info);
if (media) {
mediaMap.set(entity.media_id, media);
}
}
const coverMedia = isRecord(articleResult.cover_media) ? articleResult.cover_media : null;
if (coverMedia && typeof coverMedia.media_id === "string" && isRecord(coverMedia.media_info)) {
const url = resolveArticleMediaUrl(coverMedia.media_info);
if (url) {
mediaMap.set(coverMedia.media_id, url);
const media = resolveArticleMedia(coverMedia.media_info);
if (media) {
mediaMap.set(coverMedia.media_id, media);
}
}
return mediaMap;
}
function getMediaMarkdown(entityMap: Map<string, JsonObject>, entityKey: unknown, mediaMap: Map<string, string>): string[] {
function getMediaMarkdown(
entityMap: Map<string, JsonObject>,
entityKey: unknown,
mediaMap: Map<string, ArticleMedia>,
): string[] {
const key =
typeof entityKey === "string" || typeof entityKey === "number"
? String(entityKey)
@@ -182,19 +210,19 @@ function getMediaMarkdown(entityMap: Map<string, JsonObject>, entityKey: unknown
const data = isRecord(entity.data) ? entity.data : {};
const mediaItems = Array.isArray(data.mediaItems) ? data.mediaItems : [];
const urls: string[] = [];
const media: ArticleMedia[] = [];
for (const item of mediaItems) {
if (!isRecord(item) || typeof item.mediaId !== "string") {
continue;
}
const url = mediaMap.get(item.mediaId);
if (url && !urls.includes(url)) {
urls.push(url);
const mediaItem = mediaMap.get(item.mediaId);
if (mediaItem && !media.some((value) => value.url === mediaItem.url)) {
media.push(mediaItem);
}
}
return urls.map((url) => `![](${url})`);
return media.map((item) => item.kind === "image" ? `![](${item.url})` : `[video](${item.url})`);
}
function resolveTweetMarkdown(payloads: unknown[], tweetId: string, pageUrl: string): string | null {
@@ -250,7 +278,7 @@ function replaceLinkEntities(text: string, block: JsonObject, entityMap: Map<str
function renderAtomicBlock(
block: JsonObject,
entityMap: Map<string, JsonObject>,
mediaMap: Map<string, string>,
mediaMap: Map<string, ArticleMedia>,
payloads: unknown[],
pageUrl: string,
): string | null {
@@ -293,7 +321,7 @@ function renderAtomicBlock(
function renderArticleBlocks(
blocks: unknown[],
entityMap: Map<string, JsonObject>,
mediaMap: Map<string, string>,
mediaMap: Map<string, ArticleMedia>,
payloads: unknown[],
pageUrl: string,
): string {
@@ -396,7 +424,7 @@ export function extractArticleDocumentFromPayload(
const contentState = isRecord(articleResult.content_state) ? articleResult.content_state : {};
const blocks = Array.isArray(contentState.blocks) ? contentState.blocks : [];
const entityMap = normalizeEntityMap(contentState.entityMap);
const mediaMap = buildMediaUrlMap(articleResult);
const mediaMap = buildMediaMap(articleResult);
const richMarkdown = renderArticleBlocks(blocks, entityMap, mediaMap, payloads, pageUrl);
const plainText = typeof articleResult.plain_text === "string" ? articleResult.plain_text.trim() : "";
const markdown = richMarkdown || plainText || getTweetText(tweet);
+39 -2
View File
@@ -251,6 +251,39 @@ export function toHighResXImageUrl(rawUrl: string): string {
}
}
function getVideoVariantBitrate(variant: JsonObject): number {
const value = variant.bitrate ?? variant.bit_rate;
return typeof value === "number" && Number.isFinite(value) ? value : 0;
}
function getVideoVariantContentType(variant: JsonObject): string {
const value = variant.content_type ?? variant.contentType;
return typeof value === "string" ? value.toLowerCase() : "";
}
export function resolveBestXVideoVariantUrl(mediaInfo: unknown): string | undefined {
if (!isRecord(mediaInfo)) {
return undefined;
}
const variantsSource =
Array.isArray(mediaInfo.variants)
? mediaInfo.variants
: isRecord(mediaInfo.video_info) && Array.isArray(mediaInfo.video_info.variants)
? mediaInfo.video_info.variants
: [];
const variants = variantsSource
.filter(
(variant): variant is JsonObject =>
isRecord(variant) && typeof variant.url === "string" && variant.url.length > 0,
)
.filter((variant) => getVideoVariantContentType(variant) === "video/mp4")
.sort((left, right) => getVideoVariantBitrate(right) - getVideoVariantBitrate(left));
return typeof variants[0]?.url === "string" ? variants[0].url : undefined;
}
export function getTweetMedia(tweet: JsonObject): XMedia[] {
const legacy = getLegacy(tweet);
const extendedEntities = isRecord(legacy.extended_entities) ? legacy.extended_entities : emptyObject();
@@ -268,10 +301,14 @@ export function getTweetMedia(tweet: JsonObject): XMedia[] {
alt: typeof value.ext_alt_text === "string" ? value.ext_alt_text : undefined,
};
}
if ((value.type === "video" || value.type === "animated_gif") && typeof value.media_url_https === "string") {
if (value.type === "video" || value.type === "animated_gif") {
const videoUrl = resolveBestXVideoVariantUrl(value);
if (!videoUrl) {
return null;
}
return {
type: value.type,
url: value.media_url_https,
url: videoUrl,
};
}
return null;
+43 -2
View File
@@ -10,6 +10,20 @@ const PACKAGE_DEPENDENCY_SECTIONS = [
const SKIPPED_DIRS = new Set([".git", ".clawhub", ".clawdhub", "node_modules", "out", "dist", "build"]);
const SKIPPED_FILES = new Set([".DS_Store", "bun.lockb"]);
const MIME_MAP = {
".md": "text/markdown",
".ts": "text/plain",
".js": "text/javascript",
".mjs": "text/javascript",
".json": "application/json",
".yml": "text/yaml",
".yaml": "text/yaml",
".txt": "text/plain",
".html": "text/html",
".css": "text/css",
".xml": "text/xml",
".svg": "image/svg+xml",
};
export async function listReleaseFiles(root) {
const resolvedRoot = path.resolve(root);
@@ -40,9 +54,10 @@ export async function listReleaseFiles(root) {
}
export async function validateSelfContainedRelease(root) {
const resolvedRoot = path.resolve(root);
const files = await listReleaseFiles(root);
for (const file of files.filter((entry) => path.posix.basename(entry.relPath) === "package.json")) {
const packageDir = path.resolve(root, fromPosixRel(path.posix.dirname(file.relPath)));
const packageDir = path.resolve(resolvedRoot, fromPosixRel(path.posix.dirname(file.relPath)));
const packageJson = JSON.parse(file.bytes.toString("utf8"));
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
@@ -51,7 +66,7 @@ export async function validateSelfContainedRelease(root) {
for (const [name, spec] of Object.entries(dependencies)) {
if (typeof spec !== "string" || !spec.startsWith("file:")) continue;
const targetDir = path.resolve(packageDir, spec.slice(5));
if (!isWithinRoot(root, targetDir)) {
if (!isWithinRoot(resolvedRoot, targetDir)) {
throw new Error(
`Release target is not self-contained: ${file.relPath} depends on ${name} via ${spec}`,
);
@@ -61,9 +76,35 @@ export async function validateSelfContainedRelease(root) {
});
}
}
for (const target of getPackageBinTargets(packageJson)) {
const targetPath = path.resolve(packageDir, target);
if (!isWithinRoot(resolvedRoot, targetPath)) {
throw new Error(`Release target is not self-contained: ${file.relPath} bin points to ${target}`);
}
await fs.access(targetPath).catch(() => {
throw new Error(`Missing package bin target for release: ${file.relPath} -> ${target}`);
});
}
}
}
export function mimeType(relPath) {
const ext = path.extname(relPath).toLowerCase();
return MIME_MAP[ext] || "text/plain";
}
function getPackageBinTargets(packageJson) {
const bin = packageJson.bin;
if (typeof bin === "string" && bin.trim()) {
return [bin.trim()];
}
if (!bin || typeof bin !== "object" || Array.isArray(bin)) {
return [];
}
return Object.values(bin).filter((value) => typeof value === "string" && value.trim());
}
function fromPosixRel(relPath) {
return relPath === "." ? "." : relPath.split("/").join(path.sep);
}
+34
View File
@@ -65,6 +65,40 @@ test("validateSelfContainedRelease accepts file dependencies that stay within th
await assert.doesNotReject(() => validateSelfContainedRelease(root));
});
test("validateSelfContainedRelease accepts package bin targets inside the release root", async (t) => {
const root = await makeTempDir("baoyu-release-bin-ok-");
t.after(() => fs.rm(root, { recursive: true, force: true }));
await writeJson(path.join(root, "scripts", "package.json"), {
name: "test-skill-scripts",
version: "1.0.0",
bin: {
"test-skill": "./test-skill",
},
});
await writeFile(path.join(root, "scripts", "test-skill"), "#!/usr/bin/env sh\n");
await assert.doesNotReject(() => validateSelfContainedRelease(root));
});
test("validateSelfContainedRelease rejects missing package bin targets", async (t) => {
const root = await makeTempDir("baoyu-release-bin-missing-");
t.after(() => fs.rm(root, { recursive: true, force: true }));
await writeJson(path.join(root, "scripts", "package.json"), {
name: "test-skill-scripts",
version: "1.0.0",
bin: {
"test-skill": "./missing",
},
});
await assert.rejects(
() => validateSelfContainedRelease(root),
/Missing package bin target for release/,
);
});
test("validateSelfContainedRelease rejects missing local file dependencies", async (t) => {
const root = await makeTempDir("baoyu-release-missing-");
t.after(() => fs.rm(root, { recursive: true, force: true }));
+1 -21
View File
@@ -5,7 +5,7 @@ import { existsSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { listReleaseFiles, validateSelfContainedRelease } from "./lib/release-files.mjs";
import { listReleaseFiles, mimeType, validateSelfContainedRelease } from "./lib/release-files.mjs";
const DEFAULT_REGISTRY = "https://clawhub.ai";
@@ -281,26 +281,6 @@ function titleCase(value) {
.replace(/\b\w/g, (char) => char.toUpperCase());
}
const MIME_MAP = {
".md": "text/markdown",
".ts": "text/plain",
".js": "text/javascript",
".mjs": "text/javascript",
".json": "application/json",
".yml": "text/yaml",
".yaml": "text/yaml",
".txt": "text/plain",
".html": "text/html",
".css": "text/css",
".xml": "text/xml",
".svg": "image/svg+xml",
};
function mimeType(relPath) {
const ext = path.extname(relPath).toLowerCase();
return MIME_MAP[ext] || "text/plain";
}
function parseBoolean(value) {
return ["1", "true", "yes", "on"].includes(String(value).trim().toLowerCase());
}
+8 -72
View File
@@ -6,47 +6,9 @@ import { existsSync } from "node:fs";
import path from "node:path";
import os from "node:os";
import { listReleaseFiles, mimeType, validateSelfContainedRelease } from "./lib/release-files.mjs";
const DEFAULT_REGISTRY = "https://clawhub.ai";
const TEXT_EXTENSIONS = new Set([
"md",
"mdx",
"txt",
"json",
"json5",
"yaml",
"yml",
"toml",
"js",
"cjs",
"mjs",
"ts",
"tsx",
"jsx",
"py",
"sh",
"rb",
"go",
"rs",
"swift",
"kt",
"java",
"cs",
"cpp",
"c",
"h",
"hpp",
"sql",
"csv",
"ini",
"cfg",
"env",
"xml",
"html",
"css",
"scss",
"sass",
"svg",
]);
async function main() {
const options = parseArgs(process.argv.slice(2));
@@ -75,7 +37,7 @@ async function main() {
console.log(`Roots with skills: ${roots.join(", ")}`);
const locals = await mapWithConcurrency(skills, options.concurrency, async (skill) => {
const files = await listTextFiles(skill.folder);
const files = await collectReleaseFiles(skill.folder);
const fingerprint = buildFingerprint(files);
return {
...skill,
@@ -162,7 +124,7 @@ async function main() {
console.log(`Publishing ${candidate.slug}@${version}`);
try {
const files = await listTextFiles(candidate.folder);
const files = await collectReleaseFiles(candidate.folder);
await publishSkill({
registry,
token: config.token,
@@ -363,35 +325,9 @@ async function hasSkillMarker(folder) {
);
}
async function listTextFiles(root) {
const files = [];
async function walk(folder) {
const entries = await fs.readdir(folder, { withFileTypes: true });
for (const entry of entries) {
if (entry.name.startsWith(".")) continue;
if (entry.name === "node_modules") continue;
if (entry.name === ".clawhub" || entry.name === ".clawdhub") continue;
const fullPath = path.join(folder, entry.name);
if (entry.isDirectory()) {
await walk(fullPath);
continue;
}
if (!entry.isFile()) continue;
const relPath = path.relative(root, fullPath).split(path.sep).join("/");
const ext = relPath.split(".").pop()?.toLowerCase() ?? "";
if (!TEXT_EXTENSIONS.has(ext)) continue;
const bytes = await fs.readFile(fullPath);
files.push({ relPath, bytes });
}
}
await walk(root);
files.sort((left, right) => left.relPath.localeCompare(right.relPath));
return files;
async function collectReleaseFiles(root) {
await validateSelfContainedRelease(root);
return listReleaseFiles(root);
}
function buildFingerprint(files) {
@@ -421,7 +357,7 @@ async function publishSkill({ registry, token, skill, files, version, changelog,
);
for (const file of files) {
form.append("files", new Blob([file.bytes], { type: "text/plain" }), file.relPath);
form.append("files", new Blob([file.bytes], { type: mimeType(file.relPath) }), file.relPath);
}
const response = await fetch(`${registry}/api/v1/skills`, {
+36 -5
View File
@@ -1,7 +1,7 @@
---
name: baoyu-article-illustrator
description: Analyzes article structure, identifies positions requiring visual aids, generates illustrations with Type × Style × Palette three-dimension approach. Use when user asks to "illustrate article", "add images", "generate images for article", or "为文章配图".
version: 1.57.0
version: 1.58.0
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-article-illustrator
@@ -23,16 +23,31 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## Image Generation Tools
When this skill needs to render an image:
When this skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends. Users change the pinned backend via the `## Changing Preferences` section below.
**Prompt file requirement (hard)**: write each image's full, final prompt to a standalone file under `prompts/` (naming: `NN-{type}-[slug].md`) BEFORE invoking any backend. The backend receives the prompt file (or its content); the file is the reproducibility record and lets you switch backends without regenerating prompts.
Concrete tool names (`imagegen`, `image_generate`, `baoyu-imagine`) above are examples — substitute the local equivalents under the same rule.
## Confirmation Policy
Default behavior: **confirm before generation**.
- Treat explicit skill invocation, a file path, matched signals/presets, and `EXTEND.md` defaults as **recommendation inputs only**. None of them authorizes skipping confirmation.
- Do **not** start Step 4 or later until the user completes Step 3.
- Skip confirmation only when the current request explicitly says to do so, for example: "直接生成", "不用确认", "跳过确认", "按默认出图", or equivalent wording.
- If confirmation is skipped explicitly, state the assumed type / density / style / palette / language / backend in the next user-facing update before generating.
## Reference Images
Users may supply reference images via `--ref <files...>` or by providing file paths / pasting images in conversation. Refs guide style, palette, composition, or subject for specific illustrations.
@@ -111,6 +126,8 @@ Full procedures: [references/workflow.md](references/workflow.md#step-2-setup--a
### Step 3: Confirm Settings ⚠️
**Hard gate**: this step is mandatory per the [Confirmation Policy](#confirmation-policy) — Steps 4+ cannot start until the user confirms here (or explicitly opts out with "直接生成" / equivalent wording in the current request).
**ONE AskUserQuestion, max 4 Qs. Q1-Q2 REQUIRED. Q3 required unless preset chosen.**
| Q | Options |
@@ -207,3 +224,17 @@ When input is **pasted content** (no file path), always uses `illustrations/{top
| [references/style-presets.md](references/style-presets.md) | Preset shortcuts (type + style + palette) |
| [references/prompt-construction.md](references/prompt-construction.md) | Prompt templates |
| [references/config/first-time-setup.md](references/config/first-time-setup.md) | First-time setup |
## Changing Preferences
EXTEND.md lives at the first matching path listed in Step 1.5. Three ways to change it:
- **Edit directly** — open EXTEND.md and change fields. Full schema: `references/config/preferences-schema.md`.
- **Reconfigure interactively** — delete EXTEND.md (or ask "reconfigure baoyu-article-illustrator preferences" / "重新配置"). The next run re-triggers first-time setup.
- **Common one-line edits**:
- `preferred_image_backend: auto` — default; runtime-native tool wins, falls back to the only installed backend, asks only if multiple non-native are present.
- `preferred_image_backend: codex-imagegen` — pin to Codex's built-in.
- `preferred_image_backend: baoyu-imagine` — pin to the baoyu-imagine skill.
- `preferred_image_backend: ask` — confirm backend every run.
- `preferred_type: infographic`, `preferred_style: notion`, `preferred_palette: macaron`, `language: zh`.
- `default_output_dir: imgs-subdir` — where to write generated images relative to the article.
@@ -61,8 +61,10 @@ Position defaults to bottom-right.
header: "Style"
question: "Default illustration style preference? Or type another style name or your custom style"
options:
- label: "None (Recommended)"
description: "Auto-select based on content analysis"
- label: "sketch-notes (Recommended)"
description: "Warm cream paper, black hand-drawn lines, soft pastel blocks — educational infographic feel. Great default for most articles."
- label: "None"
description: "Auto-select based on content analysis (falls back to sketch-notes when no strong signal)"
- label: "notion"
description: "Minimalist hand-drawn line art"
- label: "warm"
@@ -126,13 +128,13 @@ preferred_style:
description: ""
default_output_dir: imgs-subdir # same-dir | imgs-subdir | illustrations-subdir | independent
language: null
preferred_image_backend: auto
custom_styles: []
---
```
`preferred_image_backend: auto` is the baked-in default — first-time setup does not ask about it. The `## Image Generation Tools` rule in SKILL.md then picks the runtime-native tool (Codex `imagegen`, Hermes `image_generate`, etc.) when available, and falls back to installed backends.
## Modifying Preferences Later
Users can edit EXTEND.md directly or run setup again:
- Delete EXTEND.md to trigger setup
- Edit YAML frontmatter for quick changes
- Full schema: `config/preferences-schema.md`
See the `## Changing Preferences` section in `SKILL.md` for the canonical list of common edits (pin backend, change defaults, retrigger setup). Full schema: `preferences-schema.md`.
@@ -26,6 +26,8 @@ language: null # zh|en|ja|ko|auto
default_output_dir: null # same-dir|illustrations-subdir|independent
preferred_image_backend: auto # auto|ask|<backend-id>
custom_styles:
- name: my-style
description: "Style description"
@@ -52,6 +54,7 @@ custom_styles:
| `preferred_palette` | string | null | Palette override (macaron, warm, neon, or null) |
| `language` | string | null | Output language (null = auto-detect) |
| `default_output_dir` | enum | null | Output directory preference (null = ask each time) |
| `preferred_image_backend` | string | `auto` | Image backend selection. `auto` = prefer runtime-native tool, fall back to the only installed backend, ask if multiple non-native are present. `ask` = always confirm on every run. `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) = pin this backend when available; fall back to `auto` when it isn't. Absent = `auto`. Resolution logic is documented in `SKILL.md`'s `## Image Generation Tools` section. |
| `custom_styles` | array | [] | User-defined styles |
## Position Options
@@ -113,6 +116,8 @@ preferred_style:
language: zh
preferred_image_backend: codex-imagegen
custom_styles:
- name: corporate
description: "Professional B2B style"
@@ -139,6 +139,39 @@ STYLE: [style characteristics]
ASPECT: 16:9
```
**Infographic + sketch-notes + macaron palette** (default / `hand-drawn-edu` preset):
```
Single-page hand-drawn educational infographic in a clean presentation style.
Warm cream paper background, black hand-drawn lines with slight wobble, soft
pastel color blocks. Feels simple, friendly, and easy to understand at a glance.
Diagram-style visuals ONLY — no realistic or photographic images.
PALETTE: macaron — soft pastel blocks on warm cream
COLORS: Warm Cream background (#F5F0E8); Black (#1A1A1A) for ALL lines, text,
arrows, and doodles; section fills in Light Blue (#A8D8EA), Mint Green
(#B5E5CF), Lavender (#D5C6E0), Peach (#FFD5C2); Coral Red (#E8655A)
sparingly for one or two emphasis points only.
LAYOUT (top → bottom):
- TOP: Bold hand-lettered title, oversized, slightly wobbly, with an optional
decorative underline or small doodle.
- MIDDLE: 26 rounded-rectangle info boxes arranged in a clean grid, row, or
radial pattern. Each box = one section, one pastel fill color, one
simple icon or sketchy cartoon element, one short keyword/phrase.
Hand-drawn arrows connect related zones.
- BOTTOM: One short hand-lettered takeaway sentence summarizing the main idea.
ELEMENTS: Rounded info boxes with clear sectioning, wavy/straight hand-drawn
arrows with small inline labels, simple icons and sketchy cartoon
elements (stick figures, tools, objects), small doodle decorations
(stars, sparkles, underlines, dots, asterisks) used sparingly.
STYLE: Minimal, well-organized, airy. Color fills don't completely fill
outlines (slight "hand-painted" overshoot). ALL text hand-lettered —
no computer fonts. Short labels and keywords only, never long
paragraphs. Generous white space between sections.
```
**Infographic + vector-illustration**:
```
Flat vector illustration infographic. Clean black outlines on all elements.
@@ -2,6 +2,10 @@
`--preset X` expands to a type + style + optional palette combination. Users can override any dimension.
## Default Preset
When content analysis surfaces no strong signal (generic knowledge article, mixed-topic post, no clear data/comparison/narrative cue), recommend **`hand-drawn-edu`** as the primary option in Step 3 Q1. It is the warm, friendly educational-infographic default — safe for most articles and universally readable.
## By Category
### Technical & Engineering
@@ -23,7 +27,9 @@
| `process-flow` | `flowchart` | `notion` | — | Workflow documentation, onboarding flows |
| `warm-knowledge` | `infographic` | `vector-illustration` | `warm` | Product showcases, team intros, feature cards, brand content |
| `edu-visual` | `infographic` | `vector-illustration` | `macaron` | Knowledge summaries, concept explainers, educational articles |
| `hand-drawn-edu` | `flowchart` | `sketch-notes` | `macaron` | Hand-drawn educational diagrams, process explainers, onboarding visuals |
| `hand-drawn-edu` | `infographic` | `sketch-notes` | `macaron` | **Default preset.** Hand-drawn educational infographic — warm cream paper, black lines, pastel blocks. Great for single-page explainers, concept summaries, onboarding, general knowledge articles |
| `hand-drawn-edu-flow` | `flowchart` | `sketch-notes` | `macaron` | Hand-drawn process explainer — step-by-step workflow in the same warm educational style |
| `hand-drawn-edu-compare` | `comparison` | `sketch-notes` | `macaron` | Hand-drawn side-by-side comparison in the warm educational style |
| `ink-notes-compare` | `comparison` | `ink-notes` | `mono-ink` | Before/After essays, Traditional vs New, OS-style comparisons, mindset-shift narratives |
| `ink-notes-flow` | `flowchart` | `ink-notes` | `mono-ink` | Professional process explainers, workforce pipelines, hand-drawn technical walkthroughs |
| `ink-notes-framework` | `framework` | `ink-notes` | `mono-ink` | System analogies, command-center diagrams, architecture-as-metaphor, tech manifestos |
@@ -59,18 +65,19 @@ Use this table during Step 3 to recommend presets based on Step 2 content analys
| Content Type (Step 2) | Primary Preset | Alternatives |
|------------------------|----------------|--------------|
| Technical | `tech-explainer` | `system-design`, `architecture` |
| Tutorial | `tutorial` | `process-flow`, `knowledge-base`, `edu-visual` |
| **General / No strong signal** | `hand-drawn-edu` | `edu-visual`, `knowledge-base` |
| Education / Knowledge | `hand-drawn-edu` | `edu-visual`, `knowledge-base`, `tutorial` |
| Tutorial | `hand-drawn-edu-flow` | `tutorial`, `process-flow`, `hand-drawn-edu` |
| SaaS / Product | `hand-drawn-edu` | `saas-guide`, `knowledge-base`, `process-flow`, `warm-knowledge` |
| Technical | `tech-explainer` | `system-design`, `architecture`, `hand-drawn-edu` |
| Methodology / Framework | `system-design` | `architecture`, `process-flow` |
| Data / Metrics | `data-report` | `versus`, `tech-explainer` |
| Comparison / Review | `versus` | `business-compare`, `editorial-poster`, `ink-notes-compare` |
| Comparison / Review | `versus` | `business-compare`, `hand-drawn-edu-compare`, `editorial-poster`, `ink-notes-compare` |
| Manifesto / Mindset shift / Professional visual note | `ink-notes-compare` | `ink-notes-framework`, `ink-notes-flow` |
| Narrative / Personal | `storytelling` | `lifestyle`, `evolution` |
| Opinion / Editorial | `opinion-piece` | `cinematic`, `editorial-poster` |
| Historical / Timeline | `history` | `evolution` |
| Academic / Research | `science-paper` | `tech-explainer`, `data-report` |
| SaaS / Product | `saas-guide` | `knowledge-base`, `process-flow`, `warm-knowledge` |
| Education / Knowledge | `edu-visual` | `knowledge-base`, `tutorial`, `hand-drawn-edu` |
## Override Examples
@@ -6,15 +6,15 @@ Simplified style tier for quick selection:
| Core Style | Maps To | Best For |
|------------|---------|----------|
| `hand-drawn` | sketch-notes | **Default.** Warm cream paper, black hand-drawn lines, pastel blocks — educational infographics, concept explainers, onboarding, general knowledge articles |
| `vector` | vector-illustration | Knowledge articles, tutorials, tech content |
| `minimal-flat` | notion | General, knowledge sharing, SaaS |
| `sci-fi` | blueprint | AI, frontier tech, system design |
| `hand-drawn` | sketch/warm | Relaxed, reflective, casual content |
| `editorial` | editorial | Processes, data, journalism |
| `scene` | warm/watercolor | Narratives, emotional, lifestyle |
| `poster` | screen-print | Opinion, editorial, cultural, cinematic |
Use Core Styles for most cases. See full Style Gallery below for granular control.
Use Core Styles for most cases. **When no strong content signal is detected, default to `hand-drawn` (→ sketch-notes).** See full Style Gallery below for granular control.
---
@@ -50,42 +50,45 @@ Full specifications: `references/styles/<style>.md`
## Type × Style Compatibility Matrix
| | vector-illustration | notion | warm | minimal | blueprint | watercolor | elegant | editorial | scientific | screen-print |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| infographic | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓ |
| scene | ✓ | ✓ | ✓✓ | ✓ | ✗ | ✓✓ | ✓ | ✓ | ✗ | ✓✓ |
| flowchart | ✓✓ | ✓✓ | ✓ | ✓ | ✓✓ | ✗ | ✓ | ✓✓ | ✓ | ✗ |
| comparison | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓ | ✓ | ✓✓ | ✓✓ | ✓ | ✓ |
| framework | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✓ | ✓✓ | ✓ |
| timeline | ✓ | ✓✓ | ✓ | ✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓ |
| | sketch-notes | vector-illustration | notion | warm | minimal | blueprint | watercolor | elegant | editorial | scientific | screen-print |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| infographic | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓ |
| scene | ✗ | ✓ | ✓ | ✓✓ | ✓ | ✗ | ✓✓ | ✓ | ✓ | ✗ | ✓✓ |
| flowchart | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓ | ✓✓ | ✗ | ✓ | ✓✓ | ✓ | ✗ |
| comparison | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓ | ✓ | ✓✓ | ✓✓ | ✓ | ✓ |
| framework | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✓ | ✓✓ | ✓ |
| timeline | ✓ | ✓ | ✓✓ | ✓ | ✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓ |
✓✓ = highly recommended | ✓ = compatible | ✗ = not recommended
## Auto Selection by Type
When no content signal matches strongly, `sketch-notes` is the default primary for every diagrammatic type. Only override with another primary when the content analysis in Step 2 surfaces a clear signal (technical/data/narrative/opinion).
| Type | Primary Style | Secondary Styles |
|------|---------------|------------------|
| infographic | vector-illustration | notion, blueprint, editorial |
| infographic | sketch-notes | vector-illustration, notion, blueprint, editorial |
| scene | warm | watercolor, elegant |
| flowchart | vector-illustration | notion, blueprint |
| comparison | vector-illustration | notion, elegant |
| framework | blueprint | vector-illustration, notion |
| timeline | elegant | warm, editorial |
| flowchart | sketch-notes | vector-illustration, notion, blueprint |
| comparison | sketch-notes | vector-illustration, notion, elegant |
| framework | sketch-notes | blueprint, vector-illustration, notion |
| timeline | elegant | sketch-notes, warm, editorial |
## Auto Selection by Content Signals
| Content Signals | Recommended Type | Recommended Style |
|-----------------|------------------|-------------------|
| **(no strong signal / general article)** | **infographic** | **sketch-notes** |
| Knowledge, concept, tutorial, learning, guide, onboarding | infographic | sketch-notes, vector-illustration, notion |
| Productivity, SaaS, tool, app, software | infographic | sketch-notes, notion, vector-illustration |
| How-to, steps, workflow, process, tutorial | flowchart | sketch-notes, vector-illustration, notion |
| API, metrics, data, comparison, numbers | infographic | blueprint, vector-illustration |
| Knowledge, concept, tutorial, learning, guide | infographic | vector-illustration, notion |
| Tech, AI, programming, development, code | infographic | vector-illustration, blueprint |
| How-to, steps, workflow, process, tutorial | flowchart | vector-illustration, notion |
| Framework, model, architecture, principles | framework | blueprint, vector-illustration |
| vs, pros/cons, before/after, alternatives | comparison | vector-illustration, notion |
| Tech, AI, programming, development, code | infographic | vector-illustration, blueprint, sketch-notes |
| Framework, model, architecture, principles | framework | blueprint, vector-illustration, sketch-notes |
| vs, pros/cons, before/after, alternatives | comparison | vector-illustration, notion, sketch-notes |
| Manifesto, mindset shift, workforce, OS, whiteboard, professional visual note | comparison / framework | ink-notes |
| Story, emotion, journey, experience, personal | scene | warm, watercolor |
| History, timeline, progress, evolution | timeline | elegant, warm |
| Productivity, SaaS, tool, app, software | infographic | notion, vector-illustration |
| Business, professional, strategy, corporate | framework | elegant |
| Opinion, editorial, culture, philosophy, cinematic, dramatic, poster | scene | screen-print |
| Biology, chemistry, medical, scientific | infographic | scientific |
@@ -93,6 +96,15 @@ Full specifications: `references/styles/<style>.md`
## Style Characteristics by Type
### infographic + sketch-notes (default)
- Warm cream paper background, black hand-drawn lines with slight wobble
- 26 rounded pastel info boxes (light blue / mint / lavender / peach)
- Bold hand-lettered title at the top
- Short keyword labels, simple icons, small doodles (stars, underlines, sparkles)
- One-line hand-lettered takeaway sentence at the bottom
- Airy, minimal, diagram-style — never realistic
- Perfect for single-page educational explainers and concept summaries
### infographic + vector-illustration
- Clean flat vector shapes, bold geometric forms
- Vibrant but harmonious color palette
@@ -1,56 +1,91 @@
# sketch-notes
Soft hand-drawn illustration style with warm, educational feel
Hand-drawn educational infographic style with warm cream paper, black hand-drawn lines, and soft pastel section blocks. Optimized for single-page visual explainers.
## Design Aesthetic
Hand-drawn feel with soft, relaxed brush strokes. Fresh, refined style with minimalist editorial approach. Emphasis on precision, clarity and intelligent elegance while prioritizing warmth, approachability and friendliness.
Hand-drawn educational infographic in a clean presentation style. Feels like a visual explainer slide: simple, friendly, and easy to understand at a glance. Bold handwritten-style title at the top, clearly sectioned content in the middle with rounded boxes and small doodles, and one short takeaway sentence at the bottom. Neat, airy, and visually similar to a hand-drawn concept diagram — never realistic or photographic.
## Background
- Color: Warm Off-White (#FAF8F0)
- Texture: Subtle paper grain, warm tone
- Color: Warm Cream Paper (#F5F0E8) — preferred; fallback Warm Off-White (#FAF8F0)
- Texture: Subtle warm paper grain, matte finish, no gloss
## Color Palette
Default sketch-notes palette is the **macaron** pastel set. Lines are always black; pastel blocks are used only as rounded card fills for information sections.
| Role | Color | Hex | Usage |
|------|-------|-----|-------|
| Background | Warm Off-White | #FAF8F0 | Primary background |
| Primary Text | Deep Charcoal | #2C3E50 | Main elements |
| Alt Text | Deep Brown | #4A4A4A | Secondary elements |
| Accent 1 | Soft Orange | #F4A261 | Highlights, emphasis |
| Accent 2 | Mustard Yellow | #E9C46A | Secondary highlights |
| Accent 3 | Sage Green | #87A96B | Nature, growth concepts |
| Accent 4 | Light Blue | #7EC8E3 | Tech, digital elements |
| Accent 5 | Red Brown | #A0522D | Earthy elements |
| Background | Warm Cream | #F5F0E8 | Paper background |
| Primary Ink | Black | #1A1A1A | ALL outlines, text, arrows, doodles |
| Block Blue | Light Blue | #A8D8EA | Info block fill (cool / tech) |
| Block Mint | Mint Green | #B5E5CF | Info block fill (growth / positive) |
| Block Lavender | Lavender | #D5C6E0 | Info block fill (concept / abstract) |
| Block Peach | Peach | #FFD5C2 | Info block fill (warm / human) |
| Accent | Coral Red | #E8655A | One or two emphasis points only |
| Muted Text | Warm Gray | #6B6B6B | Small annotations |
Use **4 pastel block colors max** per image, one color per section. Black ink does all the structural line work.
## Visual Elements
- Connection lines with hand-drawn wavy feel
- Conceptual abstract icons illustrating ideas
- Color fills don't completely fill outlines (hand-painted feel)
- Simple geometric shapes with rounded corners
- Arrows and pointers with sketchy style
- Doodle decorations: stars, spirals, underlines
- Bold hand-lettered title at the top (oversized, slightly wobbly)
- Rounded-rectangle info boxes with clear sectioning (26 zones)
- Short keyword labels inside boxes — never long paragraphs
- Simple icons and sketchy cartoon elements (stick figures, tools, objects) to explain each idea
- Hand-drawn arrows (straight, curved, or wavy) connecting related zones
- Small doodle decorations: stars, sparkles, underlines, dots, asterisks — used sparingly for emphasis
- Single-line hand-lettered takeaway sentence at the bottom
- Color fills do not completely fill outlines (slight "hand-painted" overshoot/undershoot)
- Generous white space between sections — airy, never crowded
## Layout Guidelines
Canonical single-page layout (16:9 or 4:3):
1. **Top (1015%)** — Bold hand-lettered title, optionally with a small decorative underline or doodle.
2. **Middle (7080%)** — 26 rounded pastel info boxes arranged in a clear grid, row, or radial pattern. Each box = one section, one color, one icon, one keyword/phrase.
3. **Bottom (1015%)** — One short hand-lettered takeaway sentence summarizing the core insight.
Keep margins generous. Aim for breathing room around every element.
## Style Rules
### Do
- Keep layouts open and well-structured
- Emphasize information hierarchy
- Use hand-drawn quality for all elements
- Allow imperfection (slight wobbles add character)
- Layer elements with subtle overlaps
- Use warm cream paper background (no pure white)
- Use black hand-drawn lines for ALL structural elements
- Use soft pastel blocks (blue / mint / lavender / peach) for section fills
- Keep text to short keywords and phrases only
- Include a bold handwritten title at the top
- Include a short takeaway sentence at the bottom
- Use diagram-style visuals (icons, doodles, simple shapes)
- Allow slight wobble — hand-drawn imperfection is the point
- Maintain clear sectioning with rounded boxes
### Don't
- Use perfect geometric shapes
- Create photorealistic elements
- Overcrowd with too many elements
- Use pure white backgrounds
- Make it look computer-generated
- Use pure white backgrounds (that's `ink-notes`' territory)
- Render realistic or photographic images — this style is diagram-only
- Fill zones with gradients, shadows, or digital effects
- Use long paragraphs of text — keywords only
- Use computer-generated / sans-serif body fonts — ALL text must be hand-lettered
- Use more than 4 pastel block colors per image
- Overcrowd the canvas — keep it airy and minimal
- Use perfect geometric shapes — preserve the hand-drawn wobble
## Type Compatibility
| Type | Rating | Notes |
|------|--------|-------|
| infographic | ✓✓ | **Best fit** — single-page visual explainers, concept summaries, educational slides |
| framework | ✓✓ | Labeled zones and connectors render well |
| flowchart | ✓✓ | Rounded step boxes with wavy arrows |
| comparison | ✓✓ | Two pastel blocks side by side; prefer `ink-notes` for strict Before/After contrasts |
| timeline | ✓ | Hand-drawn horizontal arrow with milestone cards |
| scene | ✗ | Not recommended — too diagrammatic |
## Best For
Educational content, knowledge sharing, technical explanations, tutorials, onboarding materials, friendly articles
Educational content, knowledge sharing, concept explainers, tutorials, onboarding materials, product walkthroughs, single-page visual summaries, "how things work" posts, friendly technical articles
@@ -176,6 +176,8 @@ Based on Step 2 content analysis, recommend a preset first (sets both type & sty
- [Alternative preset] — [brief]
- Or choose type manually: infographic / scene / flowchart / comparison / framework / timeline / mixed
**Default**: if Step 2 found no strong content signal, the recommended preset MUST be `hand-drawn-edu` (infographic + sketch-notes + macaron — warm cream paper, black hand-drawn lines, soft pastel blocks). This is the universal fallback.
**If user picks a preset → skip Q3** (type & style both resolved).
**If user picks a type → Q3 is REQUIRED.**
@@ -203,13 +205,15 @@ If no `preferred_style` (present Core Styles first):
| Core Style | Maps To | Best For |
|------------|---------|----------|
| `hand-drawn` | sketch-notes | **Default.** Warm cream paper, black hand-drawn lines, pastel blocks — educational infographics, concept explainers, onboarding, general knowledge articles |
| `minimal-flat` | notion | General, knowledge sharing, SaaS |
| `sci-fi` | blueprint | AI, frontier tech, system design |
| `hand-drawn` | sketch/warm | Relaxed, reflective, casual |
| `editorial` | editorial | Processes, data, journalism |
| `scene` | warm/watercolor | Narratives, emotional, lifestyle |
| `poster` | screen-print | Opinion, editorial, cultural, cinematic |
**Default recommendation**: when Step 2 surfaces no strong content signal, recommend **`hand-drawn-edu`** preset (→ infographic + sketch-notes + macaron) as the primary option in Q1. When the user picks a type manually without a preferred_style, recommend `sketch-notes` first in Q3.
Style selection based on Type × Style compatibility matrix (styles.md).
**In Step 5.1**, read `styles/<style>.md` for visual elements and rendering rules.
+23 -4
View File
@@ -27,11 +27,17 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## Image Generation Tools
When this skill needs to render an image:
When this skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends. Users change the pinned backend via the `## Changing Preferences` section below.
**Prompt file requirement (hard)**: write each image's full, final prompt to a standalone file under `prompts/` (naming: `NN-{type}-[slug].md`) BEFORE invoking any backend. The backend receives the prompt file (or its content); the file is the reproducibility record and lets you switch backends without regenerating prompts.
@@ -295,3 +301,16 @@ If EXTEND.md is not found, first-time setup is **blocking** — complete it befo
- **Step 7.1 character sheet** - recommended for multi-page comics, optional for simple presets
- **Step 7.2 character reference** - use `--ref` if sheet exists; compress/convert on failure; fall back to prompt-only
- Watermark/language configured once in EXTEND.md
## Changing Preferences
EXTEND.md lives at `.baoyu-skills/baoyu-comic/EXTEND.md` (project) or `~/.baoyu-skills/baoyu-comic/EXTEND.md` (user). Three ways to change it:
- **Edit directly** — open EXTEND.md and change fields. Full schema: `references/config/preferences-schema.md`.
- **Reconfigure interactively** — delete EXTEND.md (or ask "reconfigure baoyu-comic preferences" / "重新配置"). The next run re-triggers first-time setup.
- **Common one-line edits**:
- `preferred_image_backend: auto` — default; runtime-native tool wins, falls back to the only installed backend, asks only if multiple non-native are present.
- `preferred_image_backend: codex-imagegen` — pin to Codex's built-in.
- `preferred_image_backend: baoyu-imagine` — pin to the baoyu-imagine skill.
- `preferred_image_backend: ask` — confirm backend every run.
- `watermark.enabled: true`, `preferred_art`, `preferred_tone`, `preferred_layout`, `language` — shift the auto-selection defaults and cosmetic choices.
@@ -142,13 +142,13 @@ preferred_tone: [selected tone or null]
preferred_layout: null
preferred_aspect: null
language: [selected or null]
preferred_image_backend: auto
character_presets: []
---
```
`preferred_image_backend: auto` is the baked-in default — first-time setup does not ask about it. The `## Image Generation Tools` rule in SKILL.md then picks the runtime-native tool (Codex `imagegen`, Hermes `image_generate`, etc.) when available, and falls back to installed backends.
## Modifying Preferences Later
Users can edit EXTEND.md directly or run setup again:
- Delete EXTEND.md to trigger setup
- Edit YAML frontmatter for quick changes
- Full schema: `config/preferences-schema.md`
See the `## Changing Preferences` section in `SKILL.md` for the canonical list of common edits (pin backend, change defaults, retrigger setup). Full schema: `config/preferences-schema.md`.
@@ -23,6 +23,8 @@ preferred_aspect: null # 3:4|4:3|16:9
language: null # zh|en|ja|ko|auto
preferred_image_backend: auto # auto|ask|<backend-id>
character_presets:
- name: my-characters
roles:
@@ -46,6 +48,7 @@ character_presets:
| `preferred_layout` | string | null | Layout preference or null |
| `preferred_aspect` | string | null | Aspect ratio (3:4, 4:3, 16:9) |
| `language` | string | null | Output language (null = auto-detect) |
| `preferred_image_backend` | string | `auto` | Image backend selection. `auto` = prefer runtime-native tool, fall back to the only installed backend, ask if multiple non-native are present. `ask` = always confirm on every run. `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) = pin this backend when available; fall back to `auto` when it isn't. Absent = `auto`. Resolution logic is documented in `SKILL.md`'s `## Image Generation Tools` section. |
| `character_presets` | array | [] | Preset character roles for styles like ohmsha |
## Art Style Options
@@ -122,6 +125,8 @@ preferred_aspect: "3:4"
language: zh
preferred_image_backend: codex-imagegen
character_presets:
- name: tech-tutorial
roles:
+32 -10
View File
@@ -1,7 +1,7 @@
---
name: baoyu-cover-image
description: Generates article cover images with 5 dimensions (type, palette, rendering, text, mood) combining 11 color palettes and 7 rendering styles. Supports cinematic (2.35:1), widescreen (16:9), and square (1:1) aspects. Use when user asks to "generate cover image", "create article cover", or "make cover".
version: 1.56.1
version: 1.56.2
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-cover-image
@@ -23,16 +23,31 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## Image Generation Tools
When this skill needs to render an image:
When this skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends. Users change the pinned backend via the `## Changing Preferences` section below.
**Prompt file requirement (hard)**: write each image's full, final prompt to a standalone file under `prompts/` (naming: `NN-{type}-[slug].md`) BEFORE invoking any backend. The backend receives the prompt file (or its content); the file is the reproducibility record and lets you switch backends without regenerating prompts.
Concrete tool names (`imagegen`, `image_generate`, `baoyu-imagine`) above are examples — substitute the local equivalents under the same rule.
## Confirmation Policy
Default behavior: **confirm before generation**.
- Treat explicit skill invocation, a file path, matched keywords/presets, `EXTEND.md` defaults, and any documented auto-selection as **recommendation inputs only**. None of them authorizes skipping confirmation.
- Do **not** start Step 3 or Step 4 until the user confirms the dimensions / aspect / language / backend choices.
- Skip confirmation only when the current request explicitly says to do so, for example: `--quick`, "直接生成", "不用确认", "跳过确认", "按默认出图", or equivalent wording. `quick_mode: true` in `EXTEND.md` counts as a standing explicit opt-out — set it only when you want every run to skip Step 2.
- If confirmation is skipped explicitly, state the assumed dimensions / aspect / language / backend in the next user-facing update before generating.
## Options
| Option | Description |
@@ -164,6 +179,8 @@ See [reference-images.md](references/workflow/reference-images.md) for full deci
### Step 2: Confirm Options ⚠️
**Hard gate**: this step is mandatory per the [Confirmation Policy](#confirmation-policy) — Steps 34 cannot start until the user confirms here (or explicitly opts out with `--quick` / `quick_mode: true` / equivalent wording in the current request).
**MUST use `AskUserQuestion` tool** to present options as interactive selection — NOT plain text tables. Present up to 4 questions in a single `AskUserQuestion` call (Type, Palette, Rendering, Font + Settings). Each question shows the recommended option first with reason, followed by alternatives.
Full confirmation flow and question format: [references/workflow/confirm-options.md](references/workflow/confirm-options.md)
@@ -228,13 +245,18 @@ Files:
- **Characters**: Simplified silhouettes; NO realistic humans
- **Title**: Use exact title from user/source; never invent
## Extension Support
## Changing Preferences
Custom configurations via EXTEND.md. See **Step 0** for paths.
EXTEND.md lives at the path noted in **Step 0**. Three ways to change it:
Supports: Watermark | Preferred dimensions | Default aspect/output | Quick mode | Custom palettes | Language
Schema: [references/config/preferences-schema.md](references/config/preferences-schema.md)
- **Edit directly** — open EXTEND.md and change fields. Full schema: [references/config/preferences-schema.md](references/config/preferences-schema.md).
- **Reconfigure interactively** — delete EXTEND.md (or ask "reconfigure baoyu-cover-image preferences" / "重新配置"). The next run re-triggers first-time setup.
- **Common one-line edits**:
- `preferred_image_backend: auto` — default; runtime-native tool wins, falls back to the only installed backend, asks only if multiple non-native are present.
- `preferred_image_backend: codex-imagegen` — pin to Codex's built-in.
- `preferred_image_backend: baoyu-imagine` — pin to the baoyu-imagine skill.
- `preferred_image_backend: ask` — confirm backend every run.
- `watermark.enabled: true`, `preferred_type`, `preferred_palette`, `preferred_rendering`, `default_aspect`, `quick_mode: true`, `language` — shift the auto-selection defaults and confirmation flow.
## References
@@ -188,15 +188,15 @@ default_aspect: [16:9/2.35:1/1:1/3:4]
default_output_dir: [independent/same-dir/imgs-subdir]
quick_mode: [true/false]
language: null
preferred_image_backend: auto
custom_palettes: []
---
```
`preferred_image_backend: auto` is the baked-in default — first-time setup does not ask about it. The `## Image Generation Tools` rule in SKILL.md then picks the runtime-native tool (Codex `imagegen`, Hermes `image_generate`, etc.) when available, and falls back to installed backends.
## Modifying Preferences Later
Users can edit EXTEND.md directly or run setup again:
- Delete EXTEND.md to trigger setup
- Edit YAML frontmatter for quick changes
- Full schema: `preferences-schema.md`
See the `## Changing Preferences` section in `SKILL.md` for the canonical list of common edits (pin backend, change defaults, retrigger setup). Full schema: `preferences-schema.md`.
**EXTEND.md Supports**: Watermark | Preferred type | Preferred palette | Preferred rendering | Preferred text | Preferred mood | Default aspect ratio | Default output directory | Quick mode | Custom palette definitions | Language preference
**EXTEND.md Supports**: Watermark | Preferred type | Preferred palette | Preferred rendering | Preferred text | Preferred mood | Default aspect ratio | Default output directory | Quick mode | Image backend preference | Custom palette definitions | Language preference
@@ -32,6 +32,8 @@ quick_mode: false # Skip confirmation when true
language: null # zh|en|ja|ko|auto (null = auto-detect)
preferred_image_backend: auto # auto|ask|<backend-id>
custom_palettes:
- name: my-palette
description: "Palette description"
@@ -60,6 +62,7 @@ custom_palettes:
| `default_aspect` | string | "2.35:1" | Default aspect ratio |
| `quick_mode` | bool | false | Skip confirmation step |
| `language` | string | null | Output language (null = auto-detect) |
| `preferred_image_backend` | string | `auto` | Image backend selection. `auto` = prefer runtime-native tool, fall back to the only installed backend, ask if multiple non-native are present. `ask` = always confirm on every run. `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) = pin this backend when available; fall back to `auto` when it isn't. Absent = `auto`. Resolution logic is documented in `SKILL.md`'s `## Image Generation Tools` section. |
| `custom_palettes` | array | [] | User-defined palettes |
## Type Options
@@ -187,6 +190,8 @@ quick_mode: true
language: en
preferred_image_backend: codex-imagegen
custom_palettes:
- name: corporate-tech
description: "Professional B2B tech palette"
@@ -0,0 +1,25 @@
import assert from "node:assert/strict";
import test from "node:test";
import { collect_generated_image_urls_from_response_parts } from "./client.ts";
test("response part fallback finds generated images when legacy generated markers are absent", () => {
const generatedUrl = "https://lh3.googleusercontent.com/gg-dl/example-generated-image";
const initialCandidate = ["rcid-1", ["image generated successfully"]];
const imageCandidate = [
"rcid-1",
["image generated successfully"],
{ nestedPayload: [{ media: generatedUrl }] },
];
const responseJson = [
["wrb.fr", null, JSON.stringify([null, [], null, null, [initialCandidate]])],
["wrb.fr", null, JSON.stringify([null, [], null, null, [imageCandidate]])],
];
assert.equal(initialCandidate[12], undefined);
assert.equal(
/http:\/\/googleusercontent\.com\/image_generation_content\/\d+/.test(String(initialCandidate[1]?.[0])),
false,
);
assert.deepEqual(collect_generated_image_urls_from_response_parts(responseJson, 0, 0), [generatedUrl]);
});
@@ -37,6 +37,8 @@ type InitOptions = {
type RequestKwargs = RequestInit & { timeout_ms?: number };
const GENERATED_IMAGE_URL_PREFIX = 'https://lh3.googleusercontent.com/gg-dl/';
function normalize_headers(h?: HeadersInit): Record<string, string> {
if (!h) return {};
if (Array.isArray(h)) return Object.fromEntries(h.map(([k, v]) => [k, v]));
@@ -78,6 +80,59 @@ function collect_strings(root: unknown, accept: (s: string) => boolean, limit: n
return out;
}
function collect_generated_image_urls(root: unknown, limit: number = 4): string[] {
return collect_strings(root, (s) => s.startsWith(GENERATED_IMAGE_URL_PREFIX), limit);
}
function parse_response_part_body(part: unknown): unknown[] | null {
if (!Array.isArray(part)) return null;
const part_body = get_nested_value<string | null>(part, [2], null);
if (!part_body) return null;
try {
const part_json = JSON.parse(part_body) as unknown;
return Array.isArray(part_json) ? part_json : null;
} catch {
return null;
}
}
function find_generated_image_part(
response_json: unknown[],
body_index: number,
candidate_index: number,
limit: number = 4,
): { body: unknown[]; urls: string[] } | null {
for (let part_index = body_index; part_index < response_json.length; part_index++) {
const part_json = parse_response_part_body(response_json[part_index]);
if (!part_json) continue;
const cand = get_nested_value<unknown>(part_json, [4, candidate_index], null);
if (!cand) continue;
const urls = collect_generated_image_urls(cand, limit);
if (urls.length > 0) return { body: part_json, urls };
}
return null;
}
export function collect_generated_image_urls_from_response_parts(
response_json: unknown[],
body_index: number,
candidate_index: number,
limit: number = 4,
): string[] {
return find_generated_image_part(response_json, body_index, candidate_index, limit)?.urls ?? [];
}
function push_generated_images(
generated_images: GeneratedImage[],
urls: string[],
proxy: string | null,
cookies: Record<string, string>,
): void {
for (const url of urls) {
generated_images.push(new GeneratedImage(url, '[Generated Image]', '', proxy, cookies));
}
}
export class GeminiClient extends GemMixin {
public cookies: Record<string, string> = {};
public proxy: string | null = null;
@@ -404,24 +459,8 @@ export class GeminiClient extends GemMixin {
/http:\/\/googleusercontent\.com\/image_generation_content\/\d+/.test(text);
if (wants_generated) {
let img_body: unknown[] | null = null;
for (let part_index = body_index; part_index < (response_json as unknown[]).length; part_index++) {
const part = (response_json as unknown[])[part_index];
if (!Array.isArray(part)) continue;
const part_body = get_nested_value<string | null>(part, [2], null);
if (!part_body) continue;
try {
const part_json = JSON.parse(part_body) as unknown[];
const cand = get_nested_value<unknown>(part_json, [4, candidate_index], null);
if (!cand) continue;
const urls = collect_strings(cand, (s) => s.startsWith('https://lh3.googleusercontent.com/gg-dl/'), 1);
if (urls.length > 0) {
img_body = part_json;
break;
}
} catch {}
}
const image_part = find_generated_image_part(response_json as unknown[], body_index, candidate_index, 1);
const img_body = image_part?.body ?? null;
if (!img_body) {
throw new ImageGenerationError(
@@ -452,13 +491,22 @@ export class GeminiClient extends GemMixin {
}
if (generated_images.length === 0) {
const urls = collect_strings(img_candidate, (s) => s.startsWith('https://lh3.googleusercontent.com/gg-dl/'), 4);
for (const url of urls) {
generated_images.push(new GeneratedImage(url, '[Generated Image]', '', this.proxy, this.cookies));
}
push_generated_images(generated_images, collect_generated_image_urls(img_candidate), this.proxy, this.cookies);
}
}
// Fallback: unconditionally scan all response parts for generated image URLs.
// The `wants_generated` detection above relies on `candidate[12][7][0]` and an old
// `googleusercontent.com/image_generation_content/` URL pattern, both of which no
// longer appear in the current Gemini Web API response format.
// When Gemini does generate images, their URLs now start with
// `https://lh3.googleusercontent.com/gg-dl/` and are present somewhere in the
// response parts — this fallback finds them so images are not silently dropped.
if (generated_images.length === 0) {
const urls = collect_generated_image_urls_from_response_parts(response_json as unknown[], body_index, candidate_index);
push_generated_images(generated_images, urls, this.proxy, this.cookies);
}
out.push(new Candidate({ rcid, text, thoughts, web_images, generated_images }));
}
+37 -6
View File
@@ -1,7 +1,7 @@
---
name: baoyu-image-cards
description: Generates infographic image card series with 12 visual styles, 8 layouts, and 3 color palettes. Breaks content into 1-10 cartoon-style image cards optimized for social media engagement. Use when user mentions "小红书图片", "小红书种草", "小绿书", "微信图文", "微信贴图", "image cards", "图片卡片", or wants social media infographic series.
version: 1.56.1
version: 1.56.2
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-image-cards
@@ -23,14 +23,31 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## Image Generation Tools
When this skill needs to render an image:
When this skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends. Users change the pinned backend via the `## Changing Preferences` section below.
**Prompt file requirement (hard)**: write each image's full, final prompt to a standalone file under `prompts/` (naming: `NN-{type}-[slug].md`) BEFORE invoking any backend. The file is the reproducibility record and lets you switch backends without regenerating prompts.
Concrete tool names (`imagegen`, `image_generate`, `baoyu-imagine`) above are examples — substitute the local equivalents under the same rule.
## Confirmation Policy
Default behavior: **confirm before generation**.
- Treat explicit skill invocation, a file path, matched signals/presets, and `EXTEND.md` defaults as **recommendation inputs only**. None of them authorizes skipping confirmation.
- Do **not** start Step 3 until the user completes Step 2.
- Skip confirmation only when the current request explicitly says to do so, for example: `--yes`, "直接生成", "不用确认", "跳过确认", "按默认出图", or equivalent wording.
- If confirmation is skipped explicitly, state the assumed strategy / style / layout / palette / count / backend in the next user-facing update before generating.
## Language
Respond in the user's language across questions, progress, errors, and completion summary. Keep technical tokens (style names, file paths, code) in English.
@@ -291,6 +308,8 @@ Check these paths in order; first hit wins:
### Step 2: Smart Confirm ⚠️ REQUIRED
**Hard gate**: this step is mandatory per the [Confirmation Policy](#confirmation-policy) — Step 3 cannot start until the user confirms here (or explicitly opts out with `--yes` / equivalent wording in the current request).
Goal: present the auto-recommended plan and let the user confirm or adjust. Skip this step entirely under `--yes` — proceed with Path A using the analysis and any CLI overrides.
**Display summary** before asking:
@@ -417,4 +436,16 @@ Always update the prompt file before regenerating — it's the source of truth a
- For sensitive public figures, use stylized cartoon alternatives.
- Smart Confirm (Step 2) is required; Detailed mode adds a second confirmation (2a + 2c).
Custom configurations via EXTEND.md. See Step 0 for paths and schema.
## Changing Preferences
EXTEND.md lives at the first matching path listed in Step 0. Three ways to change it:
- **Edit directly** — open EXTEND.md and change fields. Full schema: `references/config/preferences-schema.md`.
- **Reconfigure interactively** — delete EXTEND.md (or ask "reconfigure baoyu-image-cards preferences" / "重新配置"). The next run re-triggers first-time setup.
- **Common one-line edits**:
- `preferred_image_backend: auto` — default; runtime-native tool wins, falls back to the only installed backend, asks only if multiple non-native are present.
- `preferred_image_backend: codex-imagegen` — pin to Codex's built-in.
- `preferred_image_backend: baoyu-imagine` — pin to the baoyu-imagine skill.
- `preferred_image_backend: ask` — confirm backend every run.
- `preferred_style: notion`, `preferred_layout: dense`, `preferred_palette: macaron`, `language: zh`.
- `watermark.enabled: true` + `watermark.content: "@handle"` — add a watermark.
@@ -110,13 +110,13 @@ preferred_style:
description: ""
preferred_layout: null
language: null
preferred_image_backend: auto
custom_styles: []
---
```
`preferred_image_backend: auto` is the baked-in default — first-time setup does not ask about it. The `## Image Generation Tools` rule in SKILL.md then picks the runtime-native tool (Codex `imagegen`, Hermes `image_generate`, etc.) when available, and falls back to installed backends.
## Modifying Preferences Later
Users can edit EXTEND.md directly or run setup again:
- Delete EXTEND.md to trigger setup
- Edit YAML frontmatter for quick changes
- Full schema: `config/preferences-schema.md`
See the `## Changing Preferences` section in `SKILL.md` for the canonical list of common edits (pin backend, change defaults, retrigger setup). Full schema: `preferences-schema.md`.
@@ -24,6 +24,8 @@ preferred_layout: null # sparse|balanced|dense|list|comparison|flow
language: null # zh|en|ja|ko|auto
preferred_image_backend: auto # auto|ask|<backend-id>
custom_styles:
- name: my-style
description: "Style description"
@@ -49,6 +51,7 @@ custom_styles:
| `preferred_style.description` | string | "" | Custom notes/override |
| `preferred_layout` | string | null | Layout preference or null |
| `language` | string | null | Output language (null = auto-detect) |
| `preferred_image_backend` | string | `auto` | Image backend selection. `auto` = prefer runtime-native tool, fall back to the only installed backend, ask if multiple non-native are present. `ask` = always confirm on every run. `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) = pin this backend when available; fall back to `auto` when it isn't. Absent = `auto`. Resolution logic is documented in `SKILL.md`'s `## Image Generation Tools` section. |
| `custom_styles` | array | [] | User-defined styles |
## Position Options
@@ -104,6 +107,8 @@ preferred_layout: dense
language: zh
preferred_image_backend: codex-imagegen
custom_styles:
- name: corporate
description: "Professional B2B style"
+16 -9
View File
@@ -1,7 +1,7 @@
---
name: baoyu-imagine
description: AI image generation with OpenAI, Azure OpenAI, Google, OpenRouter, DashScope, Z.AI GLM-Image, MiniMax, Jimeng, Seedream and Replicate APIs. Supports text-to-image, reference images, aspect ratios, and batch generation from saved prompt files. Sequential by default; use batch parallel generation when the user already has multiple prompts or wants stable multi-image throughput. Use when user asks to generate, create, or draw images.
version: 1.57.0
description: AI image generation with OpenAI GPT Image 2, Azure OpenAI, Google, OpenRouter, DashScope, Z.AI GLM-Image, MiniMax, Jimeng, Seedream and Replicate APIs. Supports text-to-image, reference images, aspect ratios, and batch generation from saved prompt files. Sequential by default; use batch parallel generation when the user already has multiple prompts or wants stable multi-image throughput. Use when user asks to generate, create, or draw images.
version: 1.58.0
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-imagine
@@ -13,7 +13,7 @@ metadata:
# Image Generation (AI SDK)
Official API-based image generation. Supports OpenAI, Azure OpenAI, Google, OpenRouter, DashScope (阿里通义万象), Z.AI GLM-Image, MiniMax, Jimeng (即梦), Seedream (豆包) and Replicate.
Official API-based image generation. Supports OpenAI GPT Image 2, Azure OpenAI, Google, OpenRouter, DashScope (阿里通义万象), Z.AI GLM-Image, MiniMax, Jimeng (即梦), Seedream (豆包) and Replicate.
## User Input Tools
@@ -68,6 +68,9 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "Make blue" --image out.png --ref so
# Specific provider
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider dashscope --model qwen-image-2.0-pro
# OpenAI GPT Image 2
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider openai --model gpt-image-2
# Batch mode
${BUN_X} {baseDir}/scripts/main.ts --batchfile batch.json --jobs 4
```
@@ -84,11 +87,11 @@ ${BUN_X} {baseDir}/scripts/main.ts --batchfile batch.json --jobs 4
| `--provider google\|openai\|azure\|openrouter\|dashscope\|zai\|minimax\|jimeng\|seedream\|replicate` | Force provider (default: auto-detect) |
| `--model <id>`, `-m` | Model ID — see provider references for defaults and allowed values |
| `--ar <ratio>` | Aspect ratio (`16:9`, `1:1`, `4:3`, …) |
| `--size <WxH>` | Explicit size (e.g., `1024x1024`) |
| `--size <WxH>` | Explicit size (e.g., `1024x1024`; for `gpt-image-2`, width/height must be multiples of 16, max edge 3840px, ratio no wider than 3:1) |
| `--quality normal\|2k` | Quality preset (default: `2k`) |
| `--imageSize 1K\|2K\|4K` | Image size for Google/OpenRouter (default: from quality) |
| `--imageApiDialect openai-native\|ratio-metadata` | OpenAI-compatible endpoint dialect — use `ratio-metadata` for gateways that expect aspect-ratio `size` plus `metadata.resolution` |
| `--ref <files...>` | Reference images. Supported by Google multimodal, OpenAI GPT Image edits, Azure OpenAI edits (PNG/JPG only), OpenRouter multimodal models, Replicate supported families, MiniMax subject-reference, Seedream 5.0/4.5/4.0. Not supported by Jimeng, Seedream 3.0, SeedEdit 3.0 |
| `--ref <files...>` | Reference images. Supported by Google multimodal, OpenAI GPT Image edits, Azure OpenAI edits (PNG/JPG only), OpenRouter multimodal models, Replicate supported families, MiniMax subject-reference, Seedream 5.0/4.5/4.0, DashScope `wan2.7-image-pro`/`wan2.7-image`. Not supported by Jimeng, Seedream 3.0, SeedEdit 3.0, or any DashScope model outside the `wan2.7-image*` family |
| `--n <count>` | Number of images. Replicate requires `--n 1` (single-output save semantics) |
| `--json` | JSON output |
@@ -128,7 +131,9 @@ Priority (highest → lowest) applies to every provider:
3. Env var `<PROVIDER>_IMAGE_MODEL`
4. Built-in default
For Azure, `--model` / `default_model.azure` is the Azure deployment name. `AZURE_OPENAI_DEPLOYMENT` is the preferred env var; `AZURE_OPENAI_IMAGE_MODEL` is kept as a backward-compatible alias.
For OpenAI, the built-in default is `gpt-image-2`. `gpt-image-1.5`, `gpt-image-1`, and GPT Image snapshots remain selectable with `--model` or `OPENAI_IMAGE_MODEL`.
For Azure, `--model` / `default_model.azure` is the Azure deployment name. `AZURE_OPENAI_DEPLOYMENT` is the preferred env var; `AZURE_OPENAI_IMAGE_MODEL` is kept as a backward-compatible alias. If your Azure deployment is named after the underlying model, use `gpt-image-2`; otherwise use the exact custom deployment name.
EXTEND.md overrides env vars: if EXTEND.md sets `default_model.google: "gemini-3-pro-image-preview"` and the env var sets `GOOGLE_IMAGE_MODEL=gemini-3.1-flash-image-preview`, EXTEND.md wins.
@@ -169,17 +174,19 @@ Each provider has its own quirks (model families, size rules, ref support, limit
| Preset | Google imageSize | OpenAI size | OpenRouter size | Replicate resolution | Use case |
|--------|------------------|-------------|-----------------|----------------------|----------|
| `normal` | 1K | 1024px | 1K | 1K | Quick previews |
| `2k` (default) | 2K | 2048px | 2K | 2K | Covers, illustrations, infographics |
| `normal` | 1K | 1024px target | 1K | 1K | Quick previews |
| `2k` (default) | 2K | 2048px target | 2K | 2K | Covers, illustrations, infographics |
Google/OpenRouter `imageSize` can be overridden with `--imageSize 1K|2K|4K`.
For OpenAI native `gpt-image-2`, `normal` maps to `quality=medium` and a low-latency valid size near the requested aspect ratio; `2k` maps to `quality=high` and 2048px-class sizes such as `2048x2048`, `2048x1152`, or `1152x2048`. Use explicit `--size` for valid custom or 4K outputs, e.g. `3840x2160`.
## Aspect Ratios
Supported: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `2.35:1`.
- Google multimodal: `imageConfig.aspectRatio`
- OpenAI: closest supported size
- OpenAI: `gpt-image-2` uses the closest valid custom size for the requested ratio; older GPT Image and DALL·E models use their closest supported fixed size
- OpenRouter: `imageGenerationOptions.aspect_ratio`; if only `--size <WxH>` is given, the ratio is inferred
- Replicate: behavior is model-specific — `google/nano-banana*` uses `aspect_ratio`, `bytedance/seedream-*` uses documented Replicate ratios, Wan 2.7 maps `--ar` to a concrete `size`
- MiniMax: official `aspect_ratio` values; if `--size <WxH>` is given without `--ar`, sends `width`/`height` for `image-01`
@@ -46,7 +46,7 @@ options:
- label: "Google (Recommended)"
description: "Gemini multimodal - high quality, reference images, flexible sizes"
- label: "OpenAI"
description: "GPT Image - consistent quality, reliable output"
description: "GPT Image 2 - latest OpenAI image model, reference-image workflows"
- label: "Azure OpenAI"
description: "Azure-hosted GPT Image deployments with resource-specific routing"
- label: "OpenRouter"
@@ -101,10 +101,12 @@ Only show if user selected Azure OpenAI.
header: "Azure Deploy"
question: "Default Azure image deployment name?"
options:
- label: "gpt-image-1.5 (Recommended)"
description: "Best default if your Azure deployment uses the same name"
- label: "gpt-image-1"
- label: "gpt-image-2 (Recommended)"
description: "Use if your Azure deployment uses the GPT Image 2 model name"
- label: "gpt-image-1.5"
description: "Previous GPT Image deployment name"
- label: "gpt-image-1"
description: "Earlier GPT Image deployment name"
```
### Question 2d: Default MiniMax Model
@@ -214,10 +216,12 @@ options:
header: "OpenAI Model"
question: "Choose a default OpenAI image generation model?"
options:
- label: "gpt-image-1.5 (Recommended)"
description: "Latest GPT Image model, high quality"
- label: "gpt-image-2 (Recommended)"
description: "Latest GPT Image model, flexible sizes up to 4K, high-fidelity image inputs"
- label: "gpt-image-1.5"
description: "Previous GPT Image model"
- label: "gpt-image-1"
description: "Previous generation GPT Image model"
description: "Earlier GPT Image model"
```
### Azure Deployment Selection
@@ -226,8 +230,10 @@ options:
header: "Azure Deploy"
question: "Choose a default Azure image deployment name?"
options:
- label: "gpt-image-1.5 (Recommended)"
description: "Use when your Azure deployment name matches the GPT-image-1.5 model"
- label: "gpt-image-2 (Recommended)"
description: "Use when your Azure deployment name matches the GPT Image 2 model"
- label: "gpt-image-1.5"
description: "Use when your Azure deployment name matches the GPT Image 1.5 model"
- label: "gpt-image-1"
description: "Use when your Azure deployment name matches GPT-image-1"
```
@@ -265,6 +271,10 @@ options:
description: "Legacy Qwen model with five fixed output sizes"
- label: "qwen-image-plus"
description: "Legacy Qwen model, same current capability as qwen-image"
- label: "wan2.7-image-pro"
description: "Wan 2.7 Pro — supports up to 4K text-to-image and reference-image editing"
- label: "wan2.7-image"
description: "Wan 2.7 base — faster generation, up to 2K, supports reference-image editing"
- label: "z-image-turbo"
description: "Legacy DashScope model for compatibility"
- label: "z-image-ultra"
@@ -275,6 +285,7 @@ Notes for DashScope setup:
- Prefer `qwen-image-2.0-pro` when the user needs custom `--size`, uncommon ratios like `21:9`, or strong Chinese/English text rendering.
- `qwen-image-max` / `qwen-image-plus` / `qwen-image` only support five fixed sizes: `1664*928`, `1472*1104`, `1328*1328`, `1104*1472`, `928*1664`.
- `wan2.7-image-pro` and `wan2.7-image` are the only DashScope models that accept `--ref`. Pick one of these when the user wants reference-image editing or multi-image fusion via DashScope.
- In `baoyu-imagine`, `quality` is a compatibility preset. It is not a native DashScope parameter.
### Z.AI Model Selection
@@ -23,8 +23,8 @@ default_image_api_dialect: null # openai-native|ratio-metadata|null (OpenAI-com
default_model:
google: null # e.g., "gemini-3-pro-image-preview", "gemini-3.1-flash-image-preview"
openai: null # e.g., "gpt-image-1.5", "gpt-image-1"
azure: null # Azure deployment name, e.g., "gpt-image-1.5" or "image-prod"
openai: null # e.g., "gpt-image-2", "gpt-image-1.5", "gpt-image-1"
azure: null # Azure deployment name, e.g., "gpt-image-2" or "image-prod"
openrouter: null # e.g., "google/gemini-3.1-flash-image-preview"
dashscope: null # e.g., "qwen-image-2.0-pro"
zai: null # e.g., "glm-image"
@@ -106,8 +106,8 @@ default_image_size: 2K
default_image_api_dialect: null
default_model:
google: "gemini-3-pro-image-preview"
openai: "gpt-image-1.5"
azure: "gpt-image-1.5"
openai: "gpt-image-2"
azure: "gpt-image-2"
openrouter: "google/gemini-3.1-flash-image-preview"
dashscope: "qwen-image-2.0-pro"
zai: "glm-image"
@@ -17,6 +17,17 @@ Read when the user picks `--provider dashscope`, sets `default_model.dashscope`,
- Default is `1664*928`
- `qwen-image` currently has the same capability as `qwen-image-plus`
**`wan2.7-image*`** — multimodal Wan 2.7 family. Members: `wan2.7-image-pro`, `wan2.7-image`.
- Free-form `size` in `宽*高` format, plus aspect-ratio inference
- `wan2.7-image-pro` text-to-image (no `--ref`): total pixels in `[768*768, 4096*4096]`, ratio in `[1:8, 8:1]`
- `wan2.7-image-pro` with reference images and `wan2.7-image` (all scenarios): total pixels in `[768*768, 2048*2048]`, ratio in `[1:8, 8:1]`
- Default: `1024*1024` (`--quality normal`) or `2048*2048` (`--quality 2k`); 4K requires explicit `--size`
- Supports up to 9 reference images in `--ref` (image editing / multi-image fusion)
- Reference images are sent inline as base64 (or passed through if the path is an `http(s)://` URL)
- API does NOT use `prompt_extend`; the skill omits it for this family
- The Wan 2.7 API defaults `n` to **4** in non-collage mode and bills per generated image. baoyu-imagine forces `n: 1` and rejects `--n > 1` to avoid silently paying for and discarding extra images.
**Legacy** — `z-image-turbo`, `z-image-ultra`, `wanx-v1`. Only use when the user explicitly asks for legacy behavior.
## Size Resolution
@@ -24,7 +35,8 @@ Read when the user picks `--provider dashscope`, sets `default_model.dashscope`,
- `--size` wins over `--ar`
- For `qwen-image-2.0*`: prefer explicit `--size`; otherwise infer from `--ar` using the recommended table below
- For `qwen-image-max/plus/image`: only use the five fixed sizes; if the requested ratio doesn't fit, switch to `qwen-image-2.0-pro`
- `--quality` is a baoyu-imagine preset, not an official DashScope field. The mapping of `normal`/`2k` onto the `qwen-image-2.0*` table is an implementation choice, not an API guarantee
- For `wan2.7-image*`: explicit `--size` is validated against the per-mode pixel/ratio limits; otherwise the size is derived from `--ar` and `--quality` (`normal` ≈ 1K, `2k` ≈ 2K). To request 4K with `wan2.7-image-pro` text-to-image, pass `--size` explicitly (e.g. `4096*4096`, `3840*2160`)
- `--quality` is a baoyu-imagine preset, not an official DashScope field. The mapping of `normal`/`2k` onto the `qwen-image-2.0*` and `wan2.7-image*` tables is an implementation choice, not an API guarantee
### Recommended `qwen-image-2.0*` sizes
@@ -39,12 +51,19 @@ Read when the user picks `--provider dashscope`, sets `default_model.dashscope`,
| `16:9` | `1280*720` | `1920*1080` |
| `21:9` | `1344*576` | `2048*872` |
## Reference Images
- Only `wan2.7-image-pro` and `wan2.7-image` accept `--ref`. Other DashScope models (qwen-image-2.0*, qwen-image-max/plus/image, legacy) reject `--ref` and the user is steered to a different provider/model.
- Up to 9 reference images per request. Local files are inlined as base64 data URLs; `http(s)://` URLs are forwarded as-is.
- Supplying any `--ref` automatically clamps the wan2.7-image-pro pixel ceiling from 4K to 2K (the API only supports 4K for pure text-to-image with no image input).
## Not Exposed
DashScope APIs also support `negative_prompt`, `prompt_extend`, and `watermark`, but `baoyu-imagine` does not expose them as CLI flags today.
DashScope APIs also support `negative_prompt`, `prompt_extend`, `watermark`, `thinking_mode`, `seed`, `bbox_list`, `enable_sequential`, and `color_palette`. `baoyu-imagine` does not expose them as CLI flags today; the wan2.7 family relies on the API defaults (e.g. `thinking_mode=true`). The skill always sends `n=1` for wan2.7 — if you want grid/collage mode you currently need to call the API directly.
## Official References
- [Qwen-Image API](https://help.aliyun.com/zh/model-studio/qwen-image-api)
- [Text-to-image guide](https://help.aliyun.com/zh/model-studio/text-to-image)
- [Qwen-Image Edit API](https://help.aliyun.com/zh/model-studio/qwen-image-edit-api)
- [Wan 2.7 image generation & editing API](https://help.aliyun.com/zh/model-studio/wan-image-generation-and-editing-api-reference)
@@ -25,10 +25,13 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "Make blue" --image out.png --ref so
```bash
# OpenAI
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider openai
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider openai --model gpt-image-2
# Azure OpenAI (model = deployment name)
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider azure --model gpt-image-1.5
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider azure --model gpt-image-2
# OpenAI GPT Image 2 custom 4K size
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cinematic landscape" --image out.png --provider openai --model gpt-image-2 --size 3840x2160
# Google with explicit model
${BUN_X} {baseDir}/scripts/main.ts --prompt "Make blue" --image out.png --provider google --model gemini-3-pro-image-preview --ref source.png
@@ -48,6 +51,12 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "为咖啡品牌设计一张 21:9
# DashScope legacy fixed-size
${BUN_X} {baseDir}/scripts/main.ts --prompt "一张电影感海报" --image out.png --provider dashscope --model qwen-image-max --size 1664x928
# DashScope Wan 2.7 Image Pro (4K text-to-image)
${BUN_X} {baseDir}/scripts/main.ts --prompt "一间有着精致窗户的花店" --image out.png --provider dashscope --model wan2.7-image-pro --size 4096x4096
# DashScope Wan 2.7 Image with reference image (multi-image fusion)
${BUN_X} {baseDir}/scripts/main.ts --prompt "把图2的涂鸦喷绘在图1的汽车上" --image out.png --provider dashscope --model wan2.7-image-pro --ref car.webp paint.webp
# Z.AI GLM-image
${BUN_X} {baseDir}/scripts/main.ts --prompt "一张带清晰中文标题的科技海报" --image out.png --provider zai
+66 -4
View File
@@ -19,6 +19,7 @@ import {
parseArgs,
parseOpenAIImageApiDialect,
parseSimpleYaml,
validateReferenceImages,
} from "./main.ts";
function makeArgs(overrides: Partial<CliArgs> = {}): CliArgs {
@@ -123,6 +124,15 @@ test("parseArgs falls back to positional prompt and rejects invalid provider", (
);
});
test("validateReferenceImages can skip remote URLs for providers that support them", async () => {
await validateReferenceImages(["https://example.com/ref.png"], { allowRemoteUrls: true });
await assert.rejects(
() => validateReferenceImages(["https://example.com/ref.png"]),
/Reference image not found/,
);
});
test("parseSimpleYaml parses nested defaults and provider limits", () => {
const yaml = `
version: 2
@@ -133,7 +143,7 @@ default_image_size: 2K
default_image_api_dialect: ratio-metadata
default_model:
google: gemini-3-pro-image-preview
openai: gpt-image-1.5
openai: gpt-image-2
zai: glm-image
azure: image-prod
minimax: image-01
@@ -165,7 +175,7 @@ batch:
assert.equal(config.default_image_size, "2K");
assert.equal(config.default_image_api_dialect, "ratio-metadata");
assert.equal(config.default_model?.google, "gemini-3-pro-image-preview");
assert.equal(config.default_model?.openai, "gpt-image-1.5");
assert.equal(config.default_model?.openai, "gpt-image-2");
assert.equal(config.default_model?.zai, "glm-image");
assert.equal(config.default_model?.azure, "image-prod");
assert.equal(config.default_model?.minimax, "image-01");
@@ -308,7 +318,7 @@ test("detectProvider rejects non-ref-capable providers and prefers Google first
() =>
detectProvider(
makeArgs({
provider: "dashscope",
provider: "zai",
referenceImages: ["ref.png"],
}),
),
@@ -426,6 +436,33 @@ test("detectProvider infers Seedream from model id and allows Seedream reference
);
});
test("detectProvider allows DashScope reference-image workflows when explicitly chosen for wan2.7 models", (t) => {
useEnv(t, {
GOOGLE_API_KEY: null,
OPENAI_API_KEY: null,
AZURE_OPENAI_API_KEY: null,
AZURE_OPENAI_BASE_URL: null,
OPENROUTER_API_KEY: null,
DASHSCOPE_API_KEY: "dashscope-key",
MINIMAX_API_KEY: null,
REPLICATE_API_TOKEN: null,
JIMENG_ACCESS_KEY_ID: null,
JIMENG_SECRET_ACCESS_KEY: null,
ARK_API_KEY: null,
});
assert.equal(
detectProvider(
makeArgs({
provider: "dashscope",
model: "wan2.7-image-pro",
referenceImages: ["ref.png"],
}),
),
"dashscope",
);
});
test("detectProvider selects MiniMax when only MiniMax credentials are configured or the model id matches", (t) => {
useEnv(t, {
GOOGLE_API_KEY: null,
@@ -504,7 +541,7 @@ test("loadBatchTasks and createTaskArgs resolve batch-relative paths", async (t)
id: "hero",
promptFiles: ["prompts/hero.md"],
image: "out/hero",
ref: ["refs/hero.png"],
ref: ["refs/hero.png", "https://example.com/ref.png"],
ar: "16:9",
},
],
@@ -533,6 +570,7 @@ test("loadBatchTasks and createTaskArgs resolve batch-relative paths", async (t)
assert.equal(taskArgs.imagePath, path.join(loaded.batchDir, "out/hero"));
assert.deepEqual(taskArgs.referenceImages, [
path.join(loaded.batchDir, "refs/hero.png"),
"https://example.com/ref.png",
]);
assert.equal(taskArgs.provider, "replicate");
assert.equal(taskArgs.aspectRatio, "16:9");
@@ -557,5 +595,29 @@ test("path normalization, worker count, and retry classification follow expected
),
false,
);
assert.equal(
isRetryableGenerationError(
new Error("DashScope wan2.7 image models accept at most 9 reference images. Received 10."),
),
false,
);
assert.equal(
isRetryableGenerationError(
new Error("DashScope wan2.7 image models in baoyu-imagine support exactly one output image per request."),
),
false,
);
assert.equal(
isRetryableGenerationError(
new Error("DashScope wan2.7 image models support aspect ratios in [1:8, 8:1]."),
),
false,
);
assert.equal(
isRetryableGenerationError(
new Error("DashScope wan2.7-image requires total pixels between 768*768 and 2048*2048."),
),
false,
);
assert.equal(isRetryableGenerationError(new Error("socket hang up")), true);
});
+43 -9
View File
@@ -85,7 +85,7 @@ Options:
--quality normal|2k Quality preset (default: 2k)
--imageSize 1K|2K|4K Image size for Google/OpenRouter (default: from quality)
--imageApiDialect <id> OpenAI-compatible image dialect: openai-native|ratio-metadata
--ref <files...> Reference images (Google, OpenAI, Azure, OpenRouter, Replicate supported families, MiniMax, or Seedream 4.0/4.5/5.0)
--ref <files...> Reference images (Google, OpenAI, Azure, OpenRouter, Replicate supported families, MiniMax, Seedream 4.0/4.5/5.0, or DashScope wan2.7-image*)
--n <count> Number of images for the current task (default: 1; Replicate currently requires 1)
--json JSON output
-h, --help Show help
@@ -124,7 +124,7 @@ Environment variables:
JIMENG_ACCESS_KEY_ID Jimeng Access Key ID
JIMENG_SECRET_ACCESS_KEY Jimeng Secret Access Key
ARK_API_KEY Seedream/Ark API key
OPENAI_IMAGE_MODEL Default OpenAI model (gpt-image-1.5)
OPENAI_IMAGE_MODEL Default OpenAI model (gpt-image-2)
OPENROUTER_IMAGE_MODEL Default OpenRouter model (google/gemini-3.1-flash-image-preview)
GOOGLE_IMAGE_MODEL Default Google model (gemini-3-pro-image-preview)
DASHSCOPE_IMAGE_MODEL Default DashScope model (qwen-image-2.0-pro)
@@ -151,7 +151,7 @@ Environment variables:
AZURE_OPENAI_BASE_URL Azure OpenAI resource or deployment endpoint
AZURE_OPENAI_DEPLOYMENT Default Azure deployment name
AZURE_API_VERSION Azure API version (default: 2025-04-01-preview)
AZURE_OPENAI_IMAGE_MODEL Backward-compatible Azure deployment/model alias (defaults to gpt-image-1.5)
AZURE_OPENAI_IMAGE_MODEL Backward-compatible Azure deployment/model alias (defaults to gpt-image-2)
SEEDREAM_BASE_URL Custom Seedream endpoint
BAOYU_IMAGE_GEN_MAX_WORKERS Override batch worker cap
BAOYU_IMAGE_GEN_<PROVIDER>_CONCURRENCY Override provider concurrency
@@ -698,10 +698,11 @@ export function detectProvider(args: CliArgs): Provider {
args.provider !== "openrouter" &&
args.provider !== "replicate" &&
args.provider !== "seedream" &&
args.provider !== "minimax"
args.provider !== "minimax" &&
args.provider !== "dashscope"
) {
throw new Error(
"Reference images require a ref-capable provider. Use --provider google (Gemini multimodal), --provider openai (GPT Image edits), --provider azure (Azure OpenAI), --provider openrouter (OpenRouter multimodal), --provider replicate, --provider seedream for supported Seedream models, or --provider minimax for MiniMax subject-reference workflows."
"Reference images require a ref-capable provider. Use --provider google (Gemini multimodal), --provider openai (GPT Image edits), --provider azure (Azure OpenAI), --provider openrouter (OpenRouter multimodal), --provider replicate, --provider dashscope with a wan2.7 image model, --provider seedream for supported Seedream models, or --provider minimax for MiniMax subject-reference workflows."
);
}
@@ -775,8 +776,24 @@ export function detectProvider(args: CliArgs): Provider {
);
}
export async function validateReferenceImages(referenceImages: string[]): Promise<void> {
export type ReferenceImageValidationOptions = {
allowRemoteUrls?: boolean;
};
function isRemoteReferenceImage(refPath: string): boolean {
return /^https?:\/\//i.test(refPath);
}
function shouldAllowRemoteReferenceImages(provider: Provider | null): boolean {
return provider === "dashscope";
}
export async function validateReferenceImages(
referenceImages: string[],
options: ReferenceImageValidationOptions = {},
): Promise<void> {
for (const refPath of referenceImages) {
if (options.allowRemoteUrls && isRemoteReferenceImage(refPath)) continue;
const fullPath = path.resolve(refPath);
try {
await access(fullPath);
@@ -803,6 +820,11 @@ export function isRetryableGenerationError(error: unknown): boolean {
"API error (404)",
"temporarily disabled",
"supports saving exactly one image",
"supports only",
"support exactly one output image",
"support aspect ratios in",
"requires total pixels between",
"accept at most",
];
return !nonRetryableMarkers.some((marker) => msg.includes(marker));
}
@@ -858,7 +880,11 @@ async function prepareSingleTask(args: CliArgs, extendConfig: Partial<ExtendConf
const prompt = (await loadPromptForArgs(args)) ?? (await readPromptFromStdin());
if (!prompt) throw new Error("Prompt is required");
if (!args.imagePath) throw new Error("--image is required");
if (args.referenceImages.length > 0) await validateReferenceImages(args.referenceImages);
if (args.referenceImages.length > 0) {
await validateReferenceImages(args.referenceImages, {
allowRemoteUrls: shouldAllowRemoteReferenceImages(args.provider),
});
}
const provider = detectProvider(args);
const providerModule = await loadProviderModule(provider);
@@ -907,6 +933,10 @@ export function resolveBatchPath(batchDir: string, filePath: string): string {
return path.isAbsolute(filePath) ? filePath : path.resolve(batchDir, filePath);
}
function resolveBatchReferencePath(batchDir: string, filePath: string): string {
return isRemoteReferenceImage(filePath) ? filePath : resolveBatchPath(batchDir, filePath);
}
export function createTaskArgs(baseArgs: CliArgs, task: BatchTaskInput, batchDir: string): CliArgs {
return {
...baseArgs,
@@ -922,7 +952,7 @@ export function createTaskArgs(baseArgs: CliArgs, task: BatchTaskInput, batchDir
imageSize: task.imageSize ?? baseArgs.imageSize ?? null,
imageSizeSource: task.imageSize != null ? "task" : (baseArgs.imageSizeSource ?? null),
imageApiDialect: task.imageApiDialect ?? baseArgs.imageApiDialect ?? null,
referenceImages: task.ref ? task.ref.map((filePath) => resolveBatchPath(batchDir, filePath)) : [],
referenceImages: task.ref ? task.ref.map((filePath) => resolveBatchReferencePath(batchDir, filePath)) : [],
n: task.n ?? baseArgs.n,
batchFile: null,
jobs: baseArgs.jobs,
@@ -946,7 +976,11 @@ async function prepareBatchTasks(
const prompt = await loadPromptForArgs(taskArgs);
if (!prompt) throw new Error(`Task ${i + 1} is missing prompt or promptFiles.`);
if (!taskArgs.imagePath) throw new Error(`Task ${i + 1} is missing image output path.`);
if (taskArgs.referenceImages.length > 0) await validateReferenceImages(taskArgs.referenceImages);
if (taskArgs.referenceImages.length > 0) {
await validateReferenceImages(taskArgs.referenceImages, {
allowRemoteUrls: shouldAllowRemoteReferenceImages(taskArgs.provider),
});
}
const provider = detectProvider(taskArgs);
const providerModule = await loadProviderModule(provider);
@@ -46,7 +46,7 @@ export function getDefaultModel(): string {
}
}
return process.env.AZURE_OPENAI_IMAGE_MODEL || "gpt-image-1.5";
return process.env.AZURE_OPENAI_IMAGE_MODEL || "gpt-image-2";
}
function getEndpoint(): AzureEndpoint {
@@ -2,15 +2,42 @@ import assert from "node:assert/strict";
import test, { type TestContext } from "node:test";
import {
generateImage,
getDefaultModel,
getModelFamily,
getQwen2SizeFromAspectRatio,
getSizeFromAspectRatio,
getWan27SizeFromAspectRatio,
normalizeSize,
parseAspectRatio,
parseSize,
resolveSizeForModel,
} from "./dashscope.ts";
import type { CliArgs } from "../types.ts";
function makeCliArgs(overrides: Partial<CliArgs> = {}): CliArgs {
return {
prompt: null,
promptFiles: [],
imagePath: null,
provider: "dashscope",
model: null,
aspectRatio: null,
aspectRatioSource: null,
size: null,
quality: "2k",
imageSize: null,
imageSizeSource: null,
imageApiDialect: null,
referenceImages: [],
n: 1,
batchFile: null,
jobs: null,
json: false,
help: false,
...overrides,
};
}
function useEnv(
t: TestContext,
@@ -51,9 +78,11 @@ test("DashScope aspect-ratio parsing accepts numeric ratios only", () => {
assert.equal(parseAspectRatio("-1:2"), null);
});
test("DashScope model family routing distinguishes qwen-2.0, fixed-size qwen, and legacy models", () => {
test("DashScope model family routing distinguishes qwen-2.0, fixed-size qwen, wan2.7, and legacy models", () => {
assert.equal(getModelFamily("qwen-image-2.0-pro"), "qwen2");
assert.equal(getModelFamily("qwen-image"), "qwenFixed");
assert.equal(getModelFamily("wan2.7-image"), "wan27");
assert.equal(getModelFamily("wan2.7-image-pro"), "wan27");
assert.equal(getModelFamily("z-image-turbo"), "legacy");
assert.equal(getModelFamily("wanx-v1"), "legacy");
});
@@ -146,3 +175,218 @@ test("DashScope size normalization converts WxH into provider format", () => {
assert.equal(normalizeSize("1024x1024"), "1024*1024");
assert.equal(normalizeSize("2048*1152"), "2048*1152");
});
test("Wan 2.7 derives sizes that match the requested ratio at the chosen pixel budget", () => {
const square2k = getWan27SizeFromAspectRatio(null, "2k", 2048 * 2048);
const parsedSquare = parseSize(square2k);
assert.ok(parsedSquare);
assert.equal(parsedSquare.width, parsedSquare.height);
assert.ok(parsedSquare.width * parsedSquare.height <= 2048 * 2048);
const widescreen = getWan27SizeFromAspectRatio("16:9", "2k", 2048 * 2048);
const parsedWide = parseSize(widescreen);
assert.ok(parsedWide);
assert.ok(Math.abs(parsedWide.width / parsedWide.height - 16 / 9) < 0.05);
assert.ok(parsedWide.width * parsedWide.height <= 2048 * 2048);
const pro4k = getWan27SizeFromAspectRatio("16:9", "2k", 4096 * 4096);
const parsed4k = parseSize(pro4k);
assert.ok(parsed4k);
assert.ok(parsed4k.width * parsed4k.height > 2048 * 2048);
assert.ok(parsed4k.width * parsed4k.height <= 4096 * 4096);
});
test("Wan 2.7 rejects aspect ratios outside the [1:8, 8:1] range", () => {
assert.throws(
() => getWan27SizeFromAspectRatio("9:1", "2k", 2048 * 2048),
/1:8, 8:1/,
);
assert.throws(
() => getWan27SizeFromAspectRatio("1:9", "normal", 2048 * 2048),
/1:8, 8:1/,
);
});
test("Wan 2.7 derived sizes stay inside the boundary ratio limits after rounding", () => {
for (const ar of ["8:1", "1:8"]) {
const size = getWan27SizeFromAspectRatio(ar, "2k", 2048 * 2048);
const parsed = parseSize(size);
assert.ok(parsed);
const ratio = parsed.width / parsed.height;
assert.ok(ratio >= 1 / 8);
assert.ok(ratio <= 8);
assert.ok(parsed.width * parsed.height <= 2048 * 2048);
}
});
test("resolveSizeForModel routes wan2.7-image to the 2K-capped derivation", () => {
const size = resolveSizeForModel("wan2.7-image", {
size: null,
aspectRatio: "16:9",
quality: "2k",
});
const parsed = parseSize(size);
assert.ok(parsed);
assert.ok(parsed.width * parsed.height <= 2048 * 2048);
assert.ok(Math.abs(parsed.width / parsed.height - 16 / 9) < 0.05);
});
test("resolveSizeForModel allows wan2.7-image-pro 4K only when there are no reference images", () => {
assert.equal(
resolveSizeForModel("wan2.7-image-pro", {
size: "4096*4096",
aspectRatio: null,
quality: "2k",
}),
"4096*4096",
);
assert.throws(
() =>
resolveSizeForModel("wan2.7-image-pro", {
size: "4096*4096",
aspectRatio: null,
quality: "2k",
referenceImages: ["a.png"],
}),
/total pixels between 768\*768 and 2048\*2048/,
);
const proWithRef = resolveSizeForModel("wan2.7-image-pro", {
size: null,
aspectRatio: "1:1",
quality: "2k",
referenceImages: ["a.png"],
});
const parsedRef = parseSize(proWithRef);
assert.ok(parsedRef);
assert.ok(parsedRef.width * parsedRef.height <= 2048 * 2048);
});
test("Wan 2.7 request body forces n=1 and omits prompt_extend / negative_prompt", async (t) => {
useEnv(t, { DASHSCOPE_API_KEY: "fake-key" });
const originalFetch = globalThis.fetch;
let capturedBody: any = null;
globalThis.fetch = (async (_url: string, init?: RequestInit) => {
capturedBody = JSON.parse(String(init?.body));
return new Response(
JSON.stringify({
output: {
choices: [
{
message: {
content: [{ image: "data:image/png;base64,iVBORw0KGgo=" }],
},
},
],
},
}),
{ status: 200, headers: { "content-type": "application/json" } },
);
}) as typeof fetch;
t.after(() => {
globalThis.fetch = originalFetch;
});
await generateImage("hello", "wan2.7-image-pro", makeCliArgs({ aspectRatio: "1:1" }));
assert.equal(capturedBody.model, "wan2.7-image-pro");
assert.deepEqual(Object.keys(capturedBody.parameters).sort(), ["n", "size", "watermark"]);
assert.equal(capturedBody.parameters.n, 1);
assert.equal(capturedBody.parameters.watermark, false);
assert.equal(typeof capturedBody.parameters.size, "string");
assert.ok(!("prompt_extend" in capturedBody.parameters));
assert.ok(!("negative_prompt" in capturedBody.parameters));
assert.deepEqual(capturedBody.input.messages[0].content, [{ text: "hello" }]);
});
test("Wan 2.7 request body forwards remote reference image URLs", async (t) => {
useEnv(t, { DASHSCOPE_API_KEY: "fake-key" });
const originalFetch = globalThis.fetch;
let capturedBody: any = null;
globalThis.fetch = (async (_url: string, init?: RequestInit) => {
capturedBody = JSON.parse(String(init?.body));
return new Response(
JSON.stringify({
output: {
choices: [
{
message: {
content: [{ image: "data:image/png;base64,iVBORw0KGgo=" }],
},
},
],
},
}),
{ status: 200, headers: { "content-type": "application/json" } },
);
}) as typeof fetch;
t.after(() => {
globalThis.fetch = originalFetch;
});
await generateImage(
"combine these",
"wan2.7-image-pro",
makeCliArgs({ referenceImages: ["https://example.com/ref.png"] }),
);
assert.deepEqual(capturedBody.input.messages[0].content, [
{ image: "https://example.com/ref.png" },
{ text: "combine these" },
]);
});
test("Wan 2.7 rejects --n > 1 to prevent silent multi-image billing", async (t) => {
useEnv(t, { DASHSCOPE_API_KEY: "fake-key" });
await assert.rejects(
() => generateImage("hi", "wan2.7-image-pro", makeCliArgs({ n: 2 })),
/support exactly one output image/,
);
});
test("resolveSizeForModel validates explicit wan2.7 sizes by pixel budget and ratio", () => {
assert.equal(
resolveSizeForModel("wan2.7-image-pro", {
size: "3840x2160",
aspectRatio: null,
quality: "2k",
}),
"3840*2160",
);
assert.throws(
() =>
resolveSizeForModel("wan2.7-image-pro", {
size: "3840x2160",
aspectRatio: null,
quality: "2k",
referenceImages: ["a.png"],
}),
/total pixels between 768\*768 and 2048\*2048/,
);
assert.throws(
() =>
resolveSizeForModel("wan2.7-image", {
size: "4096x4096",
aspectRatio: null,
quality: "2k",
}),
/total pixels between 768\*768 and 2048\*2048/,
);
assert.throws(
() =>
resolveSizeForModel("wan2.7-image-pro", {
size: "3072*256",
aspectRatio: null,
quality: "2k",
}),
/1:8, 8:1/,
);
});
@@ -1,6 +1,8 @@
import path from "node:path";
import { readFile } from "node:fs/promises";
import type { CliArgs, Quality } from "../types";
type DashScopeModelFamily = "qwen2" | "qwenFixed" | "legacy";
type DashScopeModelFamily = "qwen2" | "qwenFixed" | "wan27" | "legacy";
type DashScopeModelSpec = {
family: DashScopeModelFamily;
@@ -19,6 +21,16 @@ const QWEN_2_TARGET_PIXELS: Record<Quality, number> = {
"2k": 1536 * 1536,
};
const MIN_WAN27_TOTAL_PIXELS = 768 * 768;
const MAX_WAN27_PRO_T2I_PIXELS = 4096 * 4096;
const MAX_WAN27_GENERAL_PIXELS = 2048 * 2048;
const WAN27_MAX_REFERENCE_IMAGES = 9;
const WAN27_TARGET_PIXELS: Record<Quality, number> = {
normal: 1024 * 1024,
"2k": 2048 * 2048,
};
const QWEN_2_RECOMMENDED: Record<string, Record<Quality, string>> = {
"1:1": { normal: "1024*1024", "2k": "1536*1536" },
"2:3": { normal: "768*1152", "2k": "1024*1536" },
@@ -73,6 +85,11 @@ const QWEN_FIXED_SPEC: DashScopeModelSpec = {
defaultSize: QWEN_FIXED_SIZES_BY_RATIO["16:9"],
};
const WAN27_SPEC: DashScopeModelSpec = {
family: "wan27",
defaultSize: "2048*2048",
};
const LEGACY_SPEC: DashScopeModelSpec = {
family: "legacy",
defaultSize: "1536*1536",
@@ -88,12 +105,31 @@ const MODEL_SPEC_ALIASES: Record<string, DashScopeModelSpec> = {
"qwen-image-plus": QWEN_FIXED_SPEC,
"qwen-image-plus-2026-01-09": QWEN_FIXED_SPEC,
"qwen-image": QWEN_FIXED_SPEC,
"wan2.7-image-pro": WAN27_SPEC,
"wan2.7-image": WAN27_SPEC,
};
export function getDefaultModel(): string {
return process.env.DASHSCOPE_IMAGE_MODEL || DEFAULT_MODEL;
}
function getReferenceImageMime(filePath: string): string {
const ext = path.extname(filePath).toLowerCase();
if (ext === ".jpg" || ext === ".jpeg") return "image/jpeg";
if (ext === ".webp") return "image/webp";
if (ext === ".bmp") return "image/bmp";
return "image/png";
}
async function loadReferenceImage(refPath: string): Promise<string> {
if (/^https?:\/\//i.test(refPath)) {
return refPath;
}
const fullPath = path.resolve(refPath);
const bytes = await readFile(fullPath);
return `data:${getReferenceImageMime(fullPath)};base64,${bytes.toString("base64")}`;
}
function getApiKey(): string | null {
return process.env.DASHSCOPE_API_KEY || null;
}
@@ -173,6 +209,10 @@ function roundToStep(value: number): number {
return Math.max(SIZE_STEP, Math.round(value / SIZE_STEP) * SIZE_STEP);
}
function floorToStep(value: number): number {
return Math.max(SIZE_STEP, Math.floor(value / SIZE_STEP) * SIZE_STEP);
}
function fitToPixelBudget(
width: number,
height: number,
@@ -220,6 +260,21 @@ function fitToPixelBudget(
return { width: roundedWidth, height: roundedHeight };
}
function clampWan27DerivedSizeToRatioBounds(
size: { width: number; height: number },
): { width: number; height: number } {
let { width, height } = size;
const ratio = width / height;
if (ratio > 8) {
width = floorToStep(height * 8);
} else if (ratio < 1 / 8) {
height = floorToStep(width * 8);
}
return { width, height };
}
export function getSizeFromAspectRatio(ar: string | null, quality: CliArgs["quality"]): string {
const normalizedQuality = normalizeQuality(quality);
const sizes = normalizedQuality === "2k" ? LEGACY_STANDARD_SIZES_2K : LEGACY_STANDARD_SIZES;
@@ -276,6 +331,77 @@ export function getQwen2SizeFromAspectRatio(ar: string | null, quality: CliArgs[
return formatSize(fitted.width, fitted.height);
}
function isWan27ProModel(model: string): boolean {
return model.trim().toLowerCase() === "wan2.7-image-pro";
}
function getWan27MaxPixels(model: string, hasReferenceImages: boolean): number {
if (isWan27ProModel(model) && !hasReferenceImages) {
return MAX_WAN27_PRO_T2I_PIXELS;
}
return MAX_WAN27_GENERAL_PIXELS;
}
export function getWan27SizeFromAspectRatio(
ar: string | null,
quality: CliArgs["quality"],
maxPixels: number,
): string {
const normalizedQuality = normalizeQuality(quality);
const targetPixels = Math.min(WAN27_TARGET_PIXELS[normalizedQuality], maxPixels);
if (!ar) {
const side = roundToStep(Math.sqrt(targetPixels));
return formatSize(side, side);
}
const parsed = parseAspectRatio(ar);
if (!parsed) {
const side = roundToStep(Math.sqrt(targetPixels));
return formatSize(side, side);
}
const ratio = parsed.width / parsed.height;
if (ratio < 1 / 8 || ratio > 8) {
throw new Error(
`DashScope wan2.7 image models support aspect ratios in [1:8, 8:1]. Received "${ar}".`
);
}
const rawWidth = Math.sqrt(targetPixels * ratio);
const rawHeight = Math.sqrt(targetPixels / ratio);
const fitted = fitToPixelBudget(
rawWidth,
rawHeight,
MIN_WAN27_TOTAL_PIXELS,
maxPixels,
);
const bounded = clampWan27DerivedSizeToRatioBounds(fitted);
return formatSize(bounded.width, bounded.height);
}
function validateWan27Size(size: string, maxPixels: number, model: string): string {
const normalized = normalizeSize(size);
const parsed = validateSizeFormat(normalized);
const totalPixels = parsed.width * parsed.height;
if (totalPixels < MIN_WAN27_TOTAL_PIXELS || totalPixels > maxPixels) {
const limit = maxPixels === MAX_WAN27_PRO_T2I_PIXELS ? "4096*4096" : "2048*2048";
throw new Error(
`DashScope ${model} requires total pixels between 768*768 and ${limit} ` +
`for the current request. Received ${normalized} (${totalPixels} pixels).`
);
}
const ratio = parsed.width / parsed.height;
if (ratio < 1 / 8 || ratio > 8) {
throw new Error(
`DashScope wan2.7 image models support aspect ratios in [1:8, 8:1]. ` +
`Received ${normalized} (ratio ${ratio.toFixed(3)}).`
);
}
return normalized;
}
function getQwenFixedSizeFromAspectRatio(ar: string | null, quality: CliArgs["quality"]): string {
if (quality === "normal") {
console.warn(
@@ -331,9 +457,16 @@ function validateQwenFixedSize(size: string): string {
export function resolveSizeForModel(
model: string,
args: Pick<CliArgs, "size" | "aspectRatio" | "quality">,
args: Pick<CliArgs, "size" | "aspectRatio" | "quality"> & { referenceImages?: string[] },
): string {
const spec = getModelSpec(model);
const referenceCount = args.referenceImages?.length ?? 0;
if (spec.family === "wan27") {
const maxPixels = getWan27MaxPixels(model, referenceCount > 0);
if (args.size) return validateWan27Size(args.size, maxPixels, model);
return getWan27SizeFromAspectRatio(args.aspectRatio, args.quality, maxPixels);
}
if (args.size) {
if (spec.family === "qwen2") return validateQwen2Size(args.size);
@@ -357,6 +490,14 @@ function buildParameters(
family: DashScopeModelFamily,
size: string,
): Record<string, unknown> {
if (family === "wan27") {
return {
size,
n: 1,
watermark: false,
};
}
const parameters: Record<string, unknown> = {
prompt_extend: false,
size,
@@ -419,23 +560,44 @@ export async function generateImage(
const apiKey = getApiKey();
if (!apiKey) throw new Error("DASHSCOPE_API_KEY is required");
if (args.referenceImages.length > 0) {
const spec = getModelSpec(model);
if (args.referenceImages.length > 0 && spec.family !== "wan27") {
throw new Error(
"Reference images are not supported with DashScope provider in baoyu-imagine. Use --provider google with a Gemini multimodal model."
"Reference images are not supported with this DashScope model. Use a wan2.7 image model (--model wan2.7-image-pro or wan2.7-image), or switch to --provider google with a Gemini multimodal model."
);
}
if (args.referenceImages.length > WAN27_MAX_REFERENCE_IMAGES) {
throw new Error(
`DashScope wan2.7 image models accept at most ${WAN27_MAX_REFERENCE_IMAGES} reference images. Received ${args.referenceImages.length}.`
);
}
if (spec.family === "wan27" && args.n !== 1) {
throw new Error(
"DashScope wan2.7 image models in baoyu-imagine support exactly one output image per request (extra images would be billed but discarded). Remove --n or use --n 1."
);
}
const spec = getModelSpec(model);
const size = resolveSizeForModel(model, args);
const url = `${getBaseUrl()}/api/v1/services/aigc/multimodal-generation/generation`;
const content: Array<Record<string, unknown>> = [];
if (spec.family === "wan27" && args.referenceImages.length > 0) {
for (const refPath of args.referenceImages) {
content.push({ image: await loadReferenceImage(refPath) });
}
}
content.push({ text: prompt });
const body = {
model,
input: {
messages: [
{
role: "user",
content: [{ text: prompt }],
content,
},
],
},
@@ -1,9 +1,11 @@
import assert from "node:assert/strict";
import test from "node:test";
import type { CliArgs } from "../types.ts";
import {
buildOpenAIGenerationsBody,
extractImageFromResponse,
getDefaultModel,
getOpenAIAspectRatio,
getOpenAIImageApiDialect,
getOpenAIResolution,
@@ -13,9 +15,33 @@ import {
inferAspectRatioFromSize,
inferResolutionFromSize,
parseAspectRatio,
validateArgs,
} from "./openai.ts";
function makeArgs(overrides: Partial<CliArgs> = {}): CliArgs {
return {
prompt: null,
promptFiles: [],
imagePath: null,
provider: null,
model: null,
aspectRatio: null,
size: null,
quality: "2k",
imageSize: null,
imageApiDialect: null,
referenceImages: [],
n: 1,
batchFile: null,
jobs: null,
json: false,
help: false,
...overrides,
};
}
test("OpenAI aspect-ratio parsing and size selection match model families", () => {
assert.equal(getDefaultModel(), "gpt-image-2");
assert.deepEqual(parseAspectRatio("16:9"), { width: 16, height: 9 });
assert.equal(parseAspectRatio("wide"), null);
assert.equal(parseAspectRatio("0:1"), null);
@@ -25,6 +51,10 @@ test("OpenAI aspect-ratio parsing and size selection match model families", () =
assert.equal(getOpenAISize("dall-e-2", "16:9", "2k"), "1024x1024");
assert.equal(getOpenAISize("gpt-image-1.5", "16:9", "2k"), "1536x1024");
assert.equal(getOpenAISize("gpt-image-1.5", "4:3", "2k"), "1024x1024");
assert.equal(getOpenAISize("gpt-image-2", "16:9", "2k"), "2048x1152");
assert.equal(getOpenAISize("gpt-image-2", "9:16", "2k"), "1152x2048");
assert.equal(getOpenAISize("gpt-image-2", "4:3", "2k"), "2048x1536");
assert.equal(getOpenAISize("gpt-image-2", "2.35:1", "normal"), "1248x528");
assert.equal(inferAspectRatioFromSize("1536x1024"), "3:2");
assert.equal(inferResolutionFromSize("1536x1024"), "2K");
assert.equal(getOpenAIAspectRatio({ aspectRatio: null, size: "2048x1152" }), "16:9");
@@ -37,7 +67,7 @@ test("OpenAI aspect-ratio parsing and size selection match model families", () =
test("OpenAI generations body switches between native and ratio-metadata dialects", () => {
assert.deepEqual(
buildOpenAIGenerationsBody("Draw a skyline", "gpt-image-1.5", {
buildOpenAIGenerationsBody("Draw a skyline", "gpt-image-2", {
aspectRatio: "16:9",
size: null,
quality: "2k",
@@ -45,9 +75,10 @@ test("OpenAI generations body switches between native and ratio-metadata dialect
imageApiDialect: null,
}),
{
model: "gpt-image-1.5",
model: "gpt-image-2",
prompt: "Draw a skyline",
size: "1536x1024",
size: "2048x1152",
quality: "high",
},
);
@@ -90,6 +121,28 @@ test("OpenAI generations body switches between native and ratio-metadata dialect
);
});
test("OpenAI validates gpt-image-2 custom size constraints", () => {
assert.doesNotThrow(() =>
validateArgs("gpt-image-2", makeArgs({ size: "3840x2160" })),
);
assert.doesNotThrow(() =>
validateArgs("gpt-image-2-2026-04-21", makeArgs({ aspectRatio: "2.35:1" })),
);
assert.throws(
() => validateArgs("gpt-image-2", makeArgs({ size: "1024x576" })),
/total pixels/,
);
assert.throws(
() => validateArgs("gpt-image-2", makeArgs({ size: "1025x1024" })),
/multiples of 16px/,
);
assert.throws(
() => validateArgs("gpt-image-2", makeArgs({ aspectRatio: "4:1" })),
/must not exceed 3:1/,
);
});
test("OpenAI mime-type detection covers supported reference image extensions", () => {
assert.equal(getMimeType("frame.png"), "image/png");
assert.equal(getMimeType("frame.jpg"), "image/jpeg");
@@ -3,7 +3,7 @@ import { readFile } from "node:fs/promises";
import type { CliArgs, OpenAIImageApiDialect } from "../types";
export function getDefaultModel(): string {
return process.env.OPENAI_IMAGE_MODEL || "gpt-image-1.5";
return process.env.OPENAI_IMAGE_MODEL || "gpt-image-2";
}
type OpenAIImageResponse = { data: Array<{ url?: string; b64_json?: string }> };
@@ -25,6 +25,55 @@ type SizeMapping = {
type OpenAIGenerationsBody = Record<string, unknown>;
function isGptImageModel(model: string): boolean {
return model.includes("gpt-image");
}
function isGptImage2Model(model: string): boolean {
return model.includes("gpt-image-2");
}
function roundToMultiple(value: number, multiple: number): number {
return Math.max(multiple, Math.round(value / multiple) * multiple);
}
function buildGptImage2SizeFromAspectRatio(
ar: string | null,
quality: CliArgs["quality"],
): string {
const parsed = ar ? parseAspectRatio(ar) : null;
const ratio = parsed ? parsed.width / parsed.height : 1;
if (!parsed || Math.abs(ratio - 1) < 0.1) {
const edge = quality === "2k" ? 2048 : 1024;
return `${edge}x${edge}`;
}
const targetLongEdge = quality === "2k" ? 2048 : 1024;
let width: number;
let height: number;
if (ratio > 1) {
width = targetLongEdge;
height = roundToMultiple(width / ratio, 16);
} else {
height = targetLongEdge;
width = roundToMultiple(height * ratio, 16);
}
while (width * height < 655_360) {
if (ratio > 1) {
width += 16;
height = roundToMultiple(width / ratio, 16);
} else {
height += 16;
width = roundToMultiple(height * ratio, 16);
}
}
return `${width}x${height}`;
}
export function getOpenAISize(
model: string,
ar: string | null,
@@ -37,6 +86,10 @@ export function getOpenAISize(
return "1024x1024";
}
if (isGptImage2Model(model)) {
return buildGptImage2SizeFromAspectRatio(ar, quality);
}
const sizes: SizeMapping = isDalle3
? {
square: "1024x1024",
@@ -127,6 +180,18 @@ export function getOpenAIResolution(
return args.quality === "normal" ? "1K" : "2K";
}
function getOpenAIQuality(model: string, quality: CliArgs["quality"]): "standard" | "hd" | "medium" | "high" | null {
if (model.includes("dall-e-3")) {
return quality === "2k" ? "hd" : "standard";
}
if (isGptImageModel(model)) {
return quality === "2k" ? "high" : "medium";
}
return null;
}
export function getOrientationFromAspectRatio(ar: string): "landscape" | "portrait" | null {
const parsed = parseAspectRatio(ar);
if (!parsed) return null;
@@ -163,13 +228,53 @@ export function buildOpenAIGenerationsBody(
size: args.size || getOpenAISize(model, args.aspectRatio, args.quality),
};
if (model.includes("dall-e-3")) {
body.quality = args.quality === "2k" ? "hd" : "standard";
const quality = getOpenAIQuality(model, args.quality);
if (quality) {
body.quality = quality;
}
return body;
}
export function validateArgs(model: string, args: CliArgs): void {
if (!isGptImage2Model(model)) return;
if (args.aspectRatio && !args.size) {
const parsed = parseAspectRatio(args.aspectRatio);
if (!parsed) {
throw new Error(`Invalid gpt-image-2 aspect ratio: ${args.aspectRatio}`);
}
const ratio = parsed.width / parsed.height;
if (Math.max(ratio, 1 / ratio) > 3) {
throw new Error("gpt-image-2 aspect ratio must not exceed 3:1.");
}
}
if (!args.size) return;
const parsedSize = parsePixelSize(args.size);
if (!parsedSize) {
throw new Error(`Invalid gpt-image-2 --size: ${args.size}. Expected <width>x<height>.`);
}
const { width, height } = parsedSize;
const totalPixels = width * height;
const ratio = Math.max(width, height) / Math.min(width, height);
if (Math.max(width, height) > 3840) {
throw new Error("gpt-image-2 --size maximum edge length must be 3840px or less.");
}
if (width % 16 !== 0 || height % 16 !== 0) {
throw new Error("gpt-image-2 --size width and height must both be multiples of 16px.");
}
if (ratio > 3) {
throw new Error("gpt-image-2 --size long edge to short edge ratio must not exceed 3:1.");
}
if (totalPixels < 655_360 || totalPixels > 8_294_400) {
throw new Error("gpt-image-2 --size total pixels must be between 655,360 and 8,294,400.");
}
}
export async function generateImage(
prompt: string,
model: string,
@@ -198,7 +303,7 @@ export async function generateImage(
}
if (model.includes("dall-e-2") || model.includes("dall-e-3")) {
throw new Error(
"Reference images with OpenAI in this skill require GPT Image models. Use --model gpt-image-1.5 (or another gpt-image model)."
"Reference images with OpenAI in this skill require GPT Image models. Use --model gpt-image-2 (or another gpt-image model)."
);
}
const size = args.size || getOpenAISize(model, args.aspectRatio, args.quality);
@@ -283,8 +388,9 @@ async function generateWithOpenAIEdits(
form.append("prompt", prompt);
form.append("size", size);
if (model.includes("gpt-image")) {
form.append("quality", quality === "2k" ? "high" : "medium");
const openAIQuality = getOpenAIQuality(model, quality);
if (openAIQuality && openAIQuality !== "standard" && openAIQuality !== "hd") {
form.append("quality", openAIQuality);
}
for (const refPath of referenceImages) {
+47 -17
View File
@@ -1,7 +1,7 @@
---
name: baoyu-infographic
description: Generate professional infographics with 21 layout types and 21 visual styles. Analyzes content, recommends layout×style combinations, and generates publication-ready infographics. Use when user asks to create "infographic", "信息图", "visual summary", "可视化", or "高密度信息大图".
version: 1.56.1
description: Generate professional infographics with 21 layout types and 22 visual styles. Analyzes content, recommends layout×style combinations, and generates publication-ready infographics. Use when user asks to create "infographic", "信息图", "visual summary", "可视化", or "高密度信息大图".
version: 1.58.0
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-infographic
@@ -23,11 +23,17 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## Image Generation Tools
When this skill needs to render an image:
When this skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends. Users change the pinned backend via the `## Changing Preferences` section below.
**Prompt file requirement (hard)**: write each image's full, final prompt to a standalone file under `prompts/` (naming: `NN-{type}-[slug].md`) BEFORE invoking any backend. The backend receives the prompt file (or its content); the file is the reproducibility record and lets you switch backends without regenerating prompts.
@@ -64,14 +70,24 @@ references:
- If `usage: direct` AND the chosen backend accepts reference images (e.g., `baoyu-imagine` via `--ref`) → pass the file via the backend's ref parameter
- Otherwise → embed extracted `style`/`palette` traits in the prompt text
## Confirmation Policy
Default behavior: **confirm before generation**.
- Treat explicit skill invocation, a file path, a matched keyword shortcut, `EXTEND.md` defaults, and the documented default combination as **recommendation inputs only**. None of them authorizes skipping confirmation.
- Do **not** start Step 5 or Step 6 until the user confirms the combination/aspect/language/backend choices.
- Skip confirmation only when the current request explicitly says to do so, for example: `--no-confirm`, "直接生成", "不用确认", "跳过确认", "按默认出图", or equivalent wording.
- If confirmation is skipped explicitly, state the assumed combination/aspect/language/backend in the next user-facing update before generating.
## Options
| Option | Values |
|--------|--------|
| `--layout` | 21 options (see Layout Gallery), default: bento-grid |
| `--style` | 21 options (see Style Gallery), default: craft-handmade |
| `--style` | 22 options (see Style Gallery), default: craft-handmade |
| `--aspect` | Named: landscape (16:9), portrait (9:16), square (1:1). Custom: any W:H ratio (e.g., 3:4, 4:3, 2.35:1) |
| `--lang` | en, zh, ja, etc. |
| `--no-confirm` | Skip Step 4 only when the user explicitly requests direct generation without confirmation |
| `--ref <files...>` | Reference images (file paths) for style / palette / composition / subject guidance |
## Layout Gallery (21)
@@ -102,7 +118,7 @@ references:
Full definitions live at `references/layouts/<layout>.md`.
## Style Gallery (21)
## Style Gallery (22)
| Style | Description |
|-------|-------------|
@@ -127,6 +143,7 @@ Full definitions live at `references/layouts/<layout>.md`.
| `morandi-journal` | Hand-drawn doodle, warm Morandi tones |
| `retro-pop-grid` | 1970s retro pop art, Swiss grid, thick outlines |
| `hand-drawn-edu` | Macaron pastels, hand-drawn wobble, stick figures |
| `retro-popup-pop` | Retro popup collage, vintage UI, thick outlines, flat pop colors |
Full definitions live at `references/styles/<style>.md`.
@@ -149,18 +166,19 @@ Full definitions live at `references/styles/<style>.md`.
| Product Guide | `dense-modules` + `morandi-journal` |
| Technical Guide | `dense-modules` + `pop-laboratory` |
| Trendy Guide | `dense-modules` + `retro-pop-grid` |
| Retro Pop Guide | `dense-modules` + `retro-popup-pop` |
| Educational Diagram | `hub-spoke` + `hand-drawn-edu` |
| Process Tutorial | `linear-progression` + `hand-drawn-edu` |
Default combination: `bento-grid` + `craft-handmade`.
Default combination: `bento-grid` + `craft-handmade` (fallback recommendation only — per the [Confirmation Policy](#confirmation-policy), defaults never bypass Step 4).
## Keyword Shortcuts
When the user's input contains these keywords, auto-select the layout and promote the listed styles to the top of Step 3 recommendations. Skip content-based layout inference for matched keywords. Append any `Prompt Notes` to the Step 5 prompt.
When the user's input contains these keywords, use the mapped layout as the leading Step 3 recommendation and promote the listed styles to the top of the Step 3 list. Skip content-based layout inference for matched keywords. Append any `Prompt Notes` to the Step 5 prompt.
| User Keyword | Layout | Recommended Styles | Default Aspect | Prompt Notes |
|--------------|--------|--------------------|----------------|--------------|
| 高密度信息大图 / high-density-info | `dense-modules` | `morandi-journal`, `pop-laboratory`, `retro-pop-grid` | portrait | — |
| 高密度信息大图 / high-density-info | `dense-modules` | `morandi-journal`, `pop-laboratory`, `retro-pop-grid`, `retro-popup-pop` | portrait | — |
| 信息图 / infographic | `bento-grid` | `craft-handmade` | landscape | Minimalist: clean canvas, ample whitespace, no complex background textures. Simple cartoon elements and icons only. |
## Output Structure
@@ -201,7 +219,7 @@ Check EXTEND.md in priority order — the first one found wins:
| Found | Read, parse, display a one-line summary |
| Not found | Ask the user with `AskUserQuestion` (see `references/config/first-time-setup.md`) |
**EXTEND.md supports**: preferred layout/style, default aspect ratio, custom style definitions, language preference.
**EXTEND.md supports**: preferred layout/style, default aspect ratio, language preference, preferred image backend, custom style definitions.
Schema: `references/config/preferences-schema.md`
@@ -231,7 +249,7 @@ See `references/structured-content-template.md` for detailed format.
### Step 3: Recommend Combinations
**3.1 Check Keyword Shortcuts first**: If user input matches a keyword from the **Keyword Shortcuts** table, auto-select the associated layout and prioritize associated styles as top recommendations. Skip content-based layout inference.
**3.1 Check Keyword Shortcuts first**: If user input matches a keyword from the **Keyword Shortcuts** table, use the associated layout as the leading recommendation and prioritize associated styles as top recommendations. Skip content-based layout inference.
**3.2 Otherwise**, recommend 3-5 layout×style combinations based on:
- Data structure → matching layout
@@ -241,6 +259,8 @@ See `references/structured-content-template.md` for detailed format.
### Step 4: Confirm Options
**Hard gate**: this step is mandatory per the [Confirmation Policy](#confirmation-policy) — Steps 56 cannot start until the user confirms here (or explicitly opts out with `--no-confirm` / equivalent in the current request).
Ask the user to confirm the questions below following the [User Input Tools](#user-input-tools) rule at the top of this file (batch into one call if the runtime supports multiple questions; otherwise ask one at a time in priority order).
| Priority | Question | When | Options |
@@ -248,6 +268,7 @@ Ask the user to confirm the questions below following the [User Input Tools](#us
| 1 | **Combination** | Always | 3+ layout×style combos with rationale |
| 2 | **Aspect** | Always | Named presets (landscape/portrait/square) or custom W:H ratio (e.g., 3:4, 4:3, 2.35:1) |
| 3 | **Language** | Only if source ≠ user language | Language for text content |
| 4 | **Image Backend** | Only if step 3 of the `## Image Generation Tools` rule needs to ask (no runtime-native tool AND multiple non-native backends, OR `preferred_image_backend: ask`) | Available backends |
### Step 5: Generate Prompt → `prompts/infographic.md`
@@ -266,7 +287,7 @@ Combine:
### Step 6: Generate Image
1. Select the backend via the `## Image Generation Tools` rule at the top: use whatever is available; if multiple, ask the user once. Do this once per session.
1. Resolve the backend per the `## Image Generation Tools` rule at the top of this file.
2. Ensure the full final prompt is persisted at `prompts/infographic.md` (already written in Step 5) BEFORE invoking the backend — the file is the reproducibility record.
3. **Check for existing file**: Before generating, check if `infographic.png` exists
- If exists: Rename to `infographic-backup-YYYYMMDD-HHMMSS.png`
@@ -275,7 +296,7 @@ Combine:
### Step 7: Output Summary
Report: topic, layout, style, aspect, language, output path, files created.
Report: topic, layout, style, aspect, language, image backend, output path, files created.
## References
@@ -285,6 +306,15 @@ Report: topic, layout, style, aspect, language, output path, files created.
- `references/layouts/<layout>.md` - 21 layout definitions
- `references/styles/<style>.md` - 21 style definitions
## Extension Support
## Changing Preferences
Custom configurations via EXTEND.md. See **Step 1.1** for paths and supported options.
EXTEND.md lives at the first matching path in Step 1.1. Three ways to change it:
- **Edit directly** — open EXTEND.md and change fields. Full schema: `references/config/preferences-schema.md`.
- **Reconfigure interactively** — delete EXTEND.md (or ask "reconfigure baoyu-infographic preferences" / "重新配置"). The next run re-triggers first-time setup.
- **Common one-line edits**:
- `preferred_image_backend: auto` — default; runtime-native tool wins, falls back to the only installed backend, asks only if multiple non-native are present.
- `preferred_image_backend: codex-imagegen` — pin to Codex's built-in.
- `preferred_image_backend: baoyu-imagine` — pin to the baoyu-imagine skill.
- `preferred_image_backend: ask` — confirm backend every run.
- `preferred_layout: dense-modules`, `preferred_style: morandi-journal`, `preferred_aspect: portrait`, `language: zh` — shift the Step-3 recommendations and Step-4 defaults (per [Confirmation Policy](#confirmation-policy), these never bypass Step 4).
@@ -39,7 +39,7 @@ Approach content analysis as a **world-class instructional designer**:
| **System/Structure** | Components, architecture, anatomy | structural-breakdown, bento-grid | technical-schematic, ikea-manual |
| **Journey/Narrative** | Stories, user flows, milestones | winding-roadmap, story-mountain | storybook-watercolor, comic-strip |
| **Overview/Summary** | Multiple topics, feature highlights | bento-grid, periodic-table, dense-modules | chalkboard, bold-graphic |
| **Product/Buying Guide** | Multi-dimension comparisons, specs, pitfalls | dense-modules | morandi-journal, pop-laboratory, retro-pop-grid |
| **Product/Buying Guide** | Multi-dimension comparisons, specs, pitfalls | dense-modules | morandi-journal, pop-laboratory, retro-pop-grid, retro-popup-pop |
### 2. Learning Objective Identification
@@ -7,7 +7,7 @@ description: First-time setup flow for baoyu-infographic preferences
## Overview
When no EXTEND.md is found, guide the user through preference setup before generating any infographic.
When no EXTEND.md is found, guide the user through preference setup before generating any infographic. Saved preferences shift Step-3 recommendations and Step-4 defaults only — they never bypass Step 4 confirmation (see the `## Confirmation Policy` section in SKILL.md).
**⛔ BLOCKING OPERATION**: This setup MUST complete before ANY other workflow steps. Do NOT:
- Ask about source content or topic
@@ -141,13 +141,13 @@ preferred_layout: [selected layout or null]
preferred_style: [selected style or null]
preferred_aspect: [landscape|portrait|square|null]
language: [selected language or null]
preferred_image_backend: auto
custom_styles: []
---
```
`preferred_image_backend: auto` is the baked-in default — first-time setup never asks about it. The `## Image Generation Tools` rule in SKILL.md then picks the runtime-native tool (Codex `imagegen`, Hermes `image_generate`, etc.) when one is available, and falls back to installed backends like `baoyu-imagine`.
## Modifying Preferences Later
Users can edit EXTEND.md directly or trigger setup again:
- Delete EXTEND.md to re-trigger setup
- Edit YAML frontmatter for quick changes
- Full schema: `references/config/preferences-schema.md`
See the `## Changing Preferences` section in `SKILL.md` for the canonical list of common edits (pin backend, change layout/style defaults, retrigger setup). Full schema: `references/config/preferences-schema.md`.
@@ -17,6 +17,8 @@ preferred_aspect: null # landscape|portrait|square|null (custom W:H also acc
language: null # zh|en|ja|ko|null (null = auto-detect from source)
preferred_image_backend: auto # auto|ask|<backend-id>
custom_styles: # extra style definitions merged with the 21 built-ins
- name: my-brand
description: "Short description shown in Step 3 recommendations"
@@ -33,8 +35,21 @@ custom_styles: # extra style definitions merged with the 21 built-ins
| `preferred_style` | string\|null | null | Pre-selected style — surfaces as the top recommendation in Step 3 |
| `preferred_aspect` | string\|null | null | Default aspect for Step 4 (named preset or W:H string) |
| `language` | string\|null | null | Output language (null = auto-detect from source content) |
| `preferred_image_backend` | string | `auto` | Image backend selection. `auto` = prefer runtime-native tool, fall back to the only installed backend, ask if multiple non-native are present. `ask` = always confirm on every run. `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) = pin this backend when available; fall back to `auto` when it isn't. Absent = `auto`. |
| `custom_styles` | array | [] | Additional styles available alongside the 21 built-ins |
Backend resolution logic is documented in the `## Image Generation Tools` section of `SKILL.md`. This doc only defines the field.
All fields in this schema are defaults only — they shape Step-3 recommendations and Step-4 defaults but never bypass Step 4 confirmation (see the `## Confirmation Policy` section in SKILL.md).
Example backend ids:
| Value | Meaning |
|-------|---------|
| `codex-imagegen` | Codex built-in `imagegen` tool |
| `baoyu-imagine` | `baoyu-imagine` skill / script backend |
| `image_generate` | Generic runtime image tool such as Hermes |
## Layout Options
See the **Layout Gallery (21)** table in `SKILL.md` for the canonical list. Common picks:
@@ -87,6 +102,8 @@ language: zh
---
```
`preferred_image_backend` is omitted above; absence is treated as `auto`.
## Example: Full Preferences
```yaml
@@ -99,6 +116,8 @@ preferred_aspect: portrait
language: zh
preferred_image_backend: codex-imagegen
custom_styles:
- name: my-brand
description: "Brand-aligned warm pastel infographic"
@@ -68,5 +68,6 @@ High-density modular layout with 6-7 typed information modules packed with concr
- `pop-laboratory`: Technical precision with coordinate markers and blueprint grid
- `morandi-journal`: Hand-drawn warmth with doodle illustrations and organic frames
- `retro-pop-grid`: 1970s pop art with strict grid cells and bold contrast
- `retro-popup-pop`: Vintage desktop popups with chunky pixel UI for retro-tech dense guides
- `corporate-memphis`: Clean business feel for product comparisons
- `technical-schematic`: Engineering precision for technical product guides
@@ -0,0 +1,50 @@
# retro-popup-pop
Retro pixel popup × pop-art collage — content rendered as a stack of 80/90s desktop dialog windows with thick black outlines, flat color fills, and bright cyan or vintage cream backgrounds.
## Color Palette
- Background: Bright cyan (#12B8DE) primary canvas, vintage cream (#F5F0E6) alternate
- Window fills: Vintage cream (#F5F0E6) and pure white (#FFFFFF)
- Primary text and outlines: Pure black (#000000)
- Reverse text: Pure white on solid black title bars
- Accent fills (small areas only): Salmon pink, sky blue, mustard yellow, mint green — muted retro tones
## Visual Elements
- 80s/90s desktop popup windows with title bars, close buttons (×), and chunky borders
- Multiple windows tiled or lightly overlapping with offset solid-black drop shadows for depth
- ERROR / ALERT / WARNING dialogs to highlight misconceptions, common pitfalls, risks
- File-window vignettes labeled with retro filenames (PROBLEMS.EXE, METHOD.PNG, BURNOUT.PSD, FOCUS_SCAN, SYSTEM_ALERT, IDEAS_MISSING)
- Progress bars for completion, difficulty, conversion rate, or trend
- Pixel-style chunky icons: folders, magnifiers, gears, exclamation marks, floppy disks, hourglasses
- Action buttons with thick black borders and short pop-style copy: OK, CANCEL, FIX IT, SCAN, PLAY, RETRY, SAVE
- Comparison data rendered as windowed lists or small tabular dialogs
- Uniform thick black outlines on every window, button, icon, and divider
- Pure 2D flat color fills throughout — no gradients, no glow, no glass
## Typography
- Headers: Pixel / dot-matrix / chunky bitmap display fonts, large and high-contrast
- Body: Retro monospace or system-style sans-serif, high legibility
- Title bars: Reverse white on solid black, all-caps preferred
- Decorative all-caps English allowed for filenames, status strings, button labels (PROBLEMS.EXE, OK, CANCEL); body content text remains in the confirmed output language
## Style Enforcement
- Every information chunk must live inside a window, dialog, button, or progress bar — no floating elements
- Uniform thick black outlines everywhere; offset solid-black drop shadows for stacked-window depth
- Limit accent palette to ~4 colors per composition; let cyan or cream dominate the canvas
- Maintain low-fi pixel/popup feel; humorous popup copy welcome, polished modern UI prohibited
## Avoid
- Modern UI styles: glassmorphism, neumorphism, soft shadows, blurs, gradients
- 3D rendering, photorealism, ray-traced shadows
- Hand-drawn wobble, watercolor washes, or sketch textures
- Smooth anti-aliased curves on icons (prefer chunky pixel/stair-step edges)
- Pure black backgrounds — keep cyan or cream as the canvas
## Best For
High-density "干货" knowledge guides, common-pitfall and debunking posts, knowledge-pop content for design-savvy or developer audiences, Xiaohongshu-style retro-tech posts, dense-modules portrait infographics.
+1 -1
View File
@@ -1,7 +1,7 @@
---
name: baoyu-post-to-x
description: Posts content and articles to X (Twitter). Supports regular posts with images/videos and X Articles (long-form Markdown). Uses real Chrome with CDP to bypass anti-automation. Use when user asks to "post to X", "tweet", "publish to Twitter", or "share on X".
version: 1.56.1
version: 1.56.2
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-post-to-x
+7 -4
View File
@@ -5,6 +5,7 @@ import os from 'node:os';
import path from 'node:path';
import process from 'node:process';
import { createHash } from 'node:crypto';
import { pathToFileURL } from 'node:url';
import frontMatter from 'front-matter';
import hljs from 'highlight.js/lib/common';
@@ -458,7 +459,9 @@ async function main(): Promise<void> {
}
}
await main().catch((err) => {
console.error(`Error: ${err instanceof Error ? err.message : String(err)}`);
process.exit(1);
});
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
await main().catch((err) => {
console.error(`Error: ${err instanceof Error ? err.message : String(err)}`);
process.exit(1);
});
}
+35 -6
View File
@@ -1,7 +1,7 @@
---
name: baoyu-slide-deck
description: Generates professional slide deck images from content. Creates outlines with style instructions, then generates individual slide images. Use when user asks to "create slides", "make a presentation", "generate deck", "slide deck", or "PPT".
version: 1.56.1
version: 1.56.2
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-slide-deck
@@ -27,14 +27,31 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## Image Generation Tools
When this skill needs to render an image:
When this skill needs to render an image, resolve the backend in this order:
- **Use whatever image-generation tool or skill is available** in the current runtime — e.g., Codex `imagegen`, Hermes `image_generate`, `baoyu-imagine`, or any equivalent the user has installed.
- **If multiple are available**, ask the user **once** at the start which to use (batch with any other initial questions).
- **If none are available**, tell the user and ask how to proceed.
1. **Current-request override** if the user names a specific backend in the current message, use it.
2. **Saved preference** — if `EXTEND.md` sets `preferred_image_backend` to a backend available right now, use it.
3. **Auto-select** (when the preference is `auto`, unset, or the pinned backend isn't available):
- If the current runtime exposes a native image tool (e.g., Codex `imagegen`, Hermes `image_generate`), use it. Runtime-native tools are preferred by default — agents that know their own tool inventory should surface the native one here.
- Otherwise, if exactly one non-native backend is installed (e.g., `baoyu-imagine`), use it.
- Otherwise (multiple non-native backends with no runtime-native tool), ask the user once — batch with any other initial questions.
4. **If none are available**, tell the user and ask how to proceed.
Setting `preferred_image_backend: ask` forces the step-3 prompt every run regardless of available backends. Users change the pinned backend via the `## Changing Preferences` section below.
**Prompt file requirement (hard)**: write each image's full, final prompt to a standalone file under `prompts/` (naming: `NN-slide-[slug].md`) BEFORE invoking any backend. The file is the reproducibility record and lets you switch backends without regenerating prompts.
Concrete tool names (`imagegen`, `image_generate`, `baoyu-imagine`) above are examples — substitute the local equivalents under the same rule.
## Confirmation Policy
Default behavior: **confirm before generation**.
- Treat explicit skill invocation, a file path, matched signals/presets, and `EXTEND.md` defaults as **recommendation inputs only**. None of them authorizes skipping confirmation.
- Do **not** start Step 3 or later until the user completes Step 2.
- Skip confirmation only when the current request explicitly says to do so, for example: "直接生成", "不用确认", "跳过确认", "按默认出幻灯片", or equivalent wording.
- If confirmation is skipped explicitly, state the assumed style / audience / slide-count / language / backend in the next user-facing update before generating.
## Language
Respond in the user's language across questions, progress reports, error messages, and the completion summary. Keep technical tokens (style names, file paths, code) in English.
@@ -213,6 +230,8 @@ Save findings to `analysis.md`: topic, audience, signals, recommended style and
### Step 2: Confirmation ⚠️ REQUIRED
**Hard gate**: this step is mandatory per the [Confirmation Policy](#confirmation-policy) — Steps 3+ cannot start until the user confirms here (or explicitly opts out with "直接生成" / equivalent wording in the current request).
**Round 1 (always)** — batch five questions in one `AskUserQuestion` call: style, audience, slide count, review-outline?, review-prompts?. Verbatim options in `references/confirmation.md`.
Summary displayed before the questions:
@@ -320,4 +339,14 @@ See `references/modification-guide.md` for full details.
- For sensitive public figures, prefer stylized alternatives to avoid likeness issues.
- Maintain visual consistency via the session ID when the backend supports it.
Custom configurations via EXTEND.md. See Step 1.1 for paths and schema.
## Changing Preferences
EXTEND.md lives at the first matching path listed in Step 1.1. Two ways to change it:
- **Edit directly** — open EXTEND.md and change fields. Full schema: `references/config/preferences-schema.md`.
- **Common one-line edits**:
- `preferred_image_backend: auto` — default; runtime-native tool wins, falls back to the only installed backend, asks only if multiple non-native are present.
- `preferred_image_backend: codex-imagegen` — pin to Codex's built-in.
- `preferred_image_backend: baoyu-imagine` — pin to the baoyu-imagine skill.
- `preferred_image_backend: ask` — confirm backend every run.
- `preferred_style: blueprint`, `preferred_audience: experts`, `language: zh`.
@@ -12,6 +12,7 @@ style: blueprint # Preset name OR "custom"
audience: general # beginners | intermediate | experts | executives | general
language: auto # auto | en | zh | ja | etc.
review: true # true = review outline before generation
preferred_image_backend: auto # auto | ask | <backend-id>
## Custom Dimensions (only when style: custom)
dimensions:
@@ -40,6 +41,7 @@ custom_styles:
| `audience` | string | `general` | Default target audience |
| `language` | string | `auto` | Output language (auto = detect from input) |
| `review` | boolean | `true` | Show outline review before generation |
| `preferred_image_backend` | string | `auto` | Image backend selection. `auto` = prefer runtime-native tool, fall back to the only installed backend, ask if multiple non-native are present. `ask` = always confirm on every run. `<backend-id>` (e.g., `codex-imagegen`, `baoyu-imagine`, `image_generate`) = pin this backend when available; fall back to `auto` when it isn't. Absent = `auto`. Resolution logic is documented in `SKILL.md`'s `## Image Generation Tools` section. |
### Custom Dimensions
+4 -5
View File
@@ -1,14 +1,13 @@
---
name: baoyu-url-to-markdown
description: Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.
version: 1.60.0
version: 1.61.0
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-url-to-markdown
requires:
anyBins:
- bun
- npx
---
# URL to Markdown
@@ -27,13 +26,13 @@ Concrete `AskUserQuestion` references below are examples — substitute the loca
## CLI Setup
**Important**: The CLI is provided by the npm package dependency `baoyu-fetch`.
**Important**: The CLI source is vendored in `{baseDir}/scripts/lib`. `scripts/package.json` installs only third-party runtime dependencies.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `{baseDir}`
2. Resolve `${BUN}` runtime: if `bun` installed → `bun`; else suggest installing Bun
3. If `{baseDir}/scripts/node_modules/.bin/baoyu-fetch` does not exist, run `${BUN} install --cwd {baseDir}/scripts`
4. `${READER}` = `{baseDir}/scripts/node_modules/.bin/baoyu-fetch`
3. If `{baseDir}/scripts/node_modules` does not exist, run `${BUN} install --cwd {baseDir}/scripts`
4. `${READER}` = `{baseDir}/scripts/baoyu-fetch`
5. Replace all `${READER}` in this document with the resolved value
## Preferences (EXTEND.md)
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env sh
set -eu
script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
exec bun "$script_dir/lib/cli.ts" "$@"
+55 -39
View File
@@ -5,22 +5,46 @@
"": {
"name": "baoyu-url-to-markdown-scripts",
"dependencies": {
"baoyu-fetch": "^0.1.2",
"@mozilla/readability": "^0.6.0",
"chrome-launcher": "^1.2.1",
"defuddle": "^0.17.0",
"jsdom": "^29.0.2",
"remark-gfm": "^4.0.1",
"remark-parse": "^11.0.0",
"turndown": "^7.2.0",
"turndown-plugin-gfm": "^1.0.2",
"unified": "^11.0.5",
"ws": "^8.18.3",
},
},
},
"overrides": {
"@xmldom/xmldom": "0.8.13",
},
"packages": {
"@asamuzakjp/css-color": ["@asamuzakjp/css-color@3.2.0", "", { "dependencies": { "@csstools/css-calc": "^2.1.3", "@csstools/css-color-parser": "^3.0.9", "@csstools/css-parser-algorithms": "^3.0.4", "@csstools/css-tokenizer": "^3.0.3", "lru-cache": "^10.4.3" } }, "sha512-K1A6z8tS3XsmCMM86xoWdn7Fkdn9m6RSVtocUrJYIwZnFVkng/PvkEoWtOWmP+Scc6saYWHWZYbndEEXxl24jw=="],
"@asamuzakjp/css-color": ["@asamuzakjp/css-color@5.1.11", "", { "dependencies": { "@asamuzakjp/generational-cache": "^1.0.1", "@csstools/css-calc": "^3.2.0", "@csstools/css-color-parser": "^4.1.0", "@csstools/css-parser-algorithms": "^4.0.0", "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg=="],
"@csstools/color-helpers": ["@csstools/color-helpers@5.1.0", "", {}, "sha512-S11EXWJyy0Mz5SYvRmY8nJYTFFd1LCNV+7cXyAgQtOOuzb4EsgfqDufL+9esx72/eLhsRdGZwaldu/h+E4t4BA=="],
"@asamuzakjp/dom-selector": ["@asamuzakjp/dom-selector@7.1.1", "", { "dependencies": { "@asamuzakjp/generational-cache": "^1.0.1", "@asamuzakjp/nwsapi": "^2.3.9", "bidi-js": "^1.0.3", "css-tree": "^3.2.1", "is-potential-custom-element-name": "^1.0.1" } }, "sha512-67RZDnYRc8H/8MLDgQCDE//zoqVFwajkepHZgmXrbwybzXOEwOWGPYGmALYl9J2DOLfFPPs6kKCqmbzV895hTQ=="],
"@csstools/css-calc": ["@csstools/css-calc@2.1.4", "", { "peerDependencies": { "@csstools/css-parser-algorithms": "^3.0.5", "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-3N8oaj+0juUw/1H3YwmDDJXCgTB1gKU6Hc/bB502u9zR0q2vd786XJH9QfrKIEgFlZmhZiq6epXl4rHqhzsIgQ=="],
"@asamuzakjp/generational-cache": ["@asamuzakjp/generational-cache@1.0.1", "", {}, "sha512-wajfB8KqzMCN2KGNFdLkReeHncd0AslUSrvHVvvYWuU8ghncRJoA50kT3zP9MVL0+9g4/67H+cdvBskj9THPzg=="],
"@csstools/css-color-parser": ["@csstools/css-color-parser@3.1.0", "", { "dependencies": { "@csstools/color-helpers": "^5.1.0", "@csstools/css-calc": "^2.1.4" }, "peerDependencies": { "@csstools/css-parser-algorithms": "^3.0.5", "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-nbtKwh3a6xNVIp/VRuXV64yTKnb1IjTAEEh3irzS+HkKjAOYLTGNb9pmVNntZ8iVBHcWDA2Dof0QtPgFI1BaTA=="],
"@asamuzakjp/nwsapi": ["@asamuzakjp/nwsapi@2.3.9", "", {}, "sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q=="],
"@csstools/css-parser-algorithms": ["@csstools/css-parser-algorithms@3.0.5", "", { "peerDependencies": { "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-DaDeUkXZKjdGhgYaHNJTV9pV7Y9B3b644jCLs9Upc3VeNGg6LWARAT6O+Q+/COo+2gg/bM5rhpMAtf70WqfBdQ=="],
"@bramus/specificity": ["@bramus/specificity@2.4.2", "", { "dependencies": { "css-tree": "^3.0.0" }, "bin": { "specificity": "bin/cli.js" } }, "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw=="],
"@csstools/css-tokenizer": ["@csstools/css-tokenizer@3.0.4", "", {}, "sha512-Vd/9EVDiu6PPJt9yAh6roZP6El1xHrdvIVGjyBsHR0RYwNHgL7FJPyIIW4fANJNG6FtyZfvlRPpFI4ZM/lubvw=="],
"@csstools/color-helpers": ["@csstools/color-helpers@6.0.2", "", {}, "sha512-LMGQLS9EuADloEFkcTBR3BwV/CGHV7zyDxVRtVDTwdI2Ca4it0CCVTT9wCkxSgokjE5Ho41hEPgb8OEUwoXr6Q=="],
"@csstools/css-calc": ["@csstools/css-calc@3.2.0", "", { "peerDependencies": { "@csstools/css-parser-algorithms": "^4.0.0", "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-bR9e6o2BDB12jzN/gIbjHa5wLJ4UjD1CB9pM7ehlc0ddk6EBz+yYS1EV2MF55/HUxrHcB/hehAyt5vhsA3hx7w=="],
"@csstools/css-color-parser": ["@csstools/css-color-parser@4.1.0", "", { "dependencies": { "@csstools/color-helpers": "^6.0.2", "@csstools/css-calc": "^3.2.0" }, "peerDependencies": { "@csstools/css-parser-algorithms": "^4.0.0", "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-U0KhLYmy2GVj6q4T3WaAe6NPuFYCPQoE3b0dRGxejWDgcPp8TP7S5rVdM5ZrFaqu4N67X8YaPBw14dQSYx3IyQ=="],
"@csstools/css-parser-algorithms": ["@csstools/css-parser-algorithms@4.0.0", "", { "peerDependencies": { "@csstools/css-tokenizer": "^4.0.0" } }, "sha512-+B87qS7fIG3L5h3qwJ/IFbjoVoOe/bpOdh9hAjXbvx0o8ImEmUsGXN0inFOnk2ChCFgqkkGFQ+TpM5rbhkKe4w=="],
"@csstools/css-syntax-patches-for-csstree": ["@csstools/css-syntax-patches-for-csstree@1.1.3", "", { "peerDependencies": { "css-tree": "^3.2.1" }, "optionalPeers": ["css-tree"] }, "sha512-SH60bMfrRCJF3morcdk57WklujF4Jr/EsQUzqkarfHXEFcAR1gg7fS/chAE922Sehgzc1/+Tz5H3Ypa1HiEKrg=="],
"@csstools/css-tokenizer": ["@csstools/css-tokenizer@4.0.0", "", {}, "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA=="],
"@exodus/bytes": ["@exodus/bytes@1.15.0", "", { "peerDependencies": { "@noble/hashes": "^1.8.0 || ^2.0.0" }, "optionalPeers": ["@noble/hashes"] }, "sha512-UY0nlA+feH81UGSHv92sLEPLCeZFjXOuHhrIo0HQydScuQc8s0A7kL/UdgwgDq8g8ilksmuoF35YVTNphV2aBQ=="],
"@mixmark-io/domino": ["@mixmark-io/domino@2.2.0", "", {}, "sha512-Y28PR25bHXUg88kCV7nivXrP2Nj2RueZ3/l/jdx6J9f8J4nsEGcgX0Qe6lt7Pa+J79+kPiJU3LguR6O/6zrLOw=="],
@@ -38,11 +62,9 @@
"@xmldom/xmldom": ["@xmldom/xmldom@0.8.13", "", {}, "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw=="],
"agent-base": ["agent-base@7.1.4", "", {}, "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ=="],
"bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="],
"baoyu-fetch": ["baoyu-fetch@0.1.2", "", { "dependencies": { "@mozilla/readability": "^0.6.0", "chrome-launcher": "^1.2.1", "defuddle": "^0.14.0", "jsdom": "^26.0.0", "remark-gfm": "^4.0.1", "remark-parse": "^11.0.0", "turndown": "^7.2.0", "turndown-plugin-gfm": "^1.0.2", "unified": "^11.0.5", "ws": "^8.18.3" }, "bin": { "baoyu-fetch": "dist/cli.js" } }, "sha512-VB4CEtIcoiJo3m80RrwBuYOdC257VI6kptoI1kiRcZV78VWEicWdznRoByrjddgyHTq0WWEK5i/7L67mHcVPvQ=="],
"bidi-js": ["bidi-js@1.0.3", "", { "dependencies": { "require-from-string": "^2.0.2" } }, "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw=="],
"boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="],
@@ -56,13 +78,13 @@
"css-select": ["css-select@5.2.2", "", { "dependencies": { "boolbase": "^1.0.0", "css-what": "^6.1.0", "domhandler": "^5.0.2", "domutils": "^3.0.1", "nth-check": "^2.0.1" } }, "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw=="],
"css-tree": ["css-tree@3.2.1", "", { "dependencies": { "mdn-data": "2.27.1", "source-map-js": "^1.2.1" } }, "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA=="],
"css-what": ["css-what@6.2.2", "", {}, "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA=="],
"cssom": ["cssom@0.5.0", "", {}, "sha512-iKuQcq+NdHqlAcwUY0o/HL69XQrUaQdMjmStJ8JFmUaiiQErlhrmuigkg/CU4E2J0IyUKUrMAgl36TvN67MqTw=="],
"cssstyle": ["cssstyle@4.6.0", "", { "dependencies": { "@asamuzakjp/css-color": "^3.2.0", "rrweb-cssom": "^0.8.0" } }, "sha512-2z+rWdzbbSZv6/rhtvzvqeZQHrBaqgogqt85sqFNbabZOuFbCVFb8kPeEtZjiKkbrm395irpNKiYeFeLiQnFPg=="],
"data-urls": ["data-urls@5.0.0", "", { "dependencies": { "whatwg-mimetype": "^4.0.0", "whatwg-url": "^14.0.0" } }, "sha512-ZYP5VBHshaDAiVZxjbRVcFJpc+4xGgT0bK3vzy1HLN8jTO975HEbuYzZJcHoQEY5K1a0z8YayJkyVETa08eNTg=="],
"data-urls": ["data-urls@7.0.0", "", { "dependencies": { "whatwg-mimetype": "^5.0.0", "whatwg-url": "^16.0.0" } }, "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA=="],
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
@@ -70,7 +92,7 @@
"decode-named-character-reference": ["decode-named-character-reference@1.3.0", "", { "dependencies": { "character-entities": "^2.0.0" } }, "sha512-GtpQYB283KrPp6nRw50q3U9/VfOutZOe103qlN7BPP6Ad27xYnOIWv4lPzo8HCAL+mMZofJ9KEy30fq6MfaK6Q=="],
"defuddle": ["defuddle@0.14.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "linkedom": "^0.18.12", "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-btavZGd1WgiVqrVM62WGRXMUi/aU7ckTZiq0xXWLZMHvzIqNZjwIFQEDRx8MarD7fIgsB90NXZ9xHJkKtapt2Q=="],
"defuddle": ["defuddle@0.17.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "linkedom": "^0.18.12", "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-rjRb9kxgrJLFhFJjXTSAitbmmSyvjEqGCi5JiyM6ImOf4leC/O+f7TPVPAymk/gnENFTrn9j7T/LTfW9IVwiLw=="],
"dequal": ["dequal@2.0.3", "", {}, "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA=="],
@@ -84,24 +106,18 @@
"domutils": ["domutils@3.2.2", "", { "dependencies": { "dom-serializer": "^2.0.0", "domelementtype": "^2.3.0", "domhandler": "^5.0.3" } }, "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw=="],
"entities": ["entities@6.0.1", "", {}, "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g=="],
"entities": ["entities@8.0.0", "", {}, "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA=="],
"escape-string-regexp": ["escape-string-regexp@4.0.0", "", {}, "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA=="],
"extend": ["extend@3.0.2", "", {}, "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g=="],
"html-encoding-sniffer": ["html-encoding-sniffer@4.0.0", "", { "dependencies": { "whatwg-encoding": "^3.1.1" } }, "sha512-Y22oTqIU4uuPgEemfz7NDJz6OeKf12Lsu+QC+s3BVpda64lTiMYCyGwg5ki4vFxkMwQdeZDl2adZoqUgdFuTgQ=="],
"html-encoding-sniffer": ["html-encoding-sniffer@6.0.0", "", { "dependencies": { "@exodus/bytes": "^1.6.0" } }, "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg=="],
"html-escaper": ["html-escaper@3.0.3", "", {}, "sha512-RuMffC89BOWQoY0WKGpIhn5gX3iI54O6nRA0yC124NYVtzjmFWBIiFd8M0x+ZdX0P9R4lADg1mgP8C7PxGOWuQ=="],
"htmlparser2": ["htmlparser2@10.1.0", "", { "dependencies": { "domelementtype": "^2.3.0", "domhandler": "^5.0.3", "domutils": "^3.2.2", "entities": "^7.0.1" } }, "sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ=="],
"http-proxy-agent": ["http-proxy-agent@7.0.2", "", { "dependencies": { "agent-base": "^7.1.0", "debug": "^4.3.4" } }, "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig=="],
"https-proxy-agent": ["https-proxy-agent@7.0.6", "", { "dependencies": { "agent-base": "^7.1.2", "debug": "4" } }, "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw=="],
"iconv-lite": ["iconv-lite@0.6.3", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw=="],
"is-docker": ["is-docker@2.2.1", "", { "bin": { "is-docker": "cli.js" } }, "sha512-F+i2BKsFrH66iaUFc0woD8sLy8getkwTwtOBjvs56Cx4CgJDeKQeqfz8wAYiSb8JOprWhHH5p77PbmYCvvUuXQ=="],
"is-plain-obj": ["is-plain-obj@4.1.0", "", {}, "sha512-+Pgi+vMuUNkJyExiMBt5IlFoMyKnr5zhJ4Uspz58WOhBF5QoIZkFyNHIbBAtHwzVAgk5RtndVNsDRN61/mmDqg=="],
@@ -110,7 +126,7 @@
"is-wsl": ["is-wsl@2.2.0", "", { "dependencies": { "is-docker": "^2.0.0" } }, "sha512-fKzAra0rGJUUBwGBgNkHZuToZcn+TtXHpeCgmkMJMMYx1sQDYaCSyjJBSCa2nH1DGm7s3n1oBnohoVTBaN7Lww=="],
"jsdom": ["jsdom@26.1.0", "", { "dependencies": { "cssstyle": "^4.2.1", "data-urls": "^5.0.0", "decimal.js": "^10.5.0", "html-encoding-sniffer": "^4.0.0", "http-proxy-agent": "^7.0.2", "https-proxy-agent": "^7.0.6", "is-potential-custom-element-name": "^1.0.1", "nwsapi": "^2.2.16", "parse5": "^7.2.1", "rrweb-cssom": "^0.8.0", "saxes": "^6.0.0", "symbol-tree": "^3.2.4", "tough-cookie": "^5.1.1", "w3c-xmlserializer": "^5.0.0", "webidl-conversions": "^7.0.0", "whatwg-encoding": "^3.1.1", "whatwg-mimetype": "^4.0.0", "whatwg-url": "^14.1.1", "ws": "^8.18.0", "xml-name-validator": "^5.0.0" }, "peerDependencies": { "canvas": "^3.0.0" }, "optionalPeers": ["canvas"] }, "sha512-Cvc9WUhxSMEo4McES3P7oK3QaXldCfNWp7pl2NNeiIFlCoLr3kfq9kb1fxftiwk1FLV7CvpvDfonxtzUDeSOPg=="],
"jsdom": ["jsdom@29.0.2", "", { "dependencies": { "@asamuzakjp/css-color": "^5.1.5", "@asamuzakjp/dom-selector": "^7.0.6", "@bramus/specificity": "^2.4.2", "@csstools/css-syntax-patches-for-csstree": "^1.1.1", "@exodus/bytes": "^1.15.0", "css-tree": "^3.2.1", "data-urls": "^7.0.0", "decimal.js": "^10.6.0", "html-encoding-sniffer": "^6.0.0", "is-potential-custom-element-name": "^1.0.1", "lru-cache": "^11.2.7", "parse5": "^8.0.0", "saxes": "^6.0.0", "symbol-tree": "^3.2.4", "tough-cookie": "^6.0.1", "undici": "^7.24.5", "w3c-xmlserializer": "^5.0.0", "webidl-conversions": "^8.0.1", "whatwg-mimetype": "^5.0.0", "whatwg-url": "^16.0.1", "xml-name-validator": "^5.0.0" }, "peerDependencies": { "canvas": "^3.0.0" }, "optionalPeers": ["canvas"] }, "sha512-9VnGEBosc/ZpwyOsJBCQ/3I5p7Q5ngOY14a9bf5btenAORmZfDse1ZEheMiWcJ3h81+Fv7HmJFdS0szo/waF2w=="],
"lighthouse-logger": ["lighthouse-logger@2.0.2", "", { "dependencies": { "debug": "^4.4.1", "marky": "^1.2.2" } }, "sha512-vWl2+u5jgOQuZR55Z1WM0XDdrJT6mzMP8zHUct7xTlWhuQs+eV0g+QL0RQdFjT54zVmbhLCP8vIVpy1wGn/gCg=="],
@@ -118,7 +134,7 @@
"longest-streak": ["longest-streak@3.1.0", "", {}, "sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g=="],
"lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="],
"lru-cache": ["lru-cache@11.3.5", "", {}, "sha512-NxVFwLAnrd9i7KUBxC4DrUhmgjzOs+1Qm50D3oF1/oL+r1NpZ4gA7xvG0/zJ8evR7zIKn4vLf7qTNduWFtCrRw=="],
"markdown-table": ["markdown-table@3.0.4", "", {}, "sha512-wiYz4+JrLyb/DqW2hkFJxP7Vd7JuTDm77fvbM8VfEQdmSMqcImWeeRbHwZjBjIFki/VaMK2BhFi7oUUZeM5bqw=="],
@@ -148,6 +164,8 @@
"mdast-util-to-string": ["mdast-util-to-string@4.0.0", "", { "dependencies": { "@types/mdast": "^4.0.0" } }, "sha512-0H44vDimn51F0YwvxSJSm0eCDOJTRlmN0R1yBh4HLj9wiV1Dn0QoXGbvFAWj2hSItVTlCmBF1hqKlIyUBVFLPg=="],
"mdn-data": ["mdn-data@2.27.1", "", {}, "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ=="],
"micromark": ["micromark@4.0.2", "", { "dependencies": { "@types/debug": "^4.0.0", "debug": "^4.0.0", "decode-named-character-reference": "^1.0.0", "devlop": "^1.0.0", "micromark-core-commonmark": "^2.0.0", "micromark-factory-space": "^2.0.0", "micromark-util-character": "^2.0.0", "micromark-util-chunked": "^2.0.0", "micromark-util-combine-extensions": "^2.0.0", "micromark-util-decode-numeric-character-reference": "^2.0.0", "micromark-util-encode": "^2.0.0", "micromark-util-normalize-identifier": "^2.0.0", "micromark-util-resolve-all": "^2.0.0", "micromark-util-sanitize-uri": "^2.0.0", "micromark-util-subtokenize": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-zpe98Q6kvavpCr1NPVSCMebCKfD7CA2NqZ+rykeNhONIJBpc1tFKt9hucLGwha3jNTNI8lHpctWJWoimVF4PfA=="],
"micromark-core-commonmark": ["micromark-core-commonmark@2.0.3", "", { "dependencies": { "decode-named-character-reference": "^1.0.0", "devlop": "^1.0.0", "micromark-factory-destination": "^2.0.0", "micromark-factory-label": "^2.0.0", "micromark-factory-space": "^2.0.0", "micromark-factory-title": "^2.0.0", "micromark-factory-whitespace": "^2.0.0", "micromark-util-character": "^2.0.0", "micromark-util-chunked": "^2.0.0", "micromark-util-classify-character": "^2.0.0", "micromark-util-html-tag-name": "^2.0.0", "micromark-util-normalize-identifier": "^2.0.0", "micromark-util-resolve-all": "^2.0.0", "micromark-util-subtokenize": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-RDBrHEMSxVFLg6xvnXmb1Ayr2WzLAWjeSATAoxwKYJV94TeNavgoIdA0a9ytzDSVzBy2YKFK+emCPOEibLeCrg=="],
@@ -208,9 +226,7 @@
"nth-check": ["nth-check@2.1.1", "", { "dependencies": { "boolbase": "^1.0.0" } }, "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w=="],
"nwsapi": ["nwsapi@2.2.23", "", {}, "sha512-7wfH4sLbt4M0gCDzGE6vzQBo0bfTKjU7Sfpqy/7gs1qBfYz2vEJH6vXcBKpO3+6Yu1telwd0t9HpyOoLEQQbIQ=="],
"parse5": ["parse5@7.3.0", "", { "dependencies": { "entities": "^6.0.0" } }, "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw=="],
"parse5": ["parse5@8.0.1", "", { "dependencies": { "entities": "^8.0.0" } }, "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw=="],
"punycode": ["punycode@2.3.1", "", {}, "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg=="],
@@ -220,23 +236,23 @@
"remark-stringify": ["remark-stringify@11.0.0", "", { "dependencies": { "@types/mdast": "^4.0.0", "mdast-util-to-markdown": "^2.0.0", "unified": "^11.0.0" } }, "sha512-1OSmLd3awB/t8qdoEOMazZkNsfVTeY4fTsgzcQFdXNq8ToTN4ZGwrMnlda4K6smTFKD+GRV6O48i6Z4iKgPPpw=="],
"rrweb-cssom": ["rrweb-cssom@0.8.0", "", {}, "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw=="],
"safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="],
"require-from-string": ["require-from-string@2.0.2", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="],
"saxes": ["saxes@6.0.0", "", { "dependencies": { "xmlchars": "^2.2.0" } }, "sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA=="],
"source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
"symbol-tree": ["symbol-tree@3.2.4", "", {}, "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw=="],
"temml": ["temml@0.13.2", "", {}, "sha512-n8fDRSsLscq9nh9j6z+FgkCvFMT0IJm6GCgwfzh+7AHT3Sfb4jFTQlsA6hVcF2dYYr3b66oDBVES95RfoukyrA=="],
"tldts": ["tldts@6.1.86", "", { "dependencies": { "tldts-core": "^6.1.86" }, "bin": { "tldts": "bin/cli.js" } }, "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ=="],
"tldts": ["tldts@7.0.28", "", { "dependencies": { "tldts-core": "^7.0.28" }, "bin": { "tldts": "bin/cli.js" } }, "sha512-+Zg3vWhRUv8B1maGSTFdev9mjoo8Etn2Ayfs4cnjlD3CsGkxXX4QyW3j2WJ0wdjYcYmy7Lx2RDsZMhgCWafKIw=="],
"tldts-core": ["tldts-core@6.1.86", "", {}, "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA=="],
"tldts-core": ["tldts-core@7.0.28", "", {}, "sha512-7W5Efjhsc3chVdFhqtaU0KtK32J37Zcr9RKtID54nG+tIpcY79CQK/veYPODxtD/LJ4Lue66jvrQzIX2Z2/pUQ=="],
"tough-cookie": ["tough-cookie@5.1.2", "", { "dependencies": { "tldts": "^6.1.32" } }, "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A=="],
"tough-cookie": ["tough-cookie@6.0.1", "", { "dependencies": { "tldts": "^7.0.5" } }, "sha512-LktZQb3IeoUWB9lqR5EWTHgW/VTITCXg4D21M+lvybRVdylLrRMnqaIONLVb5mav8vM19m44HIcGq4qASeu2Qw=="],
"tr46": ["tr46@5.1.1", "", { "dependencies": { "punycode": "^2.3.1" } }, "sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw=="],
"tr46": ["tr46@6.0.0", "", { "dependencies": { "punycode": "^2.3.1" } }, "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw=="],
"trough": ["trough@2.2.0", "", {}, "sha512-tmMpK00BjZiUyVyvrBK7knerNgmgvcV/KLVyuma/SC+TQN167GrMRciANTz09+k3zW8L8t60jWO1GpfkZdjTaw=="],
@@ -246,6 +262,8 @@
"uhyphen": ["uhyphen@0.2.0", "", {}, "sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA=="],
"undici": ["undici@7.25.0", "", {}, "sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ=="],
"undici-types": ["undici-types@7.19.2", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
"unified": ["unified@11.0.5", "", { "dependencies": { "@types/unist": "^3.0.0", "bail": "^2.0.0", "devlop": "^1.0.0", "extend": "^3.0.0", "is-plain-obj": "^4.0.0", "trough": "^2.0.0", "vfile": "^6.0.0" } }, "sha512-xKvGhPWw3k84Qjh8bI3ZeJjqnyadK+GEFtazSfZv/rKeTkTjOJho6mFqh2SM96iIcZokxiOpg78GazTSg8+KHA=="],
@@ -264,13 +282,11 @@
"w3c-xmlserializer": ["w3c-xmlserializer@5.0.0", "", { "dependencies": { "xml-name-validator": "^5.0.0" } }, "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA=="],
"webidl-conversions": ["webidl-conversions@7.0.0", "", {}, "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g=="],
"webidl-conversions": ["webidl-conversions@8.0.1", "", {}, "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ=="],
"whatwg-encoding": ["whatwg-encoding@3.1.1", "", { "dependencies": { "iconv-lite": "0.6.3" } }, "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ=="],
"whatwg-mimetype": ["whatwg-mimetype@5.0.0", "", {}, "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw=="],
"whatwg-mimetype": ["whatwg-mimetype@4.0.0", "", {}, "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg=="],
"whatwg-url": ["whatwg-url@14.2.0", "", { "dependencies": { "tr46": "^5.1.0", "webidl-conversions": "^7.0.0" } }, "sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw=="],
"whatwg-url": ["whatwg-url@16.0.1", "", { "dependencies": { "@exodus/bytes": "^1.11.0", "tr46": "^6.0.0", "webidl-conversions": "^8.0.1" } }, "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw=="],
"ws": ["ws@8.20.0", "", { "peerDependencies": { "bufferutil": "^4.0.1", "utf-8-validate": ">=5.0.2" }, "optionalPeers": ["bufferutil", "utf-8-validate"] }, "sha512-sAt8BhgNbzCtgGbt2OxmpuryO63ZoDk/sqaB/znQm94T4fCEsy/yV+7CdC1kJhOU9lboAEU7R3kquuycDoibVA=="],
@@ -0,0 +1,74 @@
import type { Adapter } from "../types";
import { detectInteractionGate } from "../../browser/interaction-gates";
import { captureNormalizedPageSnapshot } from "../../browser/page-snapshot";
import { convertHtmlToMarkdown } from "../../extract/html-to-markdown";
export const genericAdapter: Adapter = {
name: "generic",
match() {
return true;
},
async process(context) {
context.log.info(`Loading ${context.input.url.toString()} with generic adapter`);
await context.browser.goto(context.input.url.toString(), context.timeoutMs);
try {
await context.network.waitForIdle({
idleMs: 1_200,
timeoutMs: Math.min(context.timeoutMs, 15_000),
});
} catch {
context.log.debug("Network idle timed out on initial load; continuing.");
}
await context.browser.scrollToEnd({ maxSteps: 4, delayMs: 300 });
try {
await context.network.waitForIdle({
idleMs: 900,
timeoutMs: Math.min(context.timeoutMs, 10_000),
});
} catch {
context.log.debug("Network idle timed out after scrolling; continuing.");
}
const interaction = await detectInteractionGate(context.browser);
if (interaction) {
return {
status: "needs_interaction",
interaction,
};
}
const snapshot = await captureNormalizedPageSnapshot(context.browser);
const converted = await convertHtmlToMarkdown(snapshot.html, snapshot.finalUrl, {
enableRemoteMarkdownFallback: context.outputFormat === "markdown",
preserveBase64Images: context.downloadMedia,
});
const document = {
url: snapshot.finalUrl,
canonicalUrl: converted.metadata.canonicalUrl,
title: converted.metadata.title,
author: converted.metadata.author,
siteName: converted.metadata.siteName,
publishedAt: converted.metadata.publishedAt,
summary: converted.metadata.summary,
adapter: "generic",
metadata: {
coverImage: converted.metadata.coverImage,
language: converted.metadata.language,
capturedAt: converted.metadata.capturedAt,
conversionMethod: converted.conversionMethod,
fallbackReason: converted.fallbackReason,
kind: "generic/article",
},
content: converted.markdown ? [{ type: "markdown" as const, markdown: converted.markdown }] : [],
};
return {
status: "ok",
document,
media: converted.media,
};
},
};
@@ -0,0 +1,391 @@
import { JSDOM } from "jsdom";
import TurndownService from "turndown";
import { gfm } from "turndown-plugin-gfm";
import type { Adapter } from "../types";
import type { ExtractedDocument } from "../../extract/document";
import { collectMediaFromDocument } from "../../media/markdown-media";
const HN_BASE_URL = "https://news.ycombinator.com";
const turndown = new TurndownService({
headingStyle: "atx",
bulletListMarker: "-",
codeBlockStyle: "fenced",
});
turndown.use(gfm);
export interface HnItem {
id: number;
type: "story" | "comment" | "job" | "poll" | "pollopt" | string;
by?: string;
time?: number;
text?: string;
title?: string;
url?: string;
score?: number;
descendants?: number;
kids?: number[];
parent?: number;
deleted?: boolean;
dead?: boolean;
}
export interface HnCommentNode {
item: HnItem;
children: HnCommentNode[];
}
interface ParsedHnThread {
story: HnItem;
comments: HnCommentNode[];
}
function decodeHtmlText(value: string | undefined): string | undefined {
if (!value) {
return undefined;
}
const dom = new JSDOM(`<!doctype html><html><body>${value}</body></html>`);
return dom.window.document.body.textContent?.trim() || undefined;
}
function normalizeMarkdown(markdown: string): string {
return markdown
.replace(/\r\n/g, "\n")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
}
function convertHnHtmlToMarkdown(html: string | undefined, baseUrl: string): string {
if (!html?.trim()) {
return "";
}
const dom = new JSDOM(`<div id="__root">${html}</div>`, { url: baseUrl });
const root = dom.window.document.querySelector("#__root");
if (!root) {
return "";
}
root.querySelectorAll("a[href]").forEach((element) => {
const href = element.getAttribute("href");
if (!href) {
return;
}
try {
element.setAttribute("href", new URL(href, baseUrl).toString());
} catch {
// Ignore malformed URLs and keep the original href.
}
});
return normalizeMarkdown(turndown.turndown(root.innerHTML));
}
function formatIsoTimestamp(unixSeconds: number | undefined): string | undefined {
if (!unixSeconds || !Number.isFinite(unixSeconds)) {
return undefined;
}
return new Date(unixSeconds * 1_000).toISOString();
}
function formatDisplayTimestamp(unixSeconds: number | undefined): string {
const iso = formatIsoTimestamp(unixSeconds);
if (!iso) {
return "unknown time";
}
return iso.replace("T", " ").replace(".000Z", " UTC");
}
function indentMarkdown(markdown: string, spaces: number): string {
const prefix = " ".repeat(spaces);
return markdown
.split("\n")
.map((line) => (line ? `${prefix}${line}` : prefix))
.join("\n");
}
function renderCommentHeader(item: HnItem, pageUrl: string): string {
const author = item.by ?? "[deleted]";
const time = item.id
? `[${formatDisplayTimestamp(item.time)}](${pageUrl}#${item.id})`
: formatDisplayTimestamp(item.time);
return `${author} · ${time}`;
}
function renderCommentNode(node: HnCommentNode, pageUrl: string, depth = 0): string {
const baseIndent = " ".repeat(depth * 4);
const lines = [`${baseIndent}- ${renderCommentHeader(node.item, pageUrl)}`];
const body = convertHnHtmlToMarkdown(node.item.text, pageUrl);
if (body) {
lines.push("");
lines.push(indentMarkdown(body, depth * 4 + 4));
} else if (node.item.deleted || node.item.dead) {
lines.push("");
lines.push(`${baseIndent} [comment unavailable]`);
}
for (const child of node.children) {
lines.push("");
lines.push(renderCommentNode(child, pageUrl, depth + 1));
}
return lines.join("\n");
}
export function buildHnThreadMarkdown(
story: HnItem,
comments: HnCommentNode[],
pageUrl: string,
): string {
const lines: string[] = [];
const storyUrl = story.url ? new URL(story.url, pageUrl).toString() : undefined;
const storyText = convertHnHtmlToMarkdown(story.text, pageUrl);
if (storyUrl && storyUrl !== pageUrl) {
lines.push(`Source: [${storyUrl}](${storyUrl})`);
}
lines.push(`HN Item: [${story.id}](${pageUrl})`);
const submittedBy = story.by ? ` by ${story.by}` : "";
const submittedAt = formatDisplayTimestamp(story.time);
lines.push(`Submitted${submittedBy} at ${submittedAt}`);
const stats: string[] = [];
if (typeof story.score === "number") {
stats.push(`${story.score} points`);
}
if (typeof story.descendants === "number") {
stats.push(`${story.descendants} comments`);
}
if (stats.length > 0) {
lines.push(stats.join(" | "));
}
if (storyText) {
lines.push("");
lines.push("## Post");
lines.push("");
lines.push(storyText);
}
lines.push("");
lines.push("## Comments");
lines.push("");
if (comments.length === 0) {
lines.push("No comments.");
} else {
lines.push(comments.map((comment) => renderCommentNode(comment, pageUrl)).join("\n\n"));
}
return normalizeMarkdown(lines.join("\n"));
}
export function buildHnDocument(
story: HnItem,
comments: HnCommentNode[],
pageUrl: string,
): ExtractedDocument {
const decodedTitle = decodeHtmlText(story.title) ?? `HN Item ${story.id}`;
return {
url: pageUrl,
canonicalUrl: pageUrl,
title: decodedTitle,
author: story.by,
siteName: "Hacker News",
publishedAt: formatIsoTimestamp(story.time),
adapter: "hn",
metadata: {
kind: "hn/story",
storyId: story.id,
storyUrl: story.url ? new URL(story.url, pageUrl).toString() : undefined,
points: story.score,
commentCount: story.descendants,
},
content: [
{
type: "markdown",
markdown: buildHnThreadMarkdown(story, comments, pageUrl),
},
],
};
}
export function parseHnItemId(url: URL): number | null {
if (url.hostname !== "news.ycombinator.com") {
return null;
}
if (url.pathname !== "/item") {
return null;
}
const value = url.searchParams.get("id");
if (!value || !/^\d+$/.test(value)) {
return null;
}
return Number(value);
}
function extractUnixSecondsFromAge(element: Element | null): number | undefined {
const title = element?.getAttribute("title")?.trim();
if (!title) {
return undefined;
}
const match = title.match(/(\d{9,})$/);
return match ? Number(match[1]) : undefined;
}
function extractScore(text: string | null | undefined): number | undefined {
if (!text) {
return undefined;
}
const match = text.match(/(\d+)/);
return match ? Number(match[1]) : undefined;
}
function extractCommentCount(container: ParentNode): number | undefined {
const anchors = Array.from(container.querySelectorAll("a"));
for (const anchor of anchors) {
const match = anchor.textContent?.trim().match(/(\d+)\s+comments?/i);
if (match) {
return Number(match[1]);
}
}
return undefined;
}
function normalizeStoryUrl(storyId: number, href: string | null | undefined, pageUrl: string): string | undefined {
if (!href) {
return undefined;
}
try {
const resolved = new URL(href, pageUrl).toString();
if (resolved === pageUrl || resolved === `${HN_BASE_URL}/item?id=${storyId}`) {
return undefined;
}
return resolved;
} catch {
return undefined;
}
}
export function extractHnThreadFromHtml(html: string, pageUrl: string): ParsedHnThread | null {
const dom = new JSDOM(html, { url: pageUrl });
const { document } = dom.window;
const storyRow = document.querySelector("table.fatitem tr.athing.submission");
if (!storyRow) {
return null;
}
const storyId = Number(storyRow.getAttribute("id"));
if (!Number.isFinite(storyId)) {
return null;
}
const titleLink = storyRow.querySelector(".titleline > a");
const subline = document.querySelector("table.fatitem .subline");
const topText = document.querySelector("table.fatitem .toptext");
const story: HnItem = {
id: storyId,
type: "story",
by: subline?.querySelector(".hnuser")?.textContent?.trim() || undefined,
time: extractUnixSecondsFromAge(subline?.querySelector(".age") ?? null),
title: titleLink?.innerHTML?.trim() || undefined,
url: normalizeStoryUrl(storyId, titleLink?.getAttribute("href"), pageUrl),
text: topText?.innerHTML?.trim() || undefined,
score: extractScore(subline?.querySelector(".score")?.textContent),
descendants: extractCommentCount(subline ?? document),
};
const roots: HnCommentNode[] = [];
const stack: HnCommentNode[] = [];
document.querySelectorAll("tr.athing.comtr").forEach((row) => {
const commentId = Number(row.getAttribute("id"));
if (!Number.isFinite(commentId)) {
return;
}
const indentRaw = row.querySelector("td.ind")?.getAttribute("indent");
const depth = indentRaw && /^\d+$/.test(indentRaw) ? Number(indentRaw) : 0;
const comhead = row.querySelector(".comhead");
const item: HnItem = {
id: commentId,
type: "comment",
by: comhead?.querySelector(".hnuser")?.textContent?.trim() || undefined,
time: extractUnixSecondsFromAge(comhead?.querySelector(".age") ?? null),
text: row.querySelector(".comment > .commtext")?.innerHTML?.trim() || undefined,
deleted: row.querySelector(".comment > .commtext") === null,
};
const node: HnCommentNode = {
item,
children: [],
};
while (stack.length > depth) {
stack.pop();
}
const parent = stack[stack.length - 1];
if (parent) {
parent.children.push(node);
} else {
roots.push(node);
}
stack.push(node);
});
return {
story,
comments: roots,
};
}
export const hnAdapter: Adapter = {
name: "hn",
match(input) {
return parseHnItemId(input.url) !== null;
},
async process(context) {
const itemId = parseHnItemId(context.input.url);
if (!itemId) {
return {
status: "no_document",
};
}
const pageUrl = context.input.url.toString();
context.log.info(`Loading ${pageUrl} with hn adapter`);
await context.browser.goto(pageUrl, context.timeoutMs);
const html = await context.browser.getHTML();
const thread = extractHnThreadFromHtml(html, pageUrl);
if (!thread) {
return {
status: "no_document",
};
}
const document = buildHnDocument(thread.story, thread.comments, pageUrl);
return {
status: "ok",
document,
media: collectMediaFromDocument(document),
};
},
};
@@ -0,0 +1,29 @@
import type { Adapter, AdapterInput } from "./types";
import { genericAdapter } from "./generic";
import { hnAdapter } from "./hn";
import { xAdapter } from "./x";
import { youtubeAdapter } from "./youtube";
const adapters: Adapter[] = [xAdapter, youtubeAdapter, hnAdapter, genericAdapter];
export function listAdapters(): Adapter[] {
return adapters;
}
export function resolveAdapter(input: AdapterInput, forcedName?: string): Adapter {
if (forcedName) {
const forced = adapters.find((adapter) => adapter.name === forcedName);
if (!forced) {
throw new Error(`Unknown adapter: ${forcedName}`);
}
return forced;
}
const matched = adapters.find((adapter) => adapter.match(input));
if (!matched) {
throw new Error("No adapter matched the URL");
}
return matched;
}
export { genericAdapter };
@@ -0,0 +1,73 @@
import type { BrowserSession } from "../browser/session";
import type { CdpClient } from "../browser/cdp-client";
import type { NetworkJournal } from "../browser/network-journal";
import type { ExtractedDocument } from "../extract/document";
import type { MediaDownloadRequest, MediaDownloadResult, MediaAsset } from "../media/types";
import type { Logger } from "../utils/logger";
export interface AdapterInput {
url: URL;
}
export type LoginState = "logged_in" | "logged_out" | "unknown";
export type InteractionKind = "login" | "cloudflare" | "recaptcha" | "hcaptcha" | "captcha" | "challenge";
export interface AdapterLoginInfo {
provider: string;
state: LoginState;
required?: boolean;
username?: string;
reason?: string;
}
export interface WaitForInteractionRequest {
type: "wait_for_interaction";
kind: InteractionKind;
provider: string;
prompt: string;
reason?: string;
timeoutMs?: number;
pollIntervalMs?: number;
requiresVisibleBrowser?: boolean;
}
export type AdapterProcessResult =
| {
status: "ok";
document: ExtractedDocument;
media?: MediaAsset[];
login?: AdapterLoginInfo;
}
| {
status: "needs_interaction";
interaction: WaitForInteractionRequest;
login?: AdapterLoginInfo;
}
| {
status: "no_document";
login?: AdapterLoginInfo;
};
export interface AdapterContext {
input: AdapterInput;
browser: BrowserSession;
network: NetworkJournal;
cdp: CdpClient;
log: Logger;
outputFormat: "markdown" | "json";
timeoutMs: number;
interactive: boolean;
downloadMedia: boolean;
}
export interface Adapter {
name: string;
match(input: AdapterInput): boolean;
checkLogin?(context: AdapterContext): Promise<AdapterLoginInfo>;
exportCookies?(context: AdapterContext, profileDir?: string): Promise<boolean>;
restoreCookies?(context: AdapterContext, profileDir?: string): Promise<boolean>;
downloadMedia?(request: MediaDownloadRequest): Promise<MediaDownloadResult>;
process(context: AdapterContext): Promise<AdapterProcessResult>;
}
export type { MediaAsset };
@@ -0,0 +1,461 @@
import type { ExtractedDocument } from "../../extract/document";
import {
findTweetNode,
findTweetNodeById,
formatMediaList,
formatTweetAuthor,
getTweetAuthorMetadata,
getTweetText,
getUser,
isRecord,
normalizeTitle,
resolveBestXVideoVariantUrl,
toHighResXImageUrl,
toXTweet,
} from "./shared";
import type { JsonObject } from "./types";
interface ArticleMedia {
kind: "image" | "video";
url: string;
}
function resolveArticleMedia(mediaInfo: JsonObject): ArticleMedia | null {
const videoUrl = resolveBestXVideoVariantUrl(mediaInfo);
if (videoUrl) {
return {
kind: "video",
url: videoUrl,
};
}
const rawUrl =
(typeof mediaInfo.original_img_url === "string" && mediaInfo.original_img_url) ||
(typeof mediaInfo.url === "string" && mediaInfo.url) ||
"";
if (!rawUrl) {
return null;
}
return {
kind: "image",
url: toHighResXImageUrl(rawUrl),
};
}
function resolveArticleMediaUrl(mediaInfo: JsonObject): string {
return resolveArticleMedia(mediaInfo)?.url ?? "";
}
function normalizeEntityMap(entityMap: unknown): Map<string, JsonObject> {
const normalized = new Map<string, JsonObject>();
if (Array.isArray(entityMap)) {
for (const entry of entityMap) {
if (!isRecord(entry)) {
continue;
}
const key =
typeof entry.key === "string" || typeof entry.key === "number"
? String(entry.key)
: undefined;
const value = isRecord(entry.value) ? entry.value : undefined;
if (!key || !value) {
continue;
}
normalized.set(key, value);
}
return normalized;
}
if (!isRecord(entityMap)) {
return normalized;
}
for (const [key, value] of Object.entries(entityMap)) {
if (!isRecord(value)) {
continue;
}
normalized.set(key, value);
}
return normalized;
}
function getEntityMarkdown(entityMap: Map<string, JsonObject>, entityKey: unknown): string | null {
const key =
typeof entityKey === "string" || typeof entityKey === "number"
? String(entityKey)
: undefined;
if (!key) {
return null;
}
const entity = entityMap.get(key);
if (!entity || entity.type !== "MARKDOWN") {
return null;
}
const data = isRecord(entity.data) ? entity.data : {};
if (typeof data.markdown !== "string") {
return null;
}
const markdown = data.markdown.trim();
return markdown || null;
}
function getLinkUrl(entityMap: Map<string, JsonObject>, entityKey: unknown): string | null {
const key =
typeof entityKey === "string" || typeof entityKey === "number"
? String(entityKey)
: undefined;
if (!key) {
return null;
}
const entity = entityMap.get(key);
if (!entity || entity.type !== "LINK") {
return null;
}
const data = isRecord(entity.data) ? entity.data : {};
const candidates = [
data.expanded_url,
data.expandedUrl,
data.original_url,
data.originalUrl,
data.url,
data.display_url,
data.displayUrl,
];
for (const candidate of candidates) {
if (typeof candidate === "string" && candidate.trim()) {
return candidate.trim();
}
}
return null;
}
function getTweetId(entityMap: Map<string, JsonObject>, entityKey: unknown): string | null {
const key =
typeof entityKey === "string" || typeof entityKey === "number"
? String(entityKey)
: undefined;
if (!key) {
return null;
}
const entity = entityMap.get(key);
if (!entity || entity.type !== "TWEET") {
return null;
}
const data = isRecord(entity.data) ? entity.data : {};
if (typeof data.tweetId !== "string") {
return null;
}
return data.tweetId;
}
function buildMediaMap(articleResult: JsonObject): Map<string, ArticleMedia> {
const mediaMap = new Map<string, ArticleMedia>();
const mediaEntities = Array.isArray(articleResult.media_entities) ? articleResult.media_entities : [];
for (const entity of mediaEntities) {
if (!isRecord(entity) || typeof entity.media_id !== "string" || !isRecord(entity.media_info)) {
continue;
}
const media = resolveArticleMedia(entity.media_info);
if (media) {
mediaMap.set(entity.media_id, media);
}
}
const coverMedia = isRecord(articleResult.cover_media) ? articleResult.cover_media : null;
if (coverMedia && typeof coverMedia.media_id === "string" && isRecord(coverMedia.media_info)) {
const media = resolveArticleMedia(coverMedia.media_info);
if (media) {
mediaMap.set(coverMedia.media_id, media);
}
}
return mediaMap;
}
function getMediaMarkdown(
entityMap: Map<string, JsonObject>,
entityKey: unknown,
mediaMap: Map<string, ArticleMedia>,
): string[] {
const key =
typeof entityKey === "string" || typeof entityKey === "number"
? String(entityKey)
: undefined;
if (!key) {
return [];
}
const entity = entityMap.get(key);
if (!entity || entity.type !== "MEDIA") {
return [];
}
const data = isRecord(entity.data) ? entity.data : {};
const mediaItems = Array.isArray(data.mediaItems) ? data.mediaItems : [];
const media: ArticleMedia[] = [];
for (const item of mediaItems) {
if (!isRecord(item) || typeof item.mediaId !== "string") {
continue;
}
const mediaItem = mediaMap.get(item.mediaId);
if (mediaItem && !media.some((value) => value.url === mediaItem.url)) {
media.push(mediaItem);
}
}
return media.map((item) => item.kind === "image" ? `![](${item.url})` : `[video](${item.url})`);
}
function resolveTweetMarkdown(payloads: unknown[], tweetId: string, pageUrl: string): string | null {
for (const payload of payloads) {
const tweet = findTweetNodeById(payload, tweetId);
if (!tweet) {
continue;
}
const xTweet = toXTweet(tweet, pageUrl);
const author = formatTweetAuthor(xTweet) ?? xTweet.url;
const lines = [`> ${author}`, ...xTweet.text.split("\n").map((line) => `> ${line}`)];
const media = formatMediaList(xTweet.media).map((line) =>
line.startsWith("photo: ") ? `> ![](${line.slice("photo: ".length)})` : `> - ${line}`,
);
const parts = [lines.join("\n")];
if (media.length > 0) {
parts.push([">", ...media].join("\n"));
}
parts.push(`> ${xTweet.url}`);
return parts.join("\n").trim();
}
return `> Embedded tweet: https://x.com/i/status/${tweetId}`;
}
function replaceLinkEntities(text: string, block: JsonObject, entityMap: Map<string, JsonObject>): string {
const entityRanges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
const replacements = entityRanges
.filter((range): range is JsonObject => isRecord(range))
.map((range) => {
const offset = typeof range.offset === "number" ? range.offset : -1;
const length = typeof range.length === "number" ? range.length : -1;
const url = getLinkUrl(entityMap, range.key);
return { offset, length, url };
})
.filter((range) => range.offset >= 0 && range.length > 0 && range.url)
.sort((left, right) => right.offset - left.offset);
let next = text;
for (const replacement of replacements) {
next =
next.slice(0, replacement.offset) +
replacement.url +
next.slice(replacement.offset + replacement.length);
}
return next;
}
function renderAtomicBlock(
block: JsonObject,
entityMap: Map<string, JsonObject>,
mediaMap: Map<string, ArticleMedia>,
payloads: unknown[],
pageUrl: string,
): string | null {
const entityRanges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
const parts: string[] = [];
for (const range of entityRanges) {
if (!isRecord(range)) {
continue;
}
const markdown = getEntityMarkdown(entityMap, range.key);
if (markdown) {
parts.push(markdown);
continue;
}
const mediaMarkdown = getMediaMarkdown(entityMap, range.key, mediaMap);
if (mediaMarkdown.length > 0) {
parts.push(mediaMarkdown.join("\n\n"));
continue;
}
const tweetId = getTweetId(entityMap, range.key);
if (tweetId) {
const tweetMarkdown = resolveTweetMarkdown(payloads, tweetId, pageUrl);
if (tweetMarkdown) {
parts.push(tweetMarkdown);
}
}
}
if (parts.length === 0) {
return null;
}
return parts.join("\n\n");
}
function renderArticleBlocks(
blocks: unknown[],
entityMap: Map<string, JsonObject>,
mediaMap: Map<string, ArticleMedia>,
payloads: unknown[],
pageUrl: string,
): string {
const parts: string[] = [];
let orderedCounter = 0;
for (const block of blocks) {
if (!isRecord(block)) {
continue;
}
const blockType = typeof block.type === "string" ? block.type : "unstyled";
const rawText = typeof block.text === "string" ? block.text : "";
const text = replaceLinkEntities(rawText, block, entityMap).trim();
if (!text && blockType !== "atomic") {
continue;
}
if (blockType !== "ordered-list-item") {
orderedCounter = 0;
}
switch (blockType) {
case "header-one":
parts.push(`# ${text}`);
break;
case "header-two":
parts.push(`## ${text}`);
break;
case "header-three":
parts.push(`### ${text}`);
break;
case "blockquote":
parts.push(`> ${text}`);
break;
case "unordered-list-item":
parts.push(`- ${text}`);
break;
case "ordered-list-item":
orderedCounter += 1;
parts.push(`${orderedCounter}. ${text}`);
break;
case "code-block":
parts.push(`\`\`\`\n${text}\n\`\`\``);
break;
case "atomic": {
const markdown = renderAtomicBlock(block, entityMap, mediaMap, payloads, pageUrl);
if (markdown) {
parts.push(markdown);
}
break;
}
default:
parts.push(text);
break;
}
}
return parts.join("\n\n").trim();
}
function getArticleResult(tweet: JsonObject): JsonObject | null {
if (
isRecord(tweet.article) &&
isRecord(tweet.article.article_results) &&
isRecord(tweet.article.article_results.result)
) {
return tweet.article.article_results.result as JsonObject;
}
return null;
}
function extractSummary(markdown: string): string | undefined {
const segments = markdown
.split(/\n\n+/)
.map((segment) => segment.trim())
.filter(Boolean);
const preferred = segments.find((segment) => !/^(#|>|- |\d+\. |\`\`\`)/.test(segment));
return preferred?.slice(0, 220);
}
export function extractArticleDocumentFromPayload(
payload: unknown,
statusId: string,
pageUrl: string,
payloads: unknown[] = [payload],
): ExtractedDocument | null {
const tweet = findTweetNode(payload, statusId);
if (!tweet) {
return null;
}
const articleResult = getArticleResult(tweet);
if (!articleResult) {
return null;
}
const title = typeof articleResult.title === "string" ? articleResult.title.trim() : undefined;
const contentState = isRecord(articleResult.content_state) ? articleResult.content_state : {};
const blocks = Array.isArray(contentState.blocks) ? contentState.blocks : [];
const entityMap = normalizeEntityMap(contentState.entityMap);
const mediaMap = buildMediaMap(articleResult);
const richMarkdown = renderArticleBlocks(blocks, entityMap, mediaMap, payloads, pageUrl);
const plainText = typeof articleResult.plain_text === "string" ? articleResult.plain_text.trim() : "";
const markdown = richMarkdown || plainText || getTweetText(tweet);
if (!markdown) {
return null;
}
const xTweet = toXTweet(tweet, pageUrl);
const user = getUser(tweet);
const coverMedia = isRecord(articleResult.cover_media) ? articleResult.cover_media : null;
const coverMediaInfo = coverMedia && isRecord(coverMedia.media_info) ? coverMedia.media_info : null;
const coverImage = coverMediaInfo ? resolveArticleMediaUrl(coverMediaInfo) || undefined : undefined;
return {
url: pageUrl,
canonicalUrl: xTweet.url,
title: title || normalizeTitle(xTweet.text, "X Article"),
author: formatTweetAuthor(xTweet),
siteName: "X",
publishedAt: xTweet.createdAt,
summary: extractSummary(markdown) || xTweet.text.slice(0, 200) || undefined,
adapter: "x",
metadata: {
kind: "x/article",
tweetId: xTweet.id,
coverImage,
authorName: xTweet.authorName ?? user.name,
authorUsername: xTweet.author ?? user.screenName,
authorUrl: (xTweet.author ?? user.screenName) ? `https://x.com/${xTweet.author ?? user.screenName}` : undefined,
...getTweetAuthorMetadata(xTweet),
},
content: [{ type: "markdown", markdown }],
};
}
@@ -0,0 +1,134 @@
import type { Adapter, AdapterLoginInfo } from "../types";
import { exportCookies, restoreCookies, type CookieSidecarConfig } from "../../browser/cookie-sidecar";
import { detectInteractionGate } from "../../browser/interaction-gates";
import type { ExtractedDocument } from "../../extract/document";
import { collectMediaFromDocument } from "../../media/markdown-media";
import { extractArticleDocumentFromPayload } from "./article";
import { buildNeedsLoginResult, detectXLogin } from "./login";
import { extractStatusId, isXHost } from "./match";
import { collectXJsonPayloads, waitForInitialXPayload } from "./payloads";
import { extractSingleTweetDocumentFromPayload } from "./single";
import { extractThreadDocumentFromPayloads } from "./thread";
import { loadFullXThread } from "./thread-loader";
const cookieConfig: CookieSidecarConfig = {
urls: ["https://x.com/", "https://twitter.com/"],
filename: "x-session-cookies.json",
requiredCookieNames: ["auth_token", "ct0"],
filterCookie: (c) => {
const d = c.domain ?? "";
return d.endsWith("x.com") || d.endsWith("twitter.com");
},
};
function extractDocumentFromPayloads(
payloads: unknown[],
statusId: string,
pageUrl: string,
): ExtractedDocument | null {
for (const payload of payloads) {
const articleDocument = extractArticleDocumentFromPayload(payload, statusId, pageUrl, payloads);
if (articleDocument) {
return articleDocument;
}
}
const threadDocument = extractThreadDocumentFromPayloads(payloads, statusId, pageUrl);
if (threadDocument) {
return threadDocument;
}
for (const payload of payloads) {
const singleDocument = extractSingleTweetDocumentFromPayload(payload, statusId, pageUrl);
if (singleDocument) {
return singleDocument;
}
}
return null;
}
async function ensureXLoginState(context: Parameters<Adapter["process"]>[0]): Promise<AdapterLoginInfo> {
return detectXLogin(context);
}
export const xAdapter: Adapter = {
name: "x",
match(input) {
return isXHost(input.url.hostname);
},
async checkLogin(context) {
return detectXLogin(context);
},
async exportCookies(context, profileDir) {
return exportCookies(context.browser.targetSession, cookieConfig, profileDir);
},
async restoreCookies(context, profileDir) {
return restoreCookies(context.browser.targetSession, cookieConfig, profileDir);
},
async process(context) {
const statusId = extractStatusId(context.input.url);
if (!statusId) {
return {
status: "no_document",
};
}
context.log.info(`Loading ${context.input.url.toString()} with x adapter`);
await context.browser.goto(context.input.url.toString(), context.timeoutMs);
const interaction = await detectInteractionGate(context.browser);
if (interaction) {
return {
status: "needs_interaction",
interaction,
};
}
let login = await ensureXLoginState(context);
if (login.state === "logged_out") {
return buildNeedsLoginResult(login);
}
await waitForInitialXPayload(context);
await loadFullXThread(context, statusId);
const pageUrl = await context.browser.getURL();
const postLoadInteraction = await detectInteractionGate(context.browser);
if (postLoadInteraction) {
return {
status: "needs_interaction",
interaction: postLoadInteraction,
login,
};
}
login = await ensureXLoginState(context).catch(() => login);
if (login.state === "logged_out") {
return buildNeedsLoginResult(login);
}
const payloads = await collectXJsonPayloads(context);
if (payloads.length === 0) {
return {
status: "no_document",
login,
};
}
const document = extractDocumentFromPayloads(payloads, statusId, pageUrl);
if (document) {
return {
status: "ok",
document,
media: collectMediaFromDocument(document),
login,
};
}
return {
status: "no_document",
login,
};
},
};
@@ -0,0 +1,80 @@
import type { AdapterContext, AdapterLoginInfo, AdapterProcessResult } from "../types";
interface XLoginSnapshot {
currentUrl: string;
hasAccountMenu: boolean;
hasLoginInputs: boolean;
bodyText: string;
}
export async function detectXLogin(context: AdapterContext): Promise<AdapterLoginInfo> {
const snapshot = await context.browser.evaluate<XLoginSnapshot>(`
(() => {
const bodyText = (document.body?.innerText ?? "").slice(0, 2500);
return {
currentUrl: window.location.href,
hasAccountMenu: Boolean(
document.querySelector(
'[data-testid="SideNav_AccountSwitcher_Button"], [data-testid="AppTabBar_Profile_Link"], [aria-label="Account menu"]'
)
),
hasLoginInputs: Boolean(
document.querySelector(
'input[name="text"], input[name="password"], input[autocomplete="username"], input[autocomplete="current-password"]'
)
),
bodyText,
};
})()
`).catch(async () => ({
currentUrl: await context.browser.getURL().catch(() => context.input.url.toString()),
hasAccountMenu: false,
hasLoginInputs: false,
bodyText: "",
}));
if (
/\/i\/flow\/login|\/login/i.test(snapshot.currentUrl) ||
snapshot.hasLoginInputs ||
/sign in to x|join x today|登录 x|注册 x|登录到 x/i.test(snapshot.bodyText)
) {
return {
provider: "x",
state: "logged_out",
required: true,
reason: "X login page detected",
};
}
if (snapshot.hasAccountMenu) {
return {
provider: "x",
state: "logged_in",
};
}
return {
provider: "x",
state: "unknown",
};
}
export function buildNeedsLoginResult(login: AdapterLoginInfo): AdapterProcessResult {
return {
status: "needs_interaction",
login: {
...login,
provider: "x",
state: login.state === "logged_in" ? "unknown" : login.state,
required: true,
},
interaction: {
type: "wait_for_interaction",
kind: "login",
provider: "x",
reason: login.reason,
prompt: "Please sign in to X in the opened Chrome window. Extraction will continue automatically once login is detected.",
requiresVisibleBrowser: true,
},
};
}
@@ -0,0 +1,9 @@
export function isXHost(hostname: string): boolean {
return ["x.com", "www.x.com", "twitter.com", "www.twitter.com"].includes(hostname);
}
export function extractStatusId(url: URL): string | undefined {
const match = url.pathname.match(/\/(?:status|article)\/(\d+)/);
return match?.[1];
}
@@ -0,0 +1,50 @@
import type { AdapterContext } from "../types";
import { filterXGraphQlEntries } from "./shared";
export function getRelevantXThreadEntries(context: AdapterContext) {
return filterXGraphQlEntries(context.network.getEntries()).filter(
(entry) =>
entry.method === "GET" &&
entry.finished &&
(
entry.url.includes("TweetDetail") ||
entry.url.includes("TweetResultByRestId") ||
entry.url.includes("TweetResultsByRestIds")
),
);
}
export async function prefetchRelevantXThreadBodies(context: AdapterContext): Promise<void> {
const entries = getRelevantXThreadEntries(context).filter((entry) => entry.body === undefined && !entry.bodyError);
for (const entry of entries) {
await context.network.ensureBody(entry);
}
}
export async function collectXJsonPayloads(context: AdapterContext): Promise<unknown[]> {
await prefetchRelevantXThreadBodies(context);
const entries = getRelevantXThreadEntries(context);
const payloads: unknown[] = [];
for (const entry of entries) {
const payload = await context.network.getJsonBody(entry);
if (payload) {
payloads.push(payload);
}
}
return payloads;
}
export async function waitForInitialXPayload(context: AdapterContext): Promise<void> {
try {
await context.network.waitForResponse(
(entry) =>
entry.url.includes("/graphql/") &&
(entry.url.includes("TweetDetail") || entry.url.includes("TweetResultByRestId")),
{ timeoutMs: Math.min(context.timeoutMs, 15_000) },
);
await prefetchRelevantXThreadBodies(context);
} catch {
context.log.debug("No tweet GraphQL response observed before timeout.");
}
}
@@ -0,0 +1,47 @@
import type { AdapterContext } from "../types";
const X_SESSION_URLS = ["https://x.com/", "https://twitter.com/"] as const;
const REQUIRED_X_SESSION_COOKIES = ["auth_token", "ct0"] as const;
interface CookieLike {
name?: string;
value?: string | null;
}
interface NetworkGetCookiesResult {
cookies?: CookieLike[];
}
export function buildXSessionCookieMap(cookies: readonly CookieLike[]): Record<string, string> {
const cookieMap: Record<string, string> = {};
for (const cookie of cookies) {
const name = cookie.name?.trim();
const value = cookie.value?.trim();
if (!name || !value) {
continue;
}
cookieMap[name] = value;
}
return cookieMap;
}
export function hasRequiredXSessionCookies(cookieMap: Record<string, string>): boolean {
return REQUIRED_X_SESSION_COOKIES.every((name) => Boolean(cookieMap[name]));
}
export async function readXSessionCookieMap(
context: Pick<AdapterContext, "browser">,
): Promise<Record<string, string>> {
const { cookies } = await context.browser.targetSession.send<NetworkGetCookiesResult>(
"Network.getCookies",
{ urls: [...X_SESSION_URLS] },
);
return buildXSessionCookieMap(cookies ?? []);
}
export async function isXSessionReady(
context: Pick<AdapterContext, "browser">,
): Promise<boolean> {
const cookieMap = await readXSessionCookieMap(context);
return hasRequiredXSessionCookies(cookieMap);
}
@@ -0,0 +1,423 @@
import path from "node:path";
import type { NetworkEntry } from "../../browser/network-journal";
import type { XMedia, XQuotedTweet, XTweet, XUser, JsonObject } from "./types";
const X_IMAGE_EXTENSIONS = new Set(["jpg", "jpeg", "png", "webp", "gif", "bmp", "avif"]);
function emptyObject(): JsonObject {
return {};
}
export function isRecord(value: unknown): value is JsonObject {
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
}
export function walk(value: unknown, visitor: (node: unknown) => boolean | void): boolean {
if (visitor(value)) {
return true;
}
if (Array.isArray(value)) {
for (const item of value) {
if (walk(item, visitor)) {
return true;
}
}
return false;
}
if (isRecord(value)) {
for (const child of Object.values(value)) {
if (walk(child, visitor)) {
return true;
}
}
}
return false;
}
function hasTweetText(node: JsonObject): boolean {
const legacy = isRecord(node.legacy) ? node.legacy : emptyObject();
return (
typeof legacy.full_text === "string" ||
typeof getNoteTweetText(node) === "string"
);
}
export function findTweetNodeById(payload: unknown, tweetId: string): JsonObject | null {
let match: JsonObject | null = null;
walk(payload, (node) => {
if (!isRecord(node) || typeof node.rest_id !== "string" || !isRecord(node.legacy)) {
return false;
}
if (!hasTweetText(node)) {
return false;
}
if (node.rest_id === tweetId) {
match = node;
return true;
}
return false;
});
return match;
}
export function findTweetNode(payload: unknown, statusId: string): JsonObject | null {
let firstMatch: JsonObject | null = null;
const exactMatch = findTweetNodeById(payload, statusId);
if (exactMatch) {
return exactMatch;
}
walk(payload, (node) => {
if (!isRecord(node) || typeof node.rest_id !== "string" || !isRecord(node.legacy)) {
return false;
}
if (!hasTweetText(node)) {
return false;
}
if (!firstMatch) {
firstMatch = node;
}
return false;
});
return firstMatch;
}
export function getLegacy(tweet: JsonObject): JsonObject {
return isRecord(tweet.legacy) ? tweet.legacy : emptyObject();
}
export function unwrapTweetResult(node: unknown): JsonObject | null {
if (!isRecord(node)) {
return null;
}
if (node.__typename === "TweetWithVisibilityResults" && isRecord(node.tweet)) {
return unwrapTweetResult(node.tweet);
}
const tweet = isRecord(node.tweet) ? (node.tweet as JsonObject) : node;
if (typeof tweet.rest_id !== "string" || !isRecord(tweet.legacy)) {
return null;
}
return tweet;
}
export function getUser(tweet: JsonObject): XUser {
const result =
isRecord(tweet.core) &&
isRecord(tweet.core.user_results) &&
isRecord(tweet.core.user_results.result)
? (tweet.core.user_results.result as JsonObject)
: emptyObject();
const legacy = isRecord(result.legacy) ? result.legacy : emptyObject();
const core = isRecord(result.core) ? result.core : emptyObject();
return {
name:
(typeof legacy.name === "string" ? legacy.name : undefined) ??
(typeof core.name === "string" ? core.name : undefined),
screenName:
(typeof legacy.screen_name === "string" ? legacy.screen_name : undefined) ??
(typeof core.screen_name === "string" ? core.screen_name : undefined),
};
}
function getNoteTweetResult(tweet: JsonObject): JsonObject | null {
if (
!isRecord(tweet.note_tweet) ||
!isRecord(tweet.note_tweet.note_tweet_results) ||
!isRecord(tweet.note_tweet.note_tweet_results.result)
) {
return null;
}
return tweet.note_tweet.note_tweet_results.result as JsonObject;
}
function getNoteTweetText(tweet: JsonObject): string | undefined {
const noteTweet = getNoteTweetResult(tweet);
return typeof noteTweet?.text === "string" ? noteTweet.text : undefined;
}
interface TweetUrlEntity {
url: string;
expandedUrl?: string;
displayUrl?: string;
}
function collectTweetUrlEntities(values: unknown[]): TweetUrlEntity[] {
return values.reduce<TweetUrlEntity[]>((entities, value) => {
if (!isRecord(value) || typeof value.url !== "string" || !value.url) {
return entities;
}
entities.push({
url: value.url,
expandedUrl: typeof value.expanded_url === "string" ? value.expanded_url : undefined,
displayUrl: typeof value.display_url === "string" ? value.display_url : undefined,
});
return entities;
}, []);
}
function getTweetUrlEntities(tweet: JsonObject): TweetUrlEntity[] {
const noteTweet = getNoteTweetResult(tweet);
const noteTweetEntitySet = noteTweet && isRecord(noteTweet.entity_set) ? noteTweet.entity_set : emptyObject();
const noteTweetUrls = collectTweetUrlEntities(Array.isArray(noteTweetEntitySet.urls) ? noteTweetEntitySet.urls : []);
const legacy = getLegacy(tweet);
const legacyEntities = isRecord(legacy.entities) ? legacy.entities : emptyObject();
const legacyUrls = collectTweetUrlEntities(Array.isArray(legacyEntities.urls) ? legacyEntities.urls : []);
const seen = new Set<string>();
return [...noteTweetUrls, ...legacyUrls].filter((value) => {
if (seen.has(value.url)) {
return false;
}
seen.add(value.url);
return true;
});
}
export function getTweetText(tweet: JsonObject): string {
const legacy = getLegacy(tweet);
let text =
getNoteTweetText(tweet) ?? (typeof legacy.full_text === "string" ? legacy.full_text : "");
for (const value of getTweetUrlEntities(tweet)) {
const replacement =
(typeof value.expandedUrl === "string" && value.expandedUrl) ||
(typeof value.displayUrl === "string" && value.displayUrl) ||
value.url;
text = text.replaceAll(value.url, replacement);
}
const extendedEntities = isRecord(legacy.extended_entities) ? legacy.extended_entities : emptyObject();
const media = Array.isArray(extendedEntities.media) ? extendedEntities.media : [];
for (const value of media) {
if (isRecord(value) && typeof value.url === "string") {
text = text.replaceAll(value.url, "").trim();
}
}
return text.replace(/\n{3,}/g, "\n\n").trim();
}
function normalizeXImageExtension(raw: string | undefined | null): string | undefined {
if (!raw) {
return undefined;
}
const normalized = raw.replace(/^\./, "").trim().toLowerCase();
if (!normalized) {
return undefined;
}
return normalized === "jpeg" ? "jpg" : normalized;
}
export function toHighResXImageUrl(rawUrl: string): string {
try {
const parsed = new URL(rawUrl);
if (parsed.hostname.toLowerCase() !== "pbs.twimg.com") {
return rawUrl;
}
const pathExtension = normalizeXImageExtension(path.posix.extname(parsed.pathname));
const format = normalizeXImageExtension(parsed.searchParams.get("format")) ?? pathExtension;
if (!format || !X_IMAGE_EXTENSIONS.has(format)) {
return rawUrl;
}
if (pathExtension) {
parsed.pathname = parsed.pathname.replace(new RegExp(`\\.${pathExtension}$`, "i"), "");
}
parsed.searchParams.set("format", format);
parsed.searchParams.set("name", "4096x4096");
return parsed.toString();
} catch {
return rawUrl;
}
}
function getVideoVariantBitrate(variant: JsonObject): number {
const value = variant.bitrate ?? variant.bit_rate;
return typeof value === "number" && Number.isFinite(value) ? value : 0;
}
function getVideoVariantContentType(variant: JsonObject): string {
const value = variant.content_type ?? variant.contentType;
return typeof value === "string" ? value.toLowerCase() : "";
}
export function resolveBestXVideoVariantUrl(mediaInfo: unknown): string | undefined {
if (!isRecord(mediaInfo)) {
return undefined;
}
const variantsSource =
Array.isArray(mediaInfo.variants)
? mediaInfo.variants
: isRecord(mediaInfo.video_info) && Array.isArray(mediaInfo.video_info.variants)
? mediaInfo.video_info.variants
: [];
const variants = variantsSource
.filter(
(variant): variant is JsonObject =>
isRecord(variant) && typeof variant.url === "string" && variant.url.length > 0,
)
.filter((variant) => getVideoVariantContentType(variant) === "video/mp4")
.sort((left, right) => getVideoVariantBitrate(right) - getVideoVariantBitrate(left));
return typeof variants[0]?.url === "string" ? variants[0].url : undefined;
}
export function getTweetMedia(tweet: JsonObject): XMedia[] {
const legacy = getLegacy(tweet);
const extendedEntities = isRecord(legacy.extended_entities) ? legacy.extended_entities : emptyObject();
const media = Array.isArray(extendedEntities.media) ? extendedEntities.media : [];
return media
.map((value) => {
if (!isRecord(value) || typeof value.type !== "string") {
return null;
}
if (value.type === "photo" && typeof value.media_url_https === "string") {
return {
type: value.type,
url: toHighResXImageUrl(value.media_url_https),
alt: typeof value.ext_alt_text === "string" ? value.ext_alt_text : undefined,
};
}
if (value.type === "video" || value.type === "animated_gif") {
const videoUrl = resolveBestXVideoVariantUrl(value);
if (!videoUrl) {
return null;
}
return {
type: value.type,
url: videoUrl,
};
}
return null;
})
.filter((value): value is XMedia => value !== null);
}
export function getTweetUrl(tweet: JsonObject, fallbackUrl: string): string {
const user = getUser(tweet);
const fallbackScreenName = extractScreenNameFromUrl(fallbackUrl);
const id = typeof tweet.rest_id === "string" ? tweet.rest_id : "";
const screenName = user.screenName ?? fallbackScreenName;
if (screenName && id) {
return `https://x.com/${screenName}/status/${id}`;
}
return fallbackUrl;
}
export function getQuotedTweet(tweet: JsonObject, fallbackUrl: string): XQuotedTweet | undefined {
const quoted = unwrapTweetResult(
isRecord(tweet.quoted_status_result) ? tweet.quoted_status_result.result : null,
);
if (!quoted) {
return undefined;
}
const user = getUser(quoted);
return {
id: typeof quoted.rest_id === "string" ? quoted.rest_id : "",
author: user.screenName,
authorName: user.name,
text: getTweetText(quoted),
url: getTweetUrl(quoted, fallbackUrl),
media: getTweetMedia(quoted),
};
}
export function extractScreenNameFromUrl(url: string): string | undefined {
try {
const parsed = new URL(url);
const match = parsed.pathname.match(/^\/([^/]+)\/(?:status|article)\//);
if (!match) {
return undefined;
}
if (match[1] === "i") {
return undefined;
}
return match[1];
} catch {
return undefined;
}
}
export function toXTweet(tweet: JsonObject, fallbackUrl: string): XTweet {
const legacy = getLegacy(tweet);
const user = getUser(tweet);
const fallbackScreenName = extractScreenNameFromUrl(fallbackUrl);
const screenName = user.screenName ?? fallbackScreenName;
return {
id: typeof tweet.rest_id === "string" ? tweet.rest_id : "",
author: screenName,
authorName: user.name,
text: getTweetText(tweet),
likes: typeof legacy.favorite_count === "number" ? legacy.favorite_count : 0,
retweets: typeof legacy.retweet_count === "number" ? legacy.retweet_count : 0,
replies: typeof legacy.reply_count === "number" ? legacy.reply_count : 0,
createdAt: typeof legacy.created_at === "string" ? legacy.created_at : undefined,
inReplyTo: typeof legacy.in_reply_to_status_id_str === "string" ? legacy.in_reply_to_status_id_str : undefined,
url: getTweetUrl(tweet, fallbackUrl),
media: getTweetMedia(tweet),
quotedTweet: getQuotedTweet(tweet, fallbackUrl),
};
}
export function normalizeTitle(text: string, fallback: string): string {
const firstLine = text.split("\n")[0]?.trim();
if (!firstLine) {
return fallback;
}
return firstLine.slice(0, 120);
}
export function formatTweetAuthor(tweet: XTweet): string | undefined {
if (tweet.author && tweet.authorName) {
return `${tweet.authorName} (@${tweet.author})`;
}
if (tweet.author) {
return `@${tweet.author}`;
}
return tweet.authorName;
}
export function getTweetAuthorMetadata(tweet: XTweet): Record<string, unknown> {
return {
authorName: tweet.authorName,
authorUsername: tweet.author,
authorUrl: tweet.author ? `https://x.com/${tweet.author}` : undefined,
};
}
export function formatMediaList(media: XMedia[]): string[] {
return media.map((item) => {
if (item.type === "photo") {
return `photo: ${item.url}`;
}
return `${item.type}: ${item.url}`;
});
}
export function filterXGraphQlEntries(entries: NetworkEntry[]): NetworkEntry[] {
return entries.filter((entry) => entry.url.includes("/graphql/"));
}
@@ -0,0 +1,87 @@
import type { ExtractedDocument, ContentBlock } from "../../extract/document";
import { findTweetNode, formatMediaList, formatTweetAuthor, getTweetAuthorMetadata, normalizeTitle, toXTweet } from "./shared";
export function extractSingleTweetDocumentFromPayload(
payload: unknown,
statusId: string,
pageUrl: string,
): ExtractedDocument | null {
const tweet = findTweetNode(payload, statusId);
if (!tweet) {
return null;
}
const xTweet = toXTweet(tweet, pageUrl);
const content: ContentBlock[] = [];
if (xTweet.text) {
content.push({ type: "paragraph", text: xTweet.text });
}
for (const mediaLine of formatMediaList(xTweet.media)) {
if (mediaLine.startsWith("photo: ")) {
content.push({
type: "image",
url: mediaLine.slice("photo: ".length),
});
} else {
content.push({
type: "list",
ordered: false,
items: [mediaLine],
});
}
}
if (xTweet.quotedTweet) {
const quotedLines: string[] = [];
const quotedAuthor =
xTweet.quotedTweet.author && xTweet.quotedTweet.authorName
? `${xTweet.quotedTweet.authorName} (@${xTweet.quotedTweet.author})`
: xTweet.quotedTweet.author
? `@${xTweet.quotedTweet.author}`
: xTweet.quotedTweet.authorName;
if (quotedAuthor) {
quotedLines.push(quotedAuthor);
}
if (xTweet.quotedTweet.text) {
quotedLines.push(xTweet.quotedTweet.text);
}
quotedLines.push(...formatMediaList(xTweet.quotedTweet.media));
if (quotedLines.length > 0) {
content.push({ type: "heading", depth: 2, text: "Quoted Tweet" });
content.push({ type: "quote", text: quotedLines.join("\n\n") });
}
}
return {
url: pageUrl,
canonicalUrl: xTweet.url,
title: normalizeTitle(
xTweet.author ? `@${xTweet.author}: ${xTweet.text}` : xTweet.text,
"Tweet",
),
author: formatTweetAuthor(xTweet),
siteName: "X",
publishedAt: xTweet.createdAt,
summary: xTweet.text.slice(0, 200) || undefined,
adapter: "x",
metadata: {
kind: "x/post",
tweetId: xTweet.id,
...getTweetAuthorMetadata(xTweet),
conversationId:
typeof tweet.legacy === "object" &&
tweet.legacy !== null &&
typeof (tweet.legacy as Record<string, unknown>).conversation_id_str === "string"
? (tweet.legacy as Record<string, unknown>).conversation_id_str
: undefined,
favoriteCount: xTweet.likes,
replyCount: xTweet.replies,
retweetCount: xTweet.retweets,
},
content,
};
}
@@ -0,0 +1,286 @@
import type { AdapterContext } from "../types";
import { extractThreadTweetsFromPayloads } from "./thread";
import { collectXJsonPayloads, getRelevantXThreadEntries, prefetchRelevantXThreadBodies } from "./payloads";
interface ClickTextResult {
clicked: boolean;
text?: string;
}
interface ScrollStepResult {
moved: boolean;
atTop: boolean;
atBottom: boolean;
}
interface ThreadProgress {
tweetCount: number;
firstTweetId?: string;
lastTweetId?: string;
requestCount: number;
tweetDetailCount: number;
}
interface TopProbeState {
requestCount: number;
tweetDetailCount: number;
scrollHeight: number;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function waitForXNetworkSettle(context: AdapterContext, reason: string): Promise<void> {
try {
await context.network.waitForIdle({
idleMs: 650,
timeoutMs: Math.min(context.timeoutMs, 5_000),
});
} catch {
context.log.debug(`Network idle timed out after ${reason}.`);
}
}
async function captureTopProbeState(context: AdapterContext): Promise<TopProbeState> {
const entries = getRelevantXThreadEntries(context);
const scrollHeight = await context.browser.evaluate<number>(`
(() => {
const scrollRoot = document.scrollingElement ?? document.documentElement ?? document.body;
return scrollRoot.scrollHeight;
})()
`);
return {
requestCount: entries.length,
tweetDetailCount: entries.filter((entry) => entry.url.includes("TweetDetail")).length,
scrollHeight,
};
}
async function waitForTopProbe(context: AdapterContext): Promise<boolean> {
const initial = await captureTopProbeState(context);
const deadline = Date.now() + 1_200;
while (Date.now() < deadline) {
try {
await context.network.waitForIdle({
idleMs: 250,
timeoutMs: 350,
});
} catch {
// Keep polling until the shorter top-probe budget expires.
}
await prefetchRelevantXThreadBodies(context);
const next = await captureTopProbeState(context);
if (
next.requestCount > initial.requestCount ||
next.tweetDetailCount > initial.tweetDetailCount ||
next.scrollHeight > initial.scrollHeight + 4
) {
context.log.debug("Observed additional X thread activity while probing the page top.");
return true;
}
await sleep(120);
}
return false;
}
async function scrollThreadToTop(context: AdapterContext): Promise<void> {
let settledTopChecks = 0;
while (settledTopChecks < 2) {
const scroll = await context.browser.evaluate<ScrollStepResult>(`
(() => {
const scrollRoot = document.scrollingElement ?? document.documentElement ?? document.body;
const before = window.scrollY;
window.scrollTo({ top: 0, left: 0, behavior: "instant" });
const after = window.scrollY;
return {
moved: after !== before,
atTop: after <= 4,
atBottom: window.innerHeight + after >= scrollRoot.scrollHeight - 4,
};
})()
`);
await sleep(140);
await waitForXNetworkSettle(context, "scrolling X thread to top");
await prefetchRelevantXThreadBodies(context);
if (scroll.moved) {
settledTopChecks = 0;
continue;
}
const observedTopActivity = await waitForTopProbe(context);
if (observedTopActivity) {
settledTopChecks = 0;
continue;
}
settledTopChecks += 1;
}
}
async function clickVisibleShowReplies(context: AdapterContext): Promise<ClickTextResult> {
return context.browser.evaluate<ClickTextResult>(`
(() => {
const normalize = (value) => value.replace(/\\s+/g, " ").trim();
const matches = [
/^Show replies$/i,
/^Show more replies$/i,
/^Show additional replies$/i,
/^显示回复$/,
/^展开回复$/,
];
const isVisible = (element) => {
if (!(element instanceof HTMLElement)) {
return false;
}
const rect = element.getBoundingClientRect();
const style = window.getComputedStyle(element);
return (
rect.width > 0 &&
rect.height > 0 &&
style.visibility !== "hidden" &&
style.display !== "none"
);
};
const selectors = [
"a",
"button",
'[role="button"]',
'[role="link"]',
];
for (const element of document.querySelectorAll(selectors.join(","))) {
if (!isVisible(element)) {
continue;
}
const text = normalize(element.textContent ?? "");
if (!text || !matches.some((pattern) => pattern.test(text))) {
continue;
}
element.scrollIntoView({ block: "center", inline: "nearest" });
if (element instanceof HTMLElement) {
element.click();
return { clicked: true, text };
}
}
return { clicked: false };
})()
`);
}
async function expandVisibleShowReplies(context: AdapterContext): Promise<number> {
let clickCount = 0;
while (clickCount < 8) {
const result = await clickVisibleShowReplies(context).catch<ClickTextResult>(() => ({ clicked: false }));
if (!result.clicked) {
break;
}
clickCount += 1;
context.log.debug(`Expanded X thread replies via "${result.text ?? "Show replies"}".`);
await sleep(250);
await waitForXNetworkSettle(context, "expanding Show replies");
await prefetchRelevantXThreadBodies(context);
}
return clickCount;
}
async function scrollThreadBy(context: AdapterContext, stepPx: number): Promise<ScrollStepResult> {
const result = await context.browser.evaluate<ScrollStepResult>(`
(() => {
const scrollRoot = document.scrollingElement ?? document.documentElement ?? document.body;
const before = window.scrollY;
window.scrollBy({ top: ${stepPx}, left: 0, behavior: "instant" });
const after = window.scrollY;
return {
moved: after !== before,
atTop: after <= 4,
atBottom: window.innerHeight + after >= scrollRoot.scrollHeight - 4,
};
})()
`);
await sleep(140);
await waitForXNetworkSettle(context, "scrolling X thread");
await prefetchRelevantXThreadBodies(context);
return result;
}
async function captureThreadProgress(context: AdapterContext, statusId: string): Promise<ThreadProgress> {
const entries = getRelevantXThreadEntries(context);
const payloads = await collectXJsonPayloads(context);
const tweets = extractThreadTweetsFromPayloads(payloads, statusId, context.input.url.toString());
return {
tweetCount: tweets.length,
firstTweetId: tweets[0]?.id,
lastTweetId: tweets[tweets.length - 1]?.id,
requestCount: entries.length,
tweetDetailCount: entries.filter((entry) => entry.url.includes("TweetDetail")).length,
};
}
export async function loadFullXThread(context: AdapterContext, statusId: string): Promise<void> {
await scrollThreadToTop(context);
let progress = await captureThreadProgress(context, statusId);
let stagnantRounds = 0;
let roundsWithoutMovement = 0;
let distanceWithoutThreadActivityPx = 0;
for (let round = 0; ; round += 1) {
const stepPx = round < 12 ? 1_200 : 1_600;
let expandedCount = await expandVisibleShowReplies(context);
const scroll = await scrollThreadBy(context, stepPx);
expandedCount += await expandVisibleShowReplies(context);
const nextProgress = await captureThreadProgress(context, statusId);
const grew =
nextProgress.tweetCount > progress.tweetCount ||
nextProgress.firstTweetId !== progress.firstTweetId ||
nextProgress.lastTweetId !== progress.lastTweetId ||
nextProgress.requestCount > progress.requestCount ||
nextProgress.tweetDetailCount > progress.tweetDetailCount;
if (grew) {
context.log.debug(
`X thread progress: ${nextProgress.tweetCount} tweets (${nextProgress.firstTweetId ?? "unknown"} -> ${nextProgress.lastTweetId ?? "unknown"}), ${nextProgress.requestCount} requests, ${nextProgress.tweetDetailCount} TweetDetail.`,
);
stagnantRounds = 0;
distanceWithoutThreadActivityPx = 0;
} else if (expandedCount > 0) {
stagnantRounds = 0;
distanceWithoutThreadActivityPx = 0;
} else {
stagnantRounds += 1;
distanceWithoutThreadActivityPx += stepPx;
}
roundsWithoutMovement = scroll.moved ? 0 : roundsWithoutMovement + 1;
progress = nextProgress;
if (scroll.atBottom && stagnantRounds >= 6) {
context.log.debug("Stopping X thread scroll after reaching page bottom with no further thread progress.");
break;
}
if (roundsWithoutMovement >= 2 && stagnantRounds >= 4) {
context.log.debug("Stopping X thread scroll after repeated downward scrolls no longer move the page.");
break;
}
if (distanceWithoutThreadActivityPx >= 24_000 && stagnantRounds >= 12) {
context.log.debug("Stopping X thread scroll after a long stretch with no thread-related progress.");
break;
}
}
}
@@ -0,0 +1,316 @@
import type { ExtractedDocument } from "../../extract/document";
import {
formatMediaList,
formatTweetAuthor,
getLegacy,
getTweetAuthorMetadata,
isRecord,
normalizeTitle,
toXTweet,
unwrapTweetResult,
} from "./shared";
import type { JsonObject, XQuotedTweet, XTweet } from "./types";
interface ParsedThreadTweet extends XTweet {
userId?: string;
conversationId?: string;
inReplyToUserId?: string;
sortTimestamp: number;
}
function compareTweetIds(left: string, right: string): number {
try {
const leftId = BigInt(left);
const rightId = BigInt(right);
if (leftId === rightId) {
return 0;
}
return leftId < rightId ? -1 : 1;
} catch {
return left.localeCompare(right);
}
}
function toTimestamp(value: string | undefined): number {
if (!value) {
return 0;
}
const parsed = Date.parse(value);
return Number.isNaN(parsed) ? 0 : parsed;
}
function scoreParsedTweet(tweet: ParsedThreadTweet): number {
return (
(tweet.text ? 4 : 0) +
(tweet.author ? 2 : 0) +
(tweet.authorName ? 2 : 0) +
(tweet.media.length > 0 ? 1 : 0)
);
}
function toParsedThreadTweet(tweet: JsonObject, pageUrl: string): ParsedThreadTweet {
const legacy = getLegacy(tweet);
const xTweet = toXTweet(tweet, pageUrl);
return {
...xTweet,
userId: typeof legacy.user_id_str === "string" ? legacy.user_id_str : undefined,
conversationId: typeof legacy.conversation_id_str === "string" ? legacy.conversation_id_str : undefined,
inReplyToUserId: typeof legacy.in_reply_to_user_id_str === "string" ? legacy.in_reply_to_user_id_str : undefined,
sortTimestamp: toTimestamp(xTweet.createdAt),
};
}
function collectTweetFromItemContent(
itemContent: unknown,
pageUrl: string,
tweets: Map<string, ParsedThreadTweet>,
): void {
if (!isRecord(itemContent)) {
return;
}
const tweet = unwrapTweetResult(
isRecord(itemContent.tweet_results) ? itemContent.tweet_results.result : null,
);
if (!tweet || typeof tweet.rest_id !== "string") {
return;
}
const parsed = toParsedThreadTweet(tweet, pageUrl);
const existing = tweets.get(parsed.id);
if (!existing || scoreParsedTweet(parsed) >= scoreParsedTweet(existing)) {
tweets.set(parsed.id, parsed);
}
}
function collectTweetsFromItems(
items: unknown,
pageUrl: string,
tweets: Map<string, ParsedThreadTweet>,
): void {
if (!Array.isArray(items)) {
return;
}
for (const item of items) {
if (!isRecord(item)) {
continue;
}
if (isRecord(item.item) && isRecord(item.item.itemContent)) {
collectTweetFromItemContent(item.item.itemContent, pageUrl, tweets);
continue;
}
if (isRecord(item.itemContent)) {
collectTweetFromItemContent(item.itemContent, pageUrl, tweets);
}
}
}
function getInstructions(payload: unknown): unknown[] {
if (!isRecord(payload) || !isRecord(payload.data)) {
return [];
}
const { data } = payload;
return (
(isRecord(data.threaded_conversation_with_injections_v2) &&
Array.isArray(data.threaded_conversation_with_injections_v2.instructions)
? data.threaded_conversation_with_injections_v2.instructions
: undefined) ??
(isRecord(data.threaded_conversation_with_injections) &&
Array.isArray(data.threaded_conversation_with_injections.instructions)
? data.threaded_conversation_with_injections.instructions
: undefined) ??
(isRecord(data.tweetResult) &&
isRecord(data.tweetResult.result) &&
isRecord(data.tweetResult.result.timeline) &&
Array.isArray(data.tweetResult.result.timeline.instructions)
? data.tweetResult.result.timeline.instructions
: [])
);
}
function parseTweetDetailPayload(payload: unknown, pageUrl: string): ParsedThreadTweet[] {
const tweets = new Map<string, ParsedThreadTweet>();
const instructions = getInstructions(payload);
for (const instruction of instructions) {
if (!isRecord(instruction)) {
continue;
}
collectTweetsFromItems(instruction.moduleItems, pageUrl, tweets);
if (!Array.isArray(instruction.entries)) {
continue;
}
for (const entry of instruction.entries) {
if (!isRecord(entry)) {
continue;
}
const content = isRecord(entry.content) ? entry.content : {};
collectTweetFromItemContent(content.itemContent, pageUrl, tweets);
collectTweetsFromItems(content.items, pageUrl, tweets);
}
}
return Array.from(tweets.values());
}
function buildContinuousThread(tweets: ParsedThreadTweet[], statusId: string): ParsedThreadTweet[] {
const byId = new Map<string, ParsedThreadTweet>();
for (const tweet of tweets) {
const existing = byId.get(tweet.id);
if (!existing || scoreParsedTweet(tweet) >= scoreParsedTweet(existing)) {
byId.set(tweet.id, tweet);
}
}
const rootTweet = byId.get(statusId);
if (!rootTweet?.userId || !rootTweet.conversationId) {
return [];
}
const candidates = Array.from(byId.values()).filter(
(tweet) =>
tweet.id === statusId ||
(tweet.userId === rootTweet.userId && tweet.conversationId === rootTweet.conversationId),
);
const repliesByParent = new Map<string, ParsedThreadTweet[]>();
for (const tweet of candidates) {
if (!tweet.inReplyTo || tweet.id === statusId) {
continue;
}
const bucket = repliesByParent.get(tweet.inReplyTo) ?? [];
bucket.push(tweet);
bucket.sort((left, right) => {
if (left.sortTimestamp !== right.sortTimestamp) {
return left.sortTimestamp - right.sortTimestamp;
}
return compareTweetIds(left.id, right.id);
});
repliesByParent.set(tweet.inReplyTo, bucket);
}
const ancestorPath: ParsedThreadTweet[] = [rootTweet];
const ancestorSeen = new Set<string>([rootTweet.id]);
let currentAncestor = rootTweet;
while (currentAncestor.inReplyTo) {
const parent = byId.get(currentAncestor.inReplyTo);
if (!parent || ancestorSeen.has(parent.id)) {
break;
}
ancestorPath.unshift(parent);
ancestorSeen.add(parent.id);
currentAncestor = parent;
}
const chain = ancestorPath.slice();
const seen = new Set<string>(chain.map((tweet) => tweet.id));
let currentId = rootTweet.id;
while (true) {
const next = (repliesByParent.get(currentId) ?? []).find((tweet) => !seen.has(tweet.id));
if (!next) {
break;
}
chain.push(next);
seen.add(next.id);
currentId = next.id;
}
return chain;
}
export function extractThreadTweetsFromPayloads(
payloads: unknown[],
statusId: string,
pageUrl: string,
): XTweet[] {
const parsedTweets: ParsedThreadTweet[] = [];
for (const payload of payloads) {
parsedTweets.push(...parseTweetDetailPayload(payload, pageUrl));
}
return buildContinuousThread(parsedTweets, statusId).map(({ sortTimestamp: _sortTimestamp, ...tweet }) => tweet);
}
function buildQuotedTweetMarkdown(quotedTweet: XQuotedTweet): string {
const author = quotedTweet.author ? `@${quotedTweet.author}` : "Unknown";
const name = quotedTweet.authorName ? `${quotedTweet.authorName} ` : "";
const lines: string[] = [`Quoted Tweet${quotedTweet.author || quotedTweet.authorName ? `: ${name}${author}`.trim() : ""}`];
if (quotedTweet.text) {
lines.push(...quotedTweet.text.split("\n"));
}
for (const mediaLine of formatMediaList(quotedTweet.media)) {
lines.push(mediaLine);
}
return lines.map((line) => (line ? `> ${line}` : ">")).join("\n");
}
function buildThreadMarkdown(tweets: XTweet[]): string {
return tweets
.map((tweet, index) => {
const lines: string[] = [];
const author = tweet.author ? `@${tweet.author}` : "Unknown";
const name = tweet.authorName ? `${tweet.authorName} ` : "";
lines.push(`## ${index + 1}. ${name}${author}`.trim());
if (tweet.createdAt) {
lines.push(`_Published: ${tweet.createdAt}_`);
}
lines.push(tweet.text || "(No text)");
const mediaLines = formatMediaList(tweet.media);
if (mediaLines.length > 0) {
lines.push(mediaLines.map((line) => `- ${line}`).join("\n"));
}
if (tweet.quotedTweet) {
lines.push(buildQuotedTweetMarkdown(tweet.quotedTweet));
}
return lines.join("\n\n");
})
.join("\n\n");
}
export function extractThreadDocumentFromPayloads(
payloads: unknown[],
statusId: string,
pageUrl: string,
): ExtractedDocument | null {
const tweets = extractThreadTweetsFromPayloads(payloads, statusId, pageUrl);
if (tweets.length <= 1) {
return null;
}
const rootTweet = tweets[0];
const rootAuthor = formatTweetAuthor(rootTweet);
return {
url: pageUrl,
canonicalUrl: rootTweet.url,
title: normalizeTitle(rootTweet.text, "X Thread"),
author: rootAuthor,
siteName: "X",
publishedAt: rootTweet.createdAt,
summary: rootTweet.text.slice(0, 200) || undefined,
adapter: "x",
metadata: {
kind: "x/thread",
tweetId: rootTweet.id,
tweetCount: tweets.length,
lastTweetId: tweets[tweets.length - 1]?.id,
...getTweetAuthorMetadata(rootTweet),
},
content: [{ type: "markdown", markdown: buildThreadMarkdown(tweets) }],
};
}
@@ -0,0 +1,36 @@
export type JsonObject = Record<string, unknown>;
export interface XUser {
name?: string;
screenName?: string;
}
export interface XMedia {
type: string;
url: string;
alt?: string;
}
export interface XQuotedTweet {
id: string;
author?: string;
authorName?: string;
text: string;
url: string;
media: XMedia[];
}
export interface XTweet {
id: string;
author?: string;
authorName?: string;
text: string;
likes: number;
retweets: number;
replies: number;
createdAt?: string;
inReplyTo?: string;
url: string;
media: XMedia[];
quotedTweet?: XQuotedTweet;
}
@@ -0,0 +1,33 @@
import type { Adapter } from "../types";
import { collectMediaFromDocument } from "../../media/markdown-media";
import { extractYouTubeTranscriptDocument } from "./transcript";
import { isYouTubeHost, parseYouTubeVideoId } from "./utils";
export const youtubeAdapter: Adapter = {
name: "youtube",
match(input) {
return isYouTubeHost(input.url.hostname);
},
async process(context) {
const videoId = parseYouTubeVideoId(context.input.url);
if (!videoId) {
return {
status: "no_document",
};
}
context.log.info(`Loading ${context.input.url.toString()} with youtube adapter`);
const document = await extractYouTubeTranscriptDocument(context, videoId);
if (!document) {
return {
status: "no_document",
};
}
return {
status: "ok",
document,
media: collectMediaFromDocument(document),
};
},
};
@@ -0,0 +1,392 @@
import type { ExtractedDocument } from "../../extract/document";
import { detectInteractionGate } from "../../browser/interaction-gates";
import {
buildYouTubeThumbnailCandidates,
parseYouTubeDescriptionChapters,
renderYouTubeTranscriptMarkdown,
type YouTubeChapter,
type YouTubeTranscriptSegment,
} from "./utils";
interface CaptionInfo {
captionUrl: string;
language: string;
kind: string;
available: string[];
title?: string;
author?: string;
authorUrl?: string;
channelId?: string;
description?: string;
publishedAt?: string;
viewCount?: number;
durationSeconds?: number;
keywords: string[];
category?: string;
isLiveContent?: boolean;
coverImages: string[];
}
function normalizeUrl(url: string | undefined): string | undefined {
if (!url) {
return undefined;
}
try {
const parsed = new URL(url);
if (parsed.protocol === "http:") {
parsed.protocol = "https:";
}
return parsed.toString();
} catch {
return url;
}
}
function buildSummary(description: string | undefined, segments: YouTubeTranscriptSegment[]): string | undefined {
const descriptionSummary = description
?.replace(/\r\n/g, "\n")
.split("\n")
.map((line) => line.trim())
.find((line) => line && !/^https?:\/\//i.test(line));
if (descriptionSummary) {
return descriptionSummary.slice(0, 240);
}
const transcriptSummary = segments
.slice(0, 8)
.map((segment) => segment.text)
.join(" ")
.slice(0, 240)
.trim();
return transcriptSummary || undefined;
}
async function canFetchThumbnail(url: string): Promise<boolean> {
try {
const response = await fetch(url, { method: "HEAD", redirect: "follow" });
if (response.ok) {
return true;
}
if (response.status === 405) {
const fallbackResponse = await fetch(url, {
method: "GET",
headers: { Range: "bytes=0-0" },
redirect: "follow",
});
return fallbackResponse.ok;
}
} catch {
return false;
}
return false;
}
async function resolveBestCoverImage(videoId: string, coverImages: string[]): Promise<string | undefined> {
const candidates = buildYouTubeThumbnailCandidates(videoId, coverImages);
for (const candidate of candidates) {
if (await canFetchThumbnail(candidate)) {
return candidate;
}
}
return candidates[0];
}
export async function extractYouTubeTranscriptDocument(
context: Parameters<import("../types").Adapter["process"]>[0],
videoId: string,
): Promise<ExtractedDocument | null> {
const videoUrl = `https://www.youtube.com/watch?v=${videoId}`;
await context.browser.goto(videoUrl, context.timeoutMs);
const interaction = await detectInteractionGate(context.browser);
if (interaction) {
context.log.debug(`Interaction gate detected on YouTube: ${interaction.provider}`);
return null;
}
try {
await context.network.waitForIdle({
idleMs: 1_000,
timeoutMs: Math.min(context.timeoutMs, 8_000),
});
} catch {
context.log.debug("Network idle timed out on YouTube load.");
}
const captionInfo = await context.browser.evaluate<CaptionInfo | { error: string }>(`
(async () => {
function readText(value) {
if (!value) return undefined;
if (typeof value === 'string') {
const text = value.trim();
return text || undefined;
}
if (typeof value.simpleText === 'string') {
const text = value.simpleText.trim();
return text || undefined;
}
if (Array.isArray(value.runs)) {
const text = value.runs
.map((run) => typeof run?.text === 'string' ? run.text : '')
.join('')
.trim();
return text || undefined;
}
return undefined;
}
function parsePositiveInteger(value) {
if (typeof value === 'number' && Number.isFinite(value) && value >= 0) {
return Math.floor(value);
}
if (typeof value !== 'string') {
return undefined;
}
const normalized = value.replace(/[^\\d]/g, '');
if (!normalized) {
return undefined;
}
const parsed = Number.parseInt(normalized, 10);
return Number.isFinite(parsed) ? parsed : undefined;
}
const apiKey = window.ytcfg?.data_?.INNERTUBE_API_KEY;
const playerResponse = window.ytInitialPlayerResponse;
const videoDetails = playerResponse?.videoDetails || {};
const microformat = playerResponse?.microformat?.playerMicroformatRenderer || {};
const title =
videoDetails.title ||
readText(microformat.title) ||
document.title.replace(/ - YouTube$/, '').trim();
const author =
videoDetails.author ||
microformat.ownerChannelName ||
document.querySelector('link[itemprop="name"]')?.getAttribute('content') ||
undefined;
const authorUrl =
microformat.ownerProfileUrl ||
(typeof videoDetails.channelId === 'string' && videoDetails.channelId
? 'https://www.youtube.com/channel/' + videoDetails.channelId
: undefined);
const description =
readText(microformat.description) ||
(typeof videoDetails.shortDescription === 'string' ? videoDetails.shortDescription.trim() : undefined);
const keywords = Array.isArray(videoDetails.keywords)
? videoDetails.keywords.filter((keyword) => typeof keyword === 'string' && keyword.trim())
: [];
const thumbnails = [
...(Array.isArray(videoDetails.thumbnail?.thumbnails) ? videoDetails.thumbnail.thumbnails : []),
...(Array.isArray(microformat.thumbnail?.thumbnails) ? microformat.thumbnail.thumbnails : []),
]
.filter((thumbnail) => typeof thumbnail?.url === 'string' && thumbnail.url)
.sort((left, right) => ((right?.width || 0) * (right?.height || 0)) - ((left?.width || 0) * (left?.height || 0)))
.map((thumbnail) => thumbnail.url);
if (!apiKey) {
return { error: 'INNERTUBE_API_KEY not found on page' };
}
const response = await fetch('/youtubei/v1/player?key=' + apiKey + '&prettyPrint=false', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
context: { client: { clientName: 'ANDROID', clientVersion: '20.10.38' } },
videoId: ${JSON.stringify(videoId)}
})
});
if (!response.ok) {
return { error: 'InnerTube player API returned HTTP ' + response.status };
}
const data = await response.json();
const renderer = data.captions?.playerCaptionsTracklistRenderer;
if (!renderer?.captionTracks?.length) {
return { error: 'No captions available for this video' };
}
const tracks = renderer.captionTracks;
const track = tracks.find((item) => item.kind !== 'asr') || tracks[0];
return {
captionUrl: track.baseUrl,
language: track.languageCode,
kind: track.kind || 'manual',
available: tracks.map((item) => {
const languageLabel = readText(item.name) || item.languageCode;
return item.kind === 'asr'
? languageLabel + ' [' + item.languageCode + ', auto]'
: languageLabel + ' [' + item.languageCode + ']';
}),
title,
author,
authorUrl,
channelId: typeof videoDetails.channelId === 'string' ? videoDetails.channelId : undefined,
description,
publishedAt:
(typeof microformat.publishDate === 'string' && microformat.publishDate) ||
(typeof microformat.uploadDate === 'string' && microformat.uploadDate) ||
document.querySelector('meta[itemprop="datePublished"]')?.getAttribute('content') ||
undefined,
viewCount: parsePositiveInteger(videoDetails.viewCount) ?? parsePositiveInteger(microformat.viewCount),
durationSeconds: parsePositiveInteger(videoDetails.lengthSeconds),
keywords,
category: typeof microformat.category === 'string' ? microformat.category : undefined,
isLiveContent: Boolean(videoDetails.isLiveContent || microformat.isLiveContent),
coverImages: thumbnails,
};
})()
`);
if ("error" in captionInfo) {
context.log.debug(`YouTube transcript unavailable: ${captionInfo.error}`);
return null;
}
const segments = await context.browser.evaluate<YouTubeTranscriptSegment[] | { error: string }>(`
(async () => {
const response = await fetch(${JSON.stringify(captionInfo.captionUrl)});
const xml = await response.text();
if (!xml) {
return { error: 'Caption XML is empty' };
}
function getAttr(tag, name) {
const needle = name + '="';
const index = tag.indexOf(needle);
if (index === -1) return '';
const valueStart = index + needle.length;
const valueEnd = tag.indexOf('"', valueStart);
if (valueEnd === -1) return '';
return tag.substring(valueStart, valueEnd);
}
function decodeEntities(value) {
return value
.replaceAll('&amp;', '&')
.replaceAll('&lt;', '<')
.replaceAll('&gt;', '>')
.replaceAll('&quot;', '"')
.replaceAll('&#39;', "'");
}
const marker = xml.includes('<p t="') ? '<p ' : '<text ';
const endMarker = marker === '<p ' ? '</p>' : '</text>';
const results = [];
let position = 0;
while (true) {
const tagStart = xml.indexOf(marker, position);
if (tagStart === -1) break;
let contentStart = xml.indexOf('>', tagStart);
if (contentStart === -1) break;
contentStart += 1;
const tagEnd = xml.indexOf(endMarker, contentStart);
if (tagEnd === -1) break;
const attrString = xml.substring(tagStart + marker.length, contentStart - 1);
const content = xml.substring(contentStart, tagEnd);
const start = marker === '<p '
? (parseFloat(getAttr(attrString, 't')) || 0) / 1000
: (parseFloat(getAttr(attrString, 'start')) || 0);
const duration = marker === '<p '
? (parseFloat(getAttr(attrString, 'd')) || 0) / 1000
: (parseFloat(getAttr(attrString, 'dur')) || 0);
const text = decodeEntities(content.replace(/<[^>]+>/g, '')).split('\\n').join(' ').trim();
if (text) {
results.push({ start, end: start + duration, text });
}
position = tagEnd + endMarker.length;
}
if (results.length === 0) {
return { error: 'Parsed 0 transcript segments' };
}
return results;
})()
`);
if (!Array.isArray(segments) || segments.length === 0) {
context.log.debug("Parsed no YouTube transcript segments.");
return null;
}
const extractedChapters = await context.browser.evaluate<YouTubeChapter[]>(`
(() => {
const data = window.ytInitialData;
const markers = data?.playerOverlays?.playerOverlayRenderer
?.decoratedPlayerBarRenderer?.decoratedPlayerBarRenderer
?.playerBar?.multiMarkersPlayerBarRenderer?.markersMap || [];
const results = [];
for (const marker of markers) {
const chapters = marker?.value?.chapters;
if (!Array.isArray(chapters)) continue;
for (const chapter of chapters) {
const renderer = chapter?.chapterRenderer;
const title = renderer?.title?.simpleText;
const timeRangeStartMillis = renderer?.timeRangeStartMillis;
if (title && typeof timeRangeStartMillis === 'number') {
results.push({ title, time: Math.floor(timeRangeStartMillis / 1000) });
}
}
}
return results;
})()
`).catch(() => []);
const descriptionChapters = parseYouTubeDescriptionChapters(captionInfo.description);
const chapters = extractedChapters.length > 0 ? extractedChapters : descriptionChapters;
const markdown = renderYouTubeTranscriptMarkdown({
description: captionInfo.description,
segments,
chapters,
});
if (!markdown) {
return null;
}
const pageUrl = await context.browser.getURL();
const coverImage = await resolveBestCoverImage(videoId, captionInfo.coverImages);
const summary = buildSummary(captionInfo.description, segments);
return {
url: pageUrl,
canonicalUrl: pageUrl,
title: captionInfo.title || "YouTube Transcript",
author: captionInfo.author,
publishedAt: captionInfo.publishedAt,
siteName: "YouTube",
summary,
adapter: "youtube",
metadata: {
kind: "youtube/transcript",
videoId,
authorUrl: normalizeUrl(captionInfo.authorUrl),
channelId: captionInfo.channelId,
coverImage,
description: captionInfo.description,
durationSeconds: captionInfo.durationSeconds,
language: captionInfo.language,
captionKind: captionInfo.kind,
availableLanguages: captionInfo.available,
viewCount: captionInfo.viewCount,
keywords: captionInfo.keywords,
category: captionInfo.category,
isLiveContent: captionInfo.isLiveContent,
chapterCount: chapters.length,
},
content: [{ type: "markdown", markdown }],
};
}
@@ -0,0 +1,253 @@
export interface YouTubeTranscriptSegment {
start: number;
end: number;
text: string;
}
export interface YouTubeChapter {
title: string;
time: number;
}
interface RenderYouTubeTranscriptMarkdownInput {
description?: string;
segments: YouTubeTranscriptSegment[];
chapters: YouTubeChapter[];
}
const DESCRIPTION_CHAPTER_RE = /^((?:\d{1,2}:)?\d{1,2}:\d{2})(?:\s+[-|:]\s+|\s+)(.+)$/;
const YOUTUBE_THUMBNAIL_VARIANTS = [
"maxresdefault.jpg",
"sddefault.jpg",
"hqdefault.jpg",
"mqdefault.jpg",
"default.jpg",
];
export function isYouTubeHost(hostname: string): boolean {
return [
"youtube.com",
"www.youtube.com",
"m.youtube.com",
"youtu.be",
].includes(hostname);
}
export function parseYouTubeVideoId(url: URL): string | null {
if (url.hostname === "youtu.be") {
return url.pathname.split("/").filter(Boolean)[0] ?? null;
}
if (url.pathname === "/watch") {
return url.searchParams.get("v");
}
const shortsMatch = url.pathname.match(/^\/shorts\/([^/?#]+)/);
if (shortsMatch) {
return shortsMatch[1];
}
const liveMatch = url.pathname.match(/^\/live\/([^/?#]+)/);
if (liveMatch) {
return liveMatch[1];
}
return null;
}
function parseTimestampValue(raw: string): number | null {
const parts = raw
.split(":")
.map((part) => Number.parseInt(part, 10))
.filter((part) => Number.isFinite(part));
if (parts.length < 2 || parts.length > 3) {
return null;
}
if (parts.some((part) => part < 0)) {
return null;
}
if (parts.length === 2) {
const [minutes, seconds] = parts;
return minutes * 60 + seconds;
}
const [hours, minutes, seconds] = parts;
return hours * 3600 + minutes * 60 + seconds;
}
export function formatTimestamp(totalSeconds: number): string {
const rounded = Math.max(0, Math.floor(totalSeconds));
const hours = Math.floor(rounded / 3600);
const minutes = Math.floor((rounded % 3600) / 60);
const seconds = rounded % 60;
if (hours > 0) {
return `${hours}:${String(minutes).padStart(2, "0")}:${String(seconds).padStart(2, "0")}`;
}
return `${minutes}:${String(seconds).padStart(2, "0")}`;
}
export function formatTimestampRange(start: number, end: number): string {
const safeStart = Math.max(0, start);
const safeEnd = Math.max(safeStart, end);
return `[${formatTimestamp(safeStart)} -> ${formatTimestamp(safeEnd)}]`;
}
export function normalizeYouTubeChapters(chapters: YouTubeChapter[]): YouTubeChapter[] {
const seenTimes = new Set<number>();
return chapters
.map((chapter) => ({
title: chapter.title.trim(),
time: Math.max(0, Math.floor(chapter.time)),
}))
.filter((chapter) => chapter.title)
.sort((left, right) => left.time - right.time)
.filter((chapter) => {
if (seenTimes.has(chapter.time)) {
return false;
}
seenTimes.add(chapter.time);
return true;
});
}
export function parseYouTubeDescriptionChapters(description?: string | null): YouTubeChapter[] {
if (!description) {
return [];
}
const chapters: YouTubeChapter[] = [];
const seen = new Set<string>();
for (const rawLine of description.replace(/\r\n/g, "\n").split("\n")) {
const line = rawLine.trim();
if (!line) {
continue;
}
const match = line.match(DESCRIPTION_CHAPTER_RE);
if (!match) {
continue;
}
const time = parseTimestampValue(match[1]);
const title = match[2]?.trim();
if (time === null || !title) {
continue;
}
const key = `${time}:${title.toLowerCase()}`;
if (seen.has(key)) {
continue;
}
seen.add(key);
chapters.push({ title, time });
}
const normalized = normalizeYouTubeChapters(chapters);
if (normalized.length >= 2) {
return normalized;
}
if (normalized.length === 1 && normalized[0]?.time === 0) {
return normalized;
}
return [];
}
function renderDescriptionMarkdown(description: string): string {
return description
.replace(/\r\n/g, "\n")
.trim()
.split(/\n{2,}/)
.map((block) => block.split("\n").map((line) => line.trimEnd()).join(" \n"))
.join("\n\n")
.trim();
}
function renderSegmentLine(segment: YouTubeTranscriptSegment): string {
return `${formatTimestampRange(segment.start, segment.end)} ${segment.text}`;
}
export function renderYouTubeTranscriptMarkdown({
description,
segments,
chapters,
}: RenderYouTubeTranscriptMarkdownInput): string {
if (segments.length === 0) {
return "";
}
const parts: string[] = [];
const normalizedDescription = description?.trim();
const transcriptEnd = segments.reduce((maxEnd, segment) => Math.max(maxEnd, segment.end, segment.start), 0);
const normalizedChapters = normalizeYouTubeChapters(chapters).filter(
(chapter) => transcriptEnd <= 0 || chapter.time < transcriptEnd,
);
if (normalizedDescription) {
parts.push("## Description");
parts.push(renderDescriptionMarkdown(normalizedDescription));
}
if (normalizedChapters.length > 0) {
parts.push("## Chapters");
for (let index = 0; index < normalizedChapters.length; index += 1) {
const chapter = normalizedChapters[index];
const nextChapter = normalizedChapters[index + 1];
const chapterEnd = nextChapter ? nextChapter.time : transcriptEnd;
const chapterSegments = segments.filter(
(segment) => segment.start >= chapter.time && segment.start < chapterEnd,
);
parts.push(`### ${chapter.title} ${formatTimestampRange(chapter.time, chapterEnd)}`);
if (chapterSegments.length > 0) {
parts.push(chapterSegments.map(renderSegmentLine).join("\n"));
}
}
} else {
parts.push("## Transcript");
parts.push(segments.map(renderSegmentLine).join("\n"));
}
return parts.filter(Boolean).join("\n\n").trim();
}
function normalizeThumbnailKey(url: string): string {
try {
const parsed = new URL(url);
return `${parsed.origin}${parsed.pathname}`;
} catch {
return url;
}
}
export function buildYouTubeThumbnailCandidates(videoId: string, listedUrls: string[]): string[] {
const candidates = [
...YOUTUBE_THUMBNAIL_VARIANTS.map((variant) => `https://i.ytimg.com/vi/${videoId}/${variant}`),
...listedUrls,
];
const seen = new Set<string>();
return candidates.filter((candidate) => {
if (!candidate) {
return false;
}
const key = normalizeThumbnailKey(candidate);
if (seen.has(key)) {
return false;
}
seen.add(key);
return true;
});
}
@@ -0,0 +1,258 @@
import { EventEmitter } from "node:events";
import WebSocket from "ws";
type JsonObject = Record<string, unknown>;
interface CdpPendingCommand {
resolve(value: unknown): void;
reject(error: unknown): void;
method: string;
}
interface CdpErrorShape {
message?: string;
}
interface CdpCommandResult<T> {
result?: T;
error?: CdpErrorShape;
}
interface CreatePageSessionOptions {
initialUrl?: string;
visible?: boolean;
}
export class TargetSession extends EventEmitter {
constructor(
private readonly client: CdpClient,
public readonly targetId: string,
public readonly sessionId: string,
) {
super();
}
async send<T>(method: string, params: JsonObject = {}): Promise<T> {
return this.client.sendSessionCommand<T>(this.sessionId, method, params);
}
handleEvent(method: string, params: JsonObject): void {
this.emit(method, params);
this.emit("event", { method, params });
}
async waitForEvent<T extends JsonObject>(
method: string,
predicate?: (params: T) => boolean,
timeoutMs = 30_000,
): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timeout = setTimeout(() => {
this.off(method, listener);
reject(new Error(`Timed out waiting for ${method}`));
}, timeoutMs);
const listener = (params: T): void => {
if (predicate && !predicate(params)) {
return;
}
clearTimeout(timeout);
this.off(method, listener);
resolve(params);
};
this.on(method, listener);
});
}
}
export class CdpClient {
private readonly ws: WebSocket;
private readonly pending = new Map<number, CdpPendingCommand>();
private readonly sessions = new Map<string, TargetSession>();
private nextId = 1;
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.on("message", (raw) => {
this.handleMessage(raw.toString());
});
}
static async connect(browserWsUrl: string): Promise<CdpClient> {
const ws = await new Promise<WebSocket>((resolve, reject) => {
const socket = new WebSocket(browserWsUrl);
socket.once("open", () => resolve(socket));
socket.once("error", (error) => reject(error));
});
return new CdpClient(ws);
}
private handleMessage(rawMessage: string): void {
const message = JSON.parse(rawMessage) as {
id?: number;
sessionId?: string;
method?: string;
params?: JsonObject;
result?: unknown;
error?: CdpErrorShape;
};
if (typeof message.id === "number") {
const pending = this.pending.get(message.id);
if (!pending) {
return;
}
this.pending.delete(message.id);
if (message.error) {
pending.reject(new Error(`${pending.method}: ${message.error.message ?? "Unknown CDP error"}`));
return;
}
pending.resolve(message.result);
return;
}
if (typeof message.sessionId === "string" && typeof message.method === "string") {
const session = this.sessions.get(message.sessionId);
if (session) {
session.handleEvent(message.method, (message.params ?? {}) as JsonObject);
}
}
}
private async sendCommand<T>(
method: string,
params: JsonObject = {},
sessionId?: string,
): Promise<T> {
const id = this.nextId;
this.nextId += 1;
const payload = sessionId ? { id, method, params, sessionId } : { id, method, params };
const result = new Promise<T>((resolve, reject) => {
this.pending.set(id, {
resolve: (value) => resolve(value as T),
reject,
method,
});
});
this.ws.send(JSON.stringify(payload));
return result;
}
async sendBrowserCommand<T>(method: string, params: JsonObject = {}): Promise<T> {
return this.sendCommand<T>(method, params);
}
async sendSessionCommand<T>(sessionId: string, method: string, params: JsonObject = {}): Promise<T> {
return this.sendCommand<T>(method, params, sessionId);
}
private async createPageTarget(initialUrl: string, visible = false): Promise<{ targetId: string }> {
const attempts: JsonObject[] = visible
? [
{
url: initialUrl,
newWindow: true,
focus: true,
},
{
url: initialUrl,
focus: true,
},
{
url: initialUrl,
},
]
: [
{
url: initialUrl,
hidden: true,
},
{
url: initialUrl,
background: true,
focus: false,
},
{
url: initialUrl,
},
];
let lastError: unknown;
for (const params of attempts) {
try {
return await this.sendBrowserCommand<{ targetId: string }>("Target.createTarget", params);
} catch (error) {
lastError = error;
}
}
throw lastError instanceof Error ? lastError : new Error("Target.createTarget failed");
}
async createPageSession(options: CreatePageSessionOptions = {}): Promise<TargetSession> {
const initialUrl = options.initialUrl ?? "about:blank";
const created = await this.createPageTarget(initialUrl, Boolean(options.visible));
const attached = await this.sendBrowserCommand<{ sessionId: string }>("Target.attachToTarget", {
targetId: created.targetId,
flatten: true,
});
const session = new TargetSession(this, created.targetId, attached.sessionId);
this.sessions.set(attached.sessionId, session);
if (options.visible) {
await this.sendBrowserCommand("Target.activateTarget", {
targetId: created.targetId,
}).catch(() => {});
}
await session.send("Page.enable");
await session.send("Runtime.enable");
await session.send("DOM.enable");
if (options.visible) {
await session.send("Page.bringToFront").catch(() => {});
}
return session;
}
async closeTarget(targetId: string): Promise<void> {
try {
await this.sendBrowserCommand("Target.closeTarget", { targetId });
} catch {
// Target may already be gone.
}
}
async close(): Promise<void> {
await new Promise<void>((resolve) => {
if (this.ws.readyState === WebSocket.CLOSED) {
resolve();
return;
}
this.ws.once("close", () => resolve());
this.ws.close();
});
}
}
export async function evaluateRuntime<T>(session: TargetSession, expression: string): Promise<T> {
const response = await session.send<CdpCommandResult<{ value?: T; description?: string }>>("Runtime.evaluate", {
expression,
awaitPromise: true,
returnByValue: true,
});
if (response.error) {
throw new Error(response.error.message ?? "Runtime.evaluate failed");
}
return (response.result?.value as T | undefined) ?? (undefined as T);
}
@@ -0,0 +1,187 @@
import { launch, type LaunchedChrome } from "chrome-launcher";
import WebSocket from "ws";
import type { Logger } from "../utils/logger";
import {
cleanChromeLockArtifacts,
ensureChromeProfileDir,
findChromeProcessUsingProfile,
findExistingChromeDebugPort,
hasChromeLockArtifacts,
listChromeProfileEntries,
resolveChromeProfileDir,
shouldRetryChromeLaunchRecovery,
} from "./profile";
interface ChromeVersionResponse {
webSocketDebuggerUrl: string;
}
export interface ChromeConnectOptions {
cdpUrl?: string;
browserPath?: string;
headless?: boolean;
logger?: Logger;
profileDir?: string;
}
export interface ChromeConnection {
browserWsUrl: string;
origin?: string;
port?: number;
profileDir?: string;
launched: boolean;
close(): Promise<void>;
}
async function fetchJson<T>(url: string): Promise<T> {
const response = await fetch(url);
if (!response.ok) {
throw new Error(`Failed to fetch ${url}: HTTP ${response.status}`);
}
return (await response.json()) as T;
}
async function connectToHttpEndpoint(origin: string): Promise<ChromeConnection> {
const normalizedOrigin = origin.replace(/\/$/, "");
const version = await fetchJson<ChromeVersionResponse>(`${normalizedOrigin}/json/version`);
return {
browserWsUrl: version.webSocketDebuggerUrl,
origin: normalizedOrigin,
port: Number(new URL(normalizedOrigin).port || 80),
launched: false,
async close() {
// Reused external Chrome, nothing to close here.
},
};
}
async function tryReuseChrome(profileDir: string, logger?: Logger): Promise<ChromeConnection | null> {
const port = await findExistingChromeDebugPort({ profileDir });
if (!port) {
return null;
}
const origin = `http://127.0.0.1:${port}`;
try {
const connection = await connectToHttpEndpoint(origin);
logger?.info(`Reusing Chrome debugger at ${origin} for profile ${profileDir}`);
return {
...connection,
profileDir,
};
} catch {
// Debugger disappeared between detection and connect.
}
return null;
}
async function launchFreshChrome(
profileDir: string,
options: Pick<ChromeConnectOptions, "browserPath" | "headless">,
): Promise<ChromeConnection> {
let launchedChrome: LaunchedChrome | null = null;
try {
launchedChrome = await launch({
chromePath: options.browserPath,
userDataDir: profileDir,
chromeFlags: [
"--disable-background-networking",
"--disable-default-apps",
"--disable-popup-blocking",
"--disable-sync",
"--no-first-run",
"--no-default-browser-check",
"--remote-allow-origins=*",
...(!options.headless ? ["--no-startup-window"] : []),
...(options.headless ? ["--headless=new"] : []),
],
});
const origin = `http://127.0.0.1:${launchedChrome.port}`;
const version = await fetchJson<ChromeVersionResponse>(`${origin}/json/version`);
const chrome = launchedChrome;
return {
browserWsUrl: version.webSocketDebuggerUrl,
origin,
port: launchedChrome.port,
profileDir,
launched: true,
async close() {
if (!chrome) return;
await gracefulCloseChrome(chrome, origin);
},
};
} catch (error) {
launchedChrome?.kill();
throw error;
}
}
async function gracefulCloseChrome(chrome: LaunchedChrome, origin: string): Promise<void> {
try {
const resp = await fetch(`${origin}/json/version`);
const { webSocketDebuggerUrl } = (await resp.json()) as ChromeVersionResponse;
if (webSocketDebuggerUrl) {
const ws = await new Promise<WebSocket>((resolve, reject) => {
const socket = new WebSocket(webSocketDebuggerUrl);
socket.once("open", () => resolve(socket));
socket.once("error", reject);
});
const id = 1;
ws.send(JSON.stringify({ id, method: "Browser.close" }));
await new Promise<void>((resolve) => {
const timer = setTimeout(() => { ws.close(); resolve(); }, 5_000);
ws.once("close", () => { clearTimeout(timer); resolve(); });
});
const exited = await new Promise<boolean>((resolve) => {
if (chrome.pid && !isProcessAlive(chrome.pid)) { resolve(true); return; }
const timer = setTimeout(() => resolve(false), 3_000);
chrome.process.once("exit", () => { clearTimeout(timer); resolve(true); });
});
if (exited) return;
}
} catch {}
chrome.kill();
}
function isProcessAlive(pid: number): boolean {
try { process.kill(pid, 0); return true; } catch { return false; }
}
export async function connectChrome(options: ChromeConnectOptions): Promise<ChromeConnection> {
if (options.cdpUrl) {
if (options.cdpUrl.startsWith("ws://") || options.cdpUrl.startsWith("wss://")) {
return {
browserWsUrl: options.cdpUrl,
launched: false,
async close() {},
};
}
return connectToHttpEndpoint(options.cdpUrl);
}
const profileDir = ensureChromeProfileDir(resolveChromeProfileDir(options.profileDir));
const reused = await tryReuseChrome(profileDir, options.logger);
if (reused) {
return reused;
}
options.logger?.warn(`No running Chrome debugger found for profile ${profileDir}. Launching Chrome with that profile.`);
try {
return await launchFreshChrome(profileDir, options);
} catch (error) {
const entries = await listChromeProfileEntries(profileDir);
const shouldRetry = shouldRetryChromeLaunchRecovery({
hasLockArtifacts: hasChromeLockArtifacts(entries),
hasLiveOwner: findChromeProcessUsingProfile(profileDir),
});
if (!shouldRetry) {
throw error;
}
options.logger?.warn(`Chrome launch failed with stale profile locks. Cleaning ${profileDir} and retrying once.`);
cleanChromeLockArtifacts(profileDir);
return await launchFreshChrome(profileDir, options);
}
}
@@ -0,0 +1,100 @@
import { readFile, writeFile, mkdir } from "node:fs/promises";
import { dirname, join } from "node:path";
import { resolveChromeProfileDir } from "./profile";
import type { TargetSession } from "./cdp-client";
export interface CdpCookie {
name: string;
value: string;
domain: string;
path: string;
expires: number;
size: number;
httpOnly: boolean;
secure: boolean;
session: boolean;
sameSite?: string;
priority?: string;
sameParty?: boolean;
sourceScheme?: string;
sourcePort?: number;
partitionKey?: string;
}
interface SidecarData {
savedAt: string;
cookies: CdpCookie[];
}
export interface CookieSidecarConfig {
urls: readonly string[];
filename: string;
requiredCookieNames: readonly string[];
filterCookie?: (cookie: CdpCookie) => boolean;
}
function sidecarPath(filename: string, profileDir?: string): string {
return join(resolveChromeProfileDir(profileDir), filename);
}
function hasRequired(cookies: CdpCookie[], names: readonly string[]): boolean {
return names.every((name) =>
cookies.some((c) => c.name === name && Boolean(c.value)),
);
}
async function getCookies(session: TargetSession, urls: readonly string[]): Promise<CdpCookie[]> {
const { cookies } = await session.send<{ cookies: CdpCookie[] }>(
"Network.getCookies",
{ urls: [...urls] },
);
return cookies ?? [];
}
export async function exportCookies(
session: TargetSession,
config: CookieSidecarConfig,
profileDir?: string,
): Promise<boolean> {
const all = await getCookies(session, config.urls);
const filtered = config.filterCookie ? all.filter(config.filterCookie) : all;
if (!hasRequired(filtered, config.requiredCookieNames)) return false;
const filePath = sidecarPath(config.filename, profileDir);
await mkdir(dirname(filePath), { recursive: true });
const data: SidecarData = { savedAt: new Date().toISOString(), cookies: filtered };
await writeFile(filePath, JSON.stringify(data, null, 2));
return true;
}
export async function restoreCookies(
session: TargetSession,
config: CookieSidecarConfig,
profileDir?: string,
): Promise<boolean> {
const live = await getCookies(session, config.urls);
if (hasRequired(live, config.requiredCookieNames)) return false;
const filePath = sidecarPath(config.filename, profileDir);
const raw = await readFile(filePath, "utf8");
const data = JSON.parse(raw) as SidecarData;
if (!data.cookies?.length) return false;
const now = Date.now() / 1000;
const valid = data.cookies.filter((c) => c.session || !c.expires || c.expires > now);
if (!hasRequired(valid, config.requiredCookieNames)) return false;
await session.send("Network.setCookies", {
cookies: valid.map((c) => ({
name: c.name,
value: c.value,
domain: c.domain,
path: c.path,
httpOnly: c.httpOnly,
secure: c.secure,
sameSite: c.sameSite,
expires: c.expires,
})),
});
return true;
}
@@ -0,0 +1,123 @@
import type { WaitForInteractionRequest } from "../adapters/types";
import type { BrowserSession } from "./session";
interface GateSnapshot {
title: string;
currentUrl: string;
bodyText: string;
hasCloudflareTurnstile: boolean;
hasCloudflareChallenge: boolean;
hasRecaptcha: boolean;
hasRecaptchaIframe: boolean;
hasHcaptcha: boolean;
hasHcaptchaIframe: boolean;
}
export function detectInteractionGateFromSnapshot(snapshot: GateSnapshot): WaitForInteractionRequest | null {
const text = snapshot.bodyText.toLowerCase();
const title = snapshot.title.toLowerCase();
const url = snapshot.currentUrl.toLowerCase();
if (
snapshot.hasCloudflareTurnstile ||
snapshot.hasCloudflareChallenge ||
title.includes("just a moment") ||
text.includes("verify you are human") ||
text.includes("checking your browser before accessing") ||
text.includes("enable javascript and cookies to continue") ||
url.includes("/cdn-cgi/challenge-platform/")
) {
return {
type: "wait_for_interaction",
kind: "cloudflare",
provider: "cloudflare",
reason: "Cloudflare human verification detected",
prompt: "Please complete the Cloudflare verification in the opened Chrome window. Extraction will continue automatically once the challenge disappears.",
requiresVisibleBrowser: true,
};
}
if (
snapshot.hasRecaptcha ||
snapshot.hasRecaptchaIframe ||
text.includes("i'm not a robot") ||
text.includes("recaptcha")
) {
return {
type: "wait_for_interaction",
kind: "recaptcha",
provider: "google_recaptcha",
reason: "Google reCAPTCHA detected",
prompt: "Please complete the reCAPTCHA verification in the opened Chrome window. Extraction will continue automatically once the challenge disappears.",
requiresVisibleBrowser: true,
};
}
if (
snapshot.hasHcaptcha ||
snapshot.hasHcaptchaIframe ||
text.includes("hcaptcha")
) {
return {
type: "wait_for_interaction",
kind: "hcaptcha",
provider: "hcaptcha",
reason: "hCaptcha verification detected",
prompt: "Please complete the hCaptcha verification in the opened Chrome window. Extraction will continue automatically once the challenge disappears.",
requiresVisibleBrowser: true,
};
}
return null;
}
export async function detectInteractionGate(browser: BrowserSession): Promise<WaitForInteractionRequest | null> {
const snapshot = await browser.evaluate<GateSnapshot>(`
(() => {
const bodyText = (document.body?.innerText ?? "").slice(0, 4000);
return {
title: document.title ?? "",
currentUrl: window.location.href,
bodyText,
hasCloudflareTurnstile: Boolean(
document.querySelector(
'.cf-turnstile, [name="cf-turnstile-response"], iframe[src*="challenges.cloudflare.com"]'
)
),
hasCloudflareChallenge: Boolean(
document.querySelector(
'#challenge-running, #cf-challenge-running, .challenge-platform, [data-ray], [data-translate="checking_browser"]'
)
),
hasRecaptcha: Boolean(
document.querySelector(
'.g-recaptcha, textarea[name="g-recaptcha-response"], iframe[title*="reCAPTCHA"]'
)
),
hasRecaptchaIframe: Boolean(
document.querySelector('iframe[src*="google.com/recaptcha"], iframe[src*="recaptcha/api2"]')
),
hasHcaptcha: Boolean(
document.querySelector(
'.h-captcha, textarea[name="h-captcha-response"], iframe[title*="hCaptcha"]'
)
),
hasHcaptchaIframe: Boolean(
document.querySelector('iframe[src*="hcaptcha.com"]')
),
};
})()
`).catch(() => ({
title: "",
currentUrl: "",
bodyText: "",
hasCloudflareTurnstile: false,
hasCloudflareChallenge: false,
hasRecaptcha: false,
hasRecaptchaIframe: false,
hasHcaptcha: false,
hasHcaptchaIframe: false,
}));
return detectInteractionGateFromSnapshot(snapshot);
}
@@ -0,0 +1,235 @@
import type { TargetSession } from "./cdp-client";
import type { Logger } from "../utils/logger";
type JsonObject = Record<string, unknown>;
export interface NetworkEntry {
requestId: string;
url: string;
method: string;
resourceType: string;
timestamp: number;
requestHeaders?: Record<string, string>;
requestBody?: string;
status?: number;
statusText?: string;
responseHeaders?: Record<string, string>;
mimeType?: string;
body?: string;
bodyBase64?: boolean;
bodyError?: string;
failed?: boolean;
failureReason?: string;
finished: boolean;
}
function normalizeHeaders(headers: unknown): Record<string, string> | undefined {
if (!headers || typeof headers !== "object") {
return undefined;
}
return Object.fromEntries(
Object.entries(headers as Record<string, unknown>).map(([key, value]) => [key, String(value)]),
);
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export class NetworkJournal {
private readonly entries = new Map<string, NetworkEntry>();
private lastActivityAt = Date.now();
private started = false;
constructor(
private readonly session: TargetSession,
private readonly log: Logger,
) {}
async start(): Promise<void> {
if (this.started) {
return;
}
this.started = true;
this.session.on("Network.requestWillBeSent", this.handleRequestWillBeSent);
this.session.on("Network.responseReceived", this.handleResponseReceived);
this.session.on("Network.loadingFinished", this.handleLoadingFinished);
this.session.on("Network.loadingFailed", this.handleLoadingFailed);
await this.session.send("Network.enable");
}
stop(): void {
if (!this.started) {
return;
}
this.session.off("Network.requestWillBeSent", this.handleRequestWillBeSent);
this.session.off("Network.responseReceived", this.handleResponseReceived);
this.session.off("Network.loadingFinished", this.handleLoadingFinished);
this.session.off("Network.loadingFailed", this.handleLoadingFailed);
this.started = false;
}
private touch(): void {
this.lastActivityAt = Date.now();
}
private readonly handleRequestWillBeSent = (params: JsonObject): void => {
const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
const request = params.request as JsonObject | undefined;
if (!requestId || !request) {
return;
}
this.touch();
this.entries.set(requestId, {
requestId,
url: String(request.url ?? ""),
method: String(request.method ?? "GET"),
resourceType: String(params.type ?? "Other"),
timestamp: Date.now(),
requestHeaders: normalizeHeaders(request.headers),
requestBody: typeof request.postData === "string" ? request.postData : undefined,
finished: false,
});
};
private readonly handleResponseReceived = (params: JsonObject): void => {
const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
const response = params.response as JsonObject | undefined;
if (!requestId || !response) {
return;
}
this.touch();
const existing = this.entries.get(requestId);
if (!existing) {
return;
}
existing.status = typeof response.status === "number" ? response.status : undefined;
existing.statusText = typeof response.statusText === "string" ? response.statusText : undefined;
existing.responseHeaders = normalizeHeaders(response.headers);
existing.mimeType = typeof response.mimeType === "string" ? response.mimeType : undefined;
this.entries.set(requestId, existing);
};
private readonly handleLoadingFinished = (params: JsonObject): void => {
const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
if (!requestId) {
return;
}
this.touch();
const existing = this.entries.get(requestId);
if (!existing) {
return;
}
existing.finished = true;
this.entries.set(requestId, existing);
};
private readonly handleLoadingFailed = (params: JsonObject): void => {
const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
if (!requestId) {
return;
}
this.touch();
const existing = this.entries.get(requestId);
if (!existing) {
return;
}
existing.finished = true;
existing.failed = true;
existing.failureReason = typeof params.errorText === "string" ? params.errorText : "Unknown error";
this.entries.set(requestId, existing);
};
getEntries(): NetworkEntry[] {
return Array.from(this.entries.values());
}
findEntries(predicate: (entry: NetworkEntry) => boolean): NetworkEntry[] {
return this.getEntries().filter(predicate);
}
async waitForIdle(options: { idleMs?: number; timeoutMs?: number } = {}): Promise<void> {
const idleMs = options.idleMs ?? 1_200;
const timeoutMs = options.timeoutMs ?? 15_000;
const startedAt = Date.now();
while (Date.now() - startedAt < timeoutMs) {
if (Date.now() - this.lastActivityAt >= idleMs) {
return;
}
await sleep(Math.min(150, idleMs));
}
throw new Error("Timed out waiting for network idle");
}
async waitForResponse(
predicate: (entry: NetworkEntry) => boolean,
options: { timeoutMs?: number } = {},
): Promise<NetworkEntry> {
const timeoutMs = options.timeoutMs ?? 10_000;
const startedAt = Date.now();
while (Date.now() - startedAt < timeoutMs) {
const matched = this.getEntries().find((entry) => entry.finished && predicate(entry));
if (matched) {
return matched;
}
await sleep(150);
}
throw new Error("Timed out waiting for matching network response");
}
async ensureBody(entry: NetworkEntry): Promise<string | undefined> {
if (entry.body !== undefined) {
return entry.body;
}
if (entry.bodyError || entry.failed || !entry.finished) {
return undefined;
}
try {
const result = await this.session.send<{ body: string; base64Encoded: boolean }>("Network.getResponseBody", {
requestId: entry.requestId,
});
entry.bodyBase64 = result.base64Encoded;
entry.body = result.base64Encoded ? Buffer.from(result.body, "base64").toString("utf8") : result.body;
return entry.body;
} catch (error) {
entry.bodyError = error instanceof Error ? error.message : String(error);
this.log.debug(`Failed to fetch response body for ${entry.url}: ${entry.bodyError}`);
return undefined;
}
}
async getJsonBody(entry: NetworkEntry): Promise<unknown | null> {
const body = await this.ensureBody(entry);
if (!body) {
return null;
}
try {
return JSON.parse(body);
} catch {
return null;
}
}
async toJSON(options: { includeBodies?: boolean } = {}): Promise<NetworkEntry[]> {
const entries = this.getEntries();
if (!options.includeBodies) {
return entries;
}
await Promise.all(entries.map((entry) => this.ensureBody(entry)));
return entries;
}
}
@@ -0,0 +1,105 @@
import type { BrowserSession } from "./session";
export interface CapturedPageSnapshot {
html: string;
finalUrl: string;
}
export const CAPTURE_NORMALIZED_PAGE_SCRIPT = String.raw`
(() => {
const baseUrl = document.baseURI || location.href;
const htmlClone = document.documentElement.cloneNode(true);
function materializeShadowDom(sourceRoot, cloneRoot) {
const sourceElements = Array.from(sourceRoot.querySelectorAll("*"));
const cloneElements = Array.from(cloneRoot.querySelectorAll("*"));
for (let index = sourceElements.length - 1; index >= 0; index -= 1) {
const sourceElement = sourceElements[index];
const cloneElement = cloneElements[index];
const shadowRoot = sourceElement && sourceElement.shadowRoot;
if (!shadowRoot || !cloneElement || !shadowRoot.innerHTML) {
continue;
}
if (cloneElement.tagName && cloneElement.tagName.includes("-")) {
const wrapper = document.createElement("div");
wrapper.setAttribute("data-shadow-host", cloneElement.tagName.toLowerCase());
wrapper.innerHTML = shadowRoot.innerHTML;
cloneElement.replaceWith(wrapper);
} else {
cloneElement.innerHTML = shadowRoot.innerHTML;
}
}
}
function toAbsolute(url) {
if (!url) return url;
try {
return new URL(url, baseUrl).href;
} catch {
return url;
}
}
function absolutizeAttribute(root, selector, attribute) {
root.querySelectorAll(selector).forEach((element) => {
const value = element.getAttribute(attribute);
if (!value) return;
const absolute = toAbsolute(value);
if (absolute) {
element.setAttribute(attribute, absolute);
}
});
}
function absolutizeSrcset(root, selector) {
root.querySelectorAll(selector).forEach((element) => {
const srcset = element.getAttribute("srcset");
if (!srcset) return;
element.setAttribute(
"srcset",
srcset
.split(",")
.map((part) => {
const trimmed = part.trim();
if (!trimmed) return "";
const [url, ...descriptor] = trimmed.split(/\s+/);
const absolute = toAbsolute(url);
return descriptor.length > 0 ? absolute + " " + descriptor.join(" ") : absolute;
})
.filter(Boolean)
.join(", "),
);
});
}
materializeShadowDom(document.documentElement, htmlClone);
htmlClone
.querySelectorAll("img[data-src], video[data-src], audio[data-src], source[data-src]")
.forEach((element) => {
const dataSource = element.getAttribute("data-src");
const current = element.getAttribute("src");
if (dataSource && (!current || current === "" || current.startsWith("data:"))) {
element.setAttribute("src", dataSource);
}
});
absolutizeAttribute(htmlClone, "a[href]", "href");
absolutizeAttribute(htmlClone, "img[src], video[src], audio[src], source[src], iframe[src]", "src");
absolutizeAttribute(htmlClone, "video[poster]", "poster");
absolutizeSrcset(htmlClone, "img[srcset], source[srcset]");
return {
html: "<!doctype html>\n" + htmlClone.outerHTML,
finalUrl: location.href,
};
})()
`;
export async function captureNormalizedPageSnapshot(
browser: BrowserSession,
): Promise<CapturedPageSnapshot> {
return browser.evaluate<CapturedPageSnapshot>(CAPTURE_NORMALIZED_PAGE_SCRIPT);
}
@@ -0,0 +1,201 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import process from "node:process";
import { spawnSync } from "node:child_process";
export interface ResolveSharedChromeProfileDirOptions {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
}
export interface FindExistingChromeDebugPortOptions {
profileDir: string;
timeoutMs?: number;
}
interface ChromeVersionResponse {
webSocketDebuggerUrl?: string;
}
const CHROME_LOCK_FILE_NAMES = ["SingletonLock", "SingletonSocket", "SingletonCookie", "chrome.pid"] as const;
function resolveDataBaseDir(): string {
if (process.platform === "darwin") {
return path.join(os.homedir(), "Library", "Application Support");
}
if (process.platform === "win32") {
return process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming");
}
return process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share");
}
export function resolveSharedChromeProfileDir(
options: ResolveSharedChromeProfileDirOptions = {},
): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) {
return path.resolve(override);
}
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
return path.join(resolveDataBaseDir(), appDataDirName, profileDirName);
}
export function resolveChromeProfileDir(profileDir?: string): string {
if (profileDir?.trim()) {
return path.resolve(profileDir.trim());
}
return resolveSharedChromeProfileDir({
envNames: ["BAOYU_CHROME_PROFILE_DIR"],
appDataDirName: "baoyu-skills",
profileDirName: "chrome-profile",
});
}
export function ensureChromeProfileDir(profileDir: string): string {
fs.mkdirSync(profileDir, { recursive: true });
return profileDir;
}
export function hasChromeLockArtifacts(entries: readonly string[]): boolean {
return CHROME_LOCK_FILE_NAMES.some((name) => entries.includes(name));
}
export function shouldRetryChromeLaunchRecovery(options: {
hasLockArtifacts: boolean;
hasLiveOwner: boolean;
}): boolean {
return options.hasLockArtifacts && !options.hasLiveOwner;
}
export function findChromeProcessUsingProfile(profileDir: string): boolean {
if (process.platform === "win32") {
return false;
}
try {
const result = spawnSync("ps", ["aux"], {
encoding: "utf8",
timeout: 5_000,
});
if (result.status !== 0 || !result.stdout) {
return false;
}
return result.stdout
.split("\n")
.some((line) => line.includes(`--user-data-dir=${profileDir}`));
} catch {
return false;
}
}
export function cleanChromeLockArtifacts(profileDir: string): void {
for (const name of CHROME_LOCK_FILE_NAMES) {
try {
fs.unlinkSync(path.join(profileDir, name));
} catch {
// Ignore missing files and continue cleaning the remaining artifacts.
}
}
}
export async function listChromeProfileEntries(profileDir: string): Promise<string[]> {
try {
return await fs.promises.readdir(profileDir);
} catch {
return [];
}
}
async function fetchWithTimeout(url: string, timeoutMs = 3_000): Promise<Response> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
try {
return await fetch(url, {
redirect: "follow",
signal: controller.signal,
});
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T>(url: string, timeoutMs = 3_000): Promise<T> {
const response = await fetchWithTimeout(url, timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return (await response.json()) as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<ChromeVersionResponse>(`http://127.0.0.1:${port}/json/version`, timeoutMs);
return Boolean(version.webSocketDebuggerUrl);
} catch {
return false;
}
}
function parseDevToolsActivePort(filePath: string): { port: number; wsPath: string } | null {
try {
const content = fs.readFileSync(filePath, "utf8");
const lines = content.split(/\r?\n/);
const port = Number.parseInt(lines[0]?.trim() ?? "", 10);
const wsPath = lines[1]?.trim() ?? "";
if (port > 0 && wsPath) {
return { port, wsPath };
}
} catch {
// Ignore and fall back to process inspection.
}
return null;
}
export async function findExistingChromeDebugPort(
options: FindExistingChromeDebugPortOptions,
): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const activePort = parseDevToolsActivePort(path.join(options.profileDir, "DevToolsActivePort"));
if (activePort && await isDebugPortReady(activePort.port, timeoutMs)) {
return activePort.port;
}
if (process.platform === "win32") {
return null;
}
try {
const result = spawnSync("ps", ["aux"], {
encoding: "utf8",
timeout: 5_000,
});
if (result.status !== 0 || !result.stdout) {
return null;
}
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const match = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(match?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) {
return port;
}
}
} catch {
// Ignore and report no reusable debugger.
}
return null;
}
@@ -0,0 +1,155 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { CdpClient, TargetSession, evaluateRuntime } from "./cdp-client";
interface NavigationResult {
errorText?: string;
}
const execFileAsync = promisify(execFile);
const MACOS_BROWSER_APP_IDS = [
"com.google.Chrome",
"org.chromium.Chromium",
"com.brave.Browser",
"com.microsoft.edgemac",
];
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function activateBrowserApp(): Promise<void> {
if (process.platform !== "darwin") {
return;
}
for (const appId of MACOS_BROWSER_APP_IDS) {
try {
await execFileAsync("osascript", ["-e", `tell application id "${appId}" to activate`]);
return;
} catch {
// Try the next installed browser bundle id.
}
}
}
export class BrowserSession {
private constructor(
private readonly cdp: CdpClient,
public readonly targetSession: TargetSession,
public readonly interactive: boolean,
) {}
static async open(
cdp: CdpClient,
options: {
initialUrl?: string;
interactive?: boolean;
} = {},
): Promise<BrowserSession> {
const targetSession = await cdp.createPageSession({
initialUrl: options.initialUrl,
visible: options.interactive,
});
const browser = new BrowserSession(cdp, targetSession, Boolean(options.interactive));
if (browser.interactive) {
await browser.bringToFront().catch(() => {});
}
return browser;
}
async goto(url: string, timeoutMs = 30_000): Promise<void> {
const loadPromise = this.targetSession.waitForEvent("Page.loadEventFired", undefined, timeoutMs).catch(() => null);
const result = await this.targetSession.send<NavigationResult>("Page.navigate", { url });
if (result.errorText) {
throw new Error(`Navigation failed: ${result.errorText}`);
}
await loadPromise;
await this.waitForReadyState(timeoutMs);
}
async waitForReadyState(timeoutMs = 30_000): Promise<void> {
const startedAt = Date.now();
while (Date.now() - startedAt < timeoutMs) {
const state = await this.evaluate<string>("document.readyState");
if (state === "interactive" || state === "complete") {
return;
}
await sleep(150);
}
throw new Error("Timed out waiting for document.readyState");
}
async evaluate<T>(expression: string): Promise<T> {
return evaluateRuntime<T>(this.targetSession, expression);
}
async getHTML(): Promise<string> {
return this.evaluate<string>("document.documentElement.outerHTML");
}
async getTitle(): Promise<string> {
return this.evaluate<string>("document.title");
}
async getURL(): Promise<string> {
return this.evaluate<string>("window.location.href");
}
async bringToFront(): Promise<void> {
await this.targetSession.send("Page.bringToFront").catch(async () => {
await this.cdp.sendBrowserCommand("Target.activateTarget", {
targetId: this.targetSession.targetId,
});
});
if (this.interactive) {
await activateBrowserApp().catch(() => {});
}
}
async click(selector: string): Promise<void> {
const result = await this.evaluate<{ ok: boolean; error?: string }>(`
(() => {
const element = document.querySelector(${JSON.stringify(selector)});
if (!element) {
return { ok: false, error: "Element not found" };
}
element.scrollIntoView({ block: "center", inline: "center" });
if (element instanceof HTMLElement) {
element.click();
return { ok: true };
}
return { ok: false, error: "Element is not clickable" };
})()
`);
if (!result.ok) {
throw new Error(result.error ?? `Failed to click ${selector}`);
}
}
async scrollToEnd(options: { stepPx?: number; delayMs?: number; maxSteps?: number } = {}): Promise<void> {
const stepPx = options.stepPx ?? 1_400;
const delayMs = options.delayMs ?? 250;
const maxSteps = options.maxSteps ?? 6;
for (let step = 0; step < maxSteps; step += 1) {
const done = await this.evaluate<boolean>(`
(() => {
const before = window.scrollY;
window.scrollBy(0, ${stepPx});
const atBottom = window.innerHeight + window.scrollY >= document.body.scrollHeight - 4;
return atBottom || window.scrollY === before;
})()
`);
if (done) {
break;
}
await sleep(delayMs);
}
}
async close(): Promise<void> {
await this.cdp.closeTarget(this.targetSession.targetId);
}
}
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env bun
import {
runConvertCommand,
type ConvertCommandOptions,
type OutputFormat,
type WaitMode,
} from "./commands/convert";
export const HELP_TEXT = `
baoyu-fetch - Read a URL into Markdown or JSON with Chrome CDP
Usage:
baoyu-fetch <url> [options]
Options:
--output <file> Save output to file
--format <type> Output format: markdown | json
--json Alias for --format json
--adapter <name> Force an adapter (e.g. x, generic)
--download-media Download adapter-reported media into ./imgs and ./videos, then rewrite markdown links
--media-dir <dir> Base directory for downloaded media. Defaults to the output directory
--debug-dir <dir> Write debug artifacts
--cdp-url <url> Reuse an existing Chrome DevTools endpoint
--browser-path <path> Explicit Chrome binary path
--chrome-profile-dir <path>
Chrome user data dir. Defaults to BAOYU_CHROME_PROFILE_DIR
or baoyu-skills/chrome-profile.
--headless Launch a temporary headless Chrome if needed
--wait-for <mode> Wait mode: interaction | force
interaction: start visible Chrome and auto-wait only when login or verification is required
force: start visible Chrome, then auto-continue after it detects login/challenge progress
or continue immediately when you press Enter
--wait-for-interaction
Alias for --wait-for interaction
--wait-for-login Alias for --wait-for interaction
--interaction-timeout <ms>
How long to wait for manual interaction before failing (default: 600000)
--interaction-poll-interval <ms>
How often to poll interaction state while waiting (default: 1500)
--login-timeout <ms> Alias for --interaction-timeout
--login-poll-interval <ms>
Alias for --interaction-poll-interval
--timeout <ms> Page timeout in milliseconds (default: 30000)
--help Show help
Examples:
baoyu-fetch https://example.com
baoyu-fetch https://example.com --format markdown --output article.md --download-media
baoyu-fetch https://example.com --format json --output article.json
baoyu-fetch https://x.com/lennysan/status/2036483059407810640 --wait-for interaction
baoyu-fetch https://x.com/lennysan/status/2036483059407810640 --wait-for force
`.trim();
interface CliOptions extends ConvertCommandOptions {
url?: string;
help: boolean;
}
function normalizeWaitMode(raw: string): WaitMode {
const value = raw.toLowerCase();
if (value === "interaction" || value === "auto") {
return "interaction";
}
if (value === "force" || value === "manual" || value === "always") {
return "force";
}
throw new Error(`Invalid wait mode: ${raw}. Expected interaction or force.`);
}
function normalizeOutputFormat(raw: string): OutputFormat {
const value = raw.toLowerCase();
if (value === "markdown" || value === "json") {
return value;
}
throw new Error(`Invalid output format: ${raw}. Expected markdown or json.`);
}
export function parseArgs(argv: string[]): CliOptions {
const options: CliOptions = {
format: "markdown",
headless: false,
downloadMedia: false,
waitMode: "none",
interactionTimeoutMs: 600_000,
interactionPollIntervalMs: 1_500,
timeoutMs: 30_000,
help: false,
};
const args = argv.slice(2);
for (let index = 0; index < args.length; index += 1) {
const value = args[index];
if (value === "--help" || value === "-h") {
options.help = true;
continue;
}
if (value === "--format") {
const format = args[index + 1];
if (!format) {
throw new Error("--format requires a value");
}
options.format = normalizeOutputFormat(format);
index += 1;
continue;
}
if (value === "--json") {
options.format = "json";
continue;
}
if (value === "--download-media") {
options.downloadMedia = true;
continue;
}
if (value === "--headless") {
options.headless = true;
continue;
}
if (value === "--wait-for") {
const mode = args[index + 1];
if (!mode) {
throw new Error("--wait-for requires a mode");
}
options.waitMode = normalizeWaitMode(mode);
index += 1;
continue;
}
if (value === "--wait-for-interaction" || value === "--wait-for-login") {
options.waitMode = "interaction";
continue;
}
if (value === "--output") {
options.output = args[index + 1];
index += 1;
continue;
}
if (value === "--adapter") {
options.adapter = args[index + 1];
index += 1;
continue;
}
if (value === "--debug-dir") {
options.debugDir = args[index + 1];
index += 1;
continue;
}
if (value === "--media-dir") {
options.mediaDir = args[index + 1];
index += 1;
continue;
}
if (value === "--cdp-url") {
options.cdpUrl = args[index + 1];
index += 1;
continue;
}
if (value === "--browser-path") {
options.browserPath = args[index + 1];
index += 1;
continue;
}
if (value === "--chrome-profile-dir") {
options.chromeProfileDir = args[index + 1];
index += 1;
continue;
}
if (value === "--timeout") {
const parsed = Number(args[index + 1]);
if (!Number.isFinite(parsed) || parsed <= 0) {
throw new Error(`Invalid timeout: ${args[index + 1]}`);
}
options.timeoutMs = parsed;
index += 1;
continue;
}
if (value === "--interaction-timeout" || value === "--login-timeout") {
const parsed = Number(args[index + 1]);
if (!Number.isFinite(parsed) || parsed <= 0) {
throw new Error(`Invalid interaction timeout: ${args[index + 1]}`);
}
options.interactionTimeoutMs = parsed;
index += 1;
continue;
}
if (value === "--interaction-poll-interval" || value === "--login-poll-interval") {
const parsed = Number(args[index + 1]);
if (!Number.isFinite(parsed) || parsed <= 0) {
throw new Error(`Invalid interaction poll interval: ${args[index + 1]}`);
}
options.interactionPollIntervalMs = parsed;
index += 1;
continue;
}
if (value.startsWith("-")) {
throw new Error(`Unknown option: ${value}`);
}
if (!options.url) {
options.url = value;
continue;
}
throw new Error(`Unexpected argument: ${value}`);
}
return options;
}
async function main(): Promise<void> {
try {
const options = parseArgs(process.argv);
if (options.help || !options.url) {
console.log(HELP_TEXT);
return;
}
await runConvertCommand(options);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.error(message);
process.exitCode = 1;
}
}
if (import.meta.main) {
void main();
}
@@ -0,0 +1,580 @@
import { mkdir, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { createInterface } from "node:readline";
import { connectChrome, type ChromeConnection } from "../browser/chrome-launcher";
import { CdpClient } from "../browser/cdp-client";
import { detectInteractionGate } from "../browser/interaction-gates";
import { NetworkJournal } from "../browser/network-journal";
import { BrowserSession } from "../browser/session";
import { genericAdapter, resolveAdapter } from "../adapters";
import { isXSessionReady } from "../adapters/x/session";
import type { ExtractedDocument } from "../extract/document";
import { renderMarkdown } from "../extract/markdown-renderer";
import { downloadMediaAssets } from "../media/default-downloader";
import { rewriteMarkdownMediaLinks } from "../media/markdown-media";
import { createLogger } from "../utils/logger";
import { normalizeUrl } from "../utils/url";
import type {
Adapter,
AdapterContext,
AdapterLoginInfo,
LoginState,
MediaAsset,
WaitForInteractionRequest,
} from "../adapters/types";
export type WaitMode = "none" | "interaction" | "force";
export type OutputFormat = "markdown" | "json";
export interface ConvertCommandOptions {
url?: string;
output?: string;
format: OutputFormat;
adapter?: string;
debugDir?: string;
cdpUrl?: string;
browserPath?: string;
chromeProfileDir?: string;
headless: boolean;
downloadMedia: boolean;
mediaDir?: string;
waitMode: WaitMode;
interactionTimeoutMs: number;
interactionPollIntervalMs: number;
timeoutMs: number;
}
interface RuntimeResources {
chrome: ChromeConnection;
cdp: CdpClient;
browser: BrowserSession;
network: NetworkJournal;
interactive: boolean;
}
interface ForceWaitSnapshot {
url: string;
hasGate: boolean;
loginState: LoginState | "unavailable";
sessionReady: boolean;
}
interface SuccessfulConvertOutput {
adapter: string;
status: "ok";
login?: AdapterLoginInfo;
media: MediaAsset[];
downloads: Awaited<ReturnType<typeof downloadMediaAssets>> | null;
document: ExtractedDocument;
markdown: string;
}
interface InteractionRequiredOutput {
adapter: string;
status: "needs_interaction";
login?: AdapterLoginInfo;
interaction: WaitForInteractionRequest;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function isForceWaitSessionReady(snapshot: ForceWaitSnapshot): boolean {
return snapshot.sessionReady;
}
export function shouldKeepBrowserOpenAfterInteraction(options: {
launched: boolean;
interaction: Pick<WaitForInteractionRequest, "kind" | "provider">;
}): boolean {
return options.launched && options.interaction.kind === "login" && options.interaction.provider === "x";
}
export function shouldAutoContinueForceWait(
initial: ForceWaitSnapshot,
current: ForceWaitSnapshot,
): boolean {
if (initial.hasGate && !current.hasGate) {
return true;
}
if (initial.loginState === "logged_out" && current.loginState !== "logged_out" && isForceWaitSessionReady(current)) {
return true;
}
if (initial.loginState !== "logged_in" && current.loginState === "logged_in" && isForceWaitSessionReady(current)) {
return true;
}
if (
current.url !== initial.url &&
!current.hasGate &&
current.loginState !== "logged_out" &&
isForceWaitSessionReady(current)
) {
return true;
}
return false;
}
async function writeOutput(path: string, content: string): Promise<void> {
const directory = path.includes("/") ? path.slice(0, path.lastIndexOf("/")) : "";
if (directory) {
await mkdir(directory, { recursive: true });
}
await writeFile(path, content, "utf8");
}
async function writeDebugArtifacts(
debugDir: string,
document: ExtractedDocument,
markdown: string,
browser: BrowserSession,
network: NetworkJournal,
): Promise<void> {
await mkdir(debugDir, { recursive: true });
const html = await browser.getHTML().catch(() => "");
const networkDump = await network.toJSON({ includeBodies: true });
await Promise.all([
writeFile(join(debugDir, "document.json"), JSON.stringify(document, null, 2), "utf8"),
writeFile(join(debugDir, "markdown.md"), markdown, "utf8"),
writeFile(join(debugDir, "page.html"), html, "utf8"),
writeFile(join(debugDir, "network.json"), JSON.stringify(networkDump, null, 2), "utf8"),
]);
}
async function openRuntime(
options: ConvertCommandOptions,
interactive: boolean,
debugEnabled: boolean,
): Promise<RuntimeResources> {
const logger = createLogger(debugEnabled);
if (interactive) {
logger.info("Opening Chrome in interactive mode.");
}
const chrome = await connectChrome({
cdpUrl: options.cdpUrl,
browserPath: options.browserPath,
profileDir: options.chromeProfileDir,
headless: interactive ? false : options.headless,
logger,
});
const cdp = await CdpClient.connect(chrome.browserWsUrl);
const browser = await BrowserSession.open(cdp, { interactive });
if (interactive) {
await browser.bringToFront().catch(() => {});
}
const network = new NetworkJournal(browser.targetSession, logger);
await network.start();
return {
chrome,
cdp,
browser,
network,
interactive,
};
}
async function closeRuntime(runtime: RuntimeResources | null | undefined): Promise<void> {
if (!runtime) {
return;
}
runtime.network.stop();
await runtime.browser.close().catch(() => {});
await runtime.cdp.close().catch(() => {});
await runtime.chrome.close().catch(() => {});
}
async function isInteractionSessionReady(
context: AdapterContext,
interaction: WaitForInteractionRequest,
): Promise<boolean> {
if (interaction.provider !== "x") {
return true;
}
return await isXSessionReady(context).catch(() => false);
}
async function reopenInteractiveRuntime(
runtime: RuntimeResources,
options: ConvertCommandOptions,
debugEnabled: boolean,
): Promise<RuntimeResources> {
if (runtime.interactive) {
return runtime;
}
await closeRuntime(runtime);
return openRuntime(options, true, debugEnabled);
}
async function captureForceWaitSnapshot(
adapter: Adapter,
context: AdapterContext,
): Promise<ForceWaitSnapshot> {
const [gate, url, login] = await Promise.all([
detectInteractionGate(context.browser).catch(() => null),
context.browser.getURL().catch(() => context.input.url.toString()),
adapter.checkLogin?.(context).catch(() => ({
provider: adapter.name,
state: "unknown" as const,
})),
]);
return {
url,
hasGate: Boolean(gate),
loginState: login?.state ?? "unavailable",
sessionReady: adapter.name === "x" ? await isXSessionReady(context).catch(() => false) : true,
};
}
async function waitForForceResume(
adapter: Adapter,
context: AdapterContext,
options: ConvertCommandOptions,
): Promise<void> {
if (context.interactive) {
await context.browser.bringToFront().catch(() => {});
}
const prompt =
"Chrome is ready. Complete any manual login or verification. Extraction will continue automatically after it detects progress, or press Enter to continue immediately.";
context.log.info(prompt);
const rl = createInterface({
input: process.stdin,
output: process.stderr,
});
let manualContinue = false;
let closed = false;
const closeReadline = (): void => {
if (!closed) {
closed = true;
rl.close();
}
};
rl.once("line", () => {
manualContinue = true;
closeReadline();
});
const initial = await captureForceWaitSnapshot(adapter, context);
const startedAt = Date.now();
try {
while (Date.now() - startedAt < options.interactionTimeoutMs) {
if (manualContinue) {
return;
}
const current = await captureForceWaitSnapshot(adapter, context);
if (shouldAutoContinueForceWait(initial, current)) {
return;
}
await sleep(options.interactionPollIntervalMs);
}
} finally {
closeReadline();
}
throw new Error("Timed out waiting for force-mode interaction to complete");
}
async function waitForInteraction(
adapter: Adapter,
context: AdapterContext,
interaction: WaitForInteractionRequest,
options: ConvertCommandOptions,
): Promise<AdapterLoginInfo> {
const timeoutMs = interaction.timeoutMs ?? options.interactionTimeoutMs;
const pollIntervalMs = interaction.pollIntervalMs ?? options.interactionPollIntervalMs;
if (context.interactive) {
await context.browser.bringToFront().catch(() => {});
}
context.log.info(interaction.prompt);
const startedAt = Date.now();
let lastLogin: AdapterLoginInfo | null = null;
while (Date.now() - startedAt < timeoutMs) {
if (interaction.kind === "login" && adapter.checkLogin) {
lastLogin = await adapter.checkLogin(context);
if (lastLogin.state === "logged_in" && await isInteractionSessionReady(context, interaction)) {
return lastLogin;
}
}
const gate = await detectInteractionGate(context.browser);
if (!gate) {
if (interaction.kind !== "login") {
return lastLogin ?? {
provider: interaction.provider,
state: "unknown",
reason: `${interaction.provider} challenge cleared`,
};
}
if (!adapter.checkLogin) {
return {
provider: interaction.provider,
state: "unknown",
};
}
lastLogin = await adapter.checkLogin(context);
if (lastLogin.state !== "logged_out" && await isInteractionSessionReady(context, interaction)) {
return lastLogin;
}
}
await sleep(pollIntervalMs);
}
const reason = lastLogin?.reason ? ` (${lastLogin.reason})` : "";
throw new Error(`Timed out waiting for ${interaction.provider} interaction${reason}`);
}
export function formatOutputContent(
format: OutputFormat,
payload: SuccessfulConvertOutput | InteractionRequiredOutput,
): string {
if (format === "json") {
return JSON.stringify(payload, null, 2);
}
if (payload.status !== "ok") {
throw new Error("Markdown output is only available for successful extraction results");
}
return payload.markdown;
}
function printOutput(content: string): void {
process.stdout.write(content);
if (!content.endsWith("\n")) {
process.stdout.write("\n");
}
}
export async function runConvertCommand(options: ConvertCommandOptions): Promise<void> {
if (!options.url) {
throw new Error("URL is required");
}
if (options.downloadMedia && !options.output) {
throw new Error("--download-media requires --output so media paths can be rewritten relative to the saved output file");
}
const url = normalizeUrl(options.url);
let runtime = await openRuntime(options, options.waitMode !== "none", Boolean(options.debugDir));
const logger = createLogger(Boolean(options.debugDir));
let didLogin = false;
let adapter: Adapter | null = null;
let context: AdapterContext | null = null;
try {
adapter = resolveAdapter({ url }, options.adapter);
context = {
input: { url },
browser: runtime.browser,
network: runtime.network,
cdp: runtime.cdp,
log: logger,
outputFormat: options.format,
timeoutMs: options.timeoutMs,
interactive: runtime.interactive,
downloadMedia: options.downloadMedia,
};
if (adapter.restoreCookies) {
const restored = await adapter.restoreCookies(context, runtime.chrome.profileDir).catch(() => false);
if (restored) logger.info(`Restored ${adapter.name} session cookies from sidecar.`);
}
if (options.waitMode === "interaction" && adapter.checkLogin) {
await context.browser.goto(url.toString(), options.timeoutMs).catch(() => {});
const preLogin = await adapter.checkLogin(context);
if (preLogin.state !== "logged_in") {
didLogin = true;
await waitForInteraction(adapter, context, {
type: "wait_for_interaction",
kind: "login",
provider: preLogin.provider ?? adapter.name,
prompt: `Please sign in to ${adapter.name === "x" ? "X" : adapter.name} in the opened Chrome window. Extraction will continue automatically once login is detected.`,
reason: preLogin.reason ?? `Not logged in to ${adapter.name}`,
requiresVisibleBrowser: true,
}, options);
}
}
if (options.waitMode === "force") {
await context.browser.goto(url.toString(), options.timeoutMs).catch(() => {});
await waitForForceResume(adapter, context, options);
}
let result = await adapter.process(context);
if (result.status === "no_document") {
const interaction = await detectInteractionGate(context.browser);
if (interaction) {
result = {
status: "needs_interaction",
interaction,
login: result.login,
};
}
}
while (result.status === "needs_interaction") {
if (options.waitMode === "none") {
if (options.format === "json") {
printOutput(
formatOutputContent(options.format, {
adapter: adapter.name,
status: result.status,
login: result.login,
interaction: result.interaction,
}),
);
return;
}
throw new Error(`${adapter.name} requires manual interaction. Re-run with --wait-for interaction to continue after completing it.`);
}
if (result.interaction.requiresVisibleBrowser !== false) {
runtime = await reopenInteractiveRuntime(runtime, options, Boolean(options.debugDir));
}
context = {
input: { url },
browser: runtime.browser,
network: runtime.network,
cdp: runtime.cdp,
log: logger,
outputFormat: options.format,
timeoutMs: options.timeoutMs,
interactive: runtime.interactive,
downloadMedia: options.downloadMedia,
};
await context.browser.goto(url.toString(), options.timeoutMs).catch(() => {});
if (result.interaction.kind === "login") {
didLogin = true;
}
await waitForInteraction(adapter, context, result.interaction, options);
result = await adapter.process(context);
if (result.status === "no_document") {
const interaction = await detectInteractionGate(context.browser);
if (interaction) {
result = {
status: "needs_interaction",
interaction,
login: result.login,
};
}
}
}
let document: ExtractedDocument | null = result.status === "ok" ? result.document : null;
let media: MediaAsset[] = result.status === "ok" ? (result.media ?? []) : [];
let login = result.login;
let mediaAdapter = adapter;
if (!document && adapter.name !== genericAdapter.name && result.status === "no_document") {
logger.info(`Adapter ${adapter.name} returned no structured document; falling back to generic extraction`);
const fallback = await genericAdapter.process(context);
if (fallback.status === "ok") {
document = fallback.document;
media = fallback.media ?? [];
mediaAdapter = genericAdapter;
}
}
if (!document) {
throw new Error("Failed to extract a document from the target URL");
}
document.requestedUrl ??= url.toString();
let markdown = renderMarkdown(document);
let downloadResult:
| Awaited<ReturnType<typeof downloadMediaAssets>>
| null = null;
if (options.downloadMedia && options.output) {
downloadResult = mediaAdapter.downloadMedia
? await mediaAdapter.downloadMedia({
media,
outputPath: options.output,
mediaDir: options.mediaDir,
log: logger,
})
: await downloadMediaAssets({
media,
outputPath: options.output,
mediaDir: options.mediaDir,
log: logger,
});
markdown = rewriteMarkdownMediaLinks(markdown, downloadResult.replacements);
if (downloadResult.downloadedImages > 0 || downloadResult.downloadedVideos > 0) {
logger.info(
`Downloaded ${downloadResult.downloadedImages} images and ${downloadResult.downloadedVideos} videos`,
);
}
}
if (options.output) {
await writeOutput(
options.output,
formatOutputContent(options.format, {
adapter: document.adapter ?? adapter.name,
status: "ok",
login,
media,
downloads: downloadResult,
document,
markdown,
}),
);
logger.info(`Saved ${options.format} to ${options.output}`);
}
if (options.debugDir) {
await writeDebugArtifacts(options.debugDir, document, markdown, runtime.browser, runtime.network);
logger.info(`Wrote debug artifacts to ${options.debugDir}`);
}
if (options.format === "json") {
printOutput(
formatOutputContent(options.format, {
adapter: document.adapter ?? adapter.name,
status: "ok",
login,
media,
downloads: downloadResult,
document,
markdown,
}),
);
return;
}
printOutput(markdown);
} finally {
if (adapter?.exportCookies && context) {
await adapter.exportCookies(context, runtime.chrome.profileDir).catch(() => {});
}
await closeRuntime(runtime);
}
}
@@ -0,0 +1,51 @@
export type ContentBlock =
| {
type: "paragraph";
text: string;
}
| {
type: "heading";
depth: number;
text: string;
}
| {
type: "list";
ordered: boolean;
items: string[];
}
| {
type: "quote";
text: string;
}
| {
type: "code";
code: string;
language?: string;
}
| {
type: "image";
url: string;
alt?: string;
}
| {
type: "html";
html: string;
}
| {
type: "markdown";
markdown: string;
};
export interface ExtractedDocument {
url: string;
requestedUrl?: string;
canonicalUrl?: string;
title?: string;
author?: string;
siteName?: string;
publishedAt?: string;
summary?: string;
content: ContentBlock[];
metadata?: Record<string, unknown>;
adapter?: string;
}
@@ -0,0 +1,467 @@
import { JSDOM } from "jsdom";
export interface CleanHtmlOptions {
removeAds?: boolean;
removeBase64Images?: boolean;
onlyMainContent?: boolean;
includeSelectors?: string[];
excludeSelectors?: string[];
}
const ALWAYS_REMOVE_SELECTORS = [
"script",
"style",
"noscript",
"link[rel='stylesheet']",
"[hidden]",
"[aria-hidden='true']",
"[style*='display: none']",
"[style*='display:none']",
"[style*='visibility: hidden']",
"[style*='visibility:hidden']",
"svg[aria-hidden='true']",
"svg.icon",
"svg[class*='icon']",
"template",
"meta",
"iframe",
"canvas",
"object",
"embed",
"form",
"input",
"select",
"textarea",
"button",
];
const OVERLAY_SELECTORS = [
"[class*='modal']",
"[class*='popup']",
"[class*='overlay']",
"[class*='dialog']",
"[role='dialog']",
"[role='alertdialog']",
"[class*='cookie']",
"[class*='consent']",
"[class*='gdpr']",
"[class*='privacy-banner']",
"[class*='notification-bar']",
"[id*='cookie']",
"[id*='consent']",
"[id*='gdpr']",
"[style*='position: fixed']",
"[style*='position:fixed']",
"[style*='position: sticky']",
"[style*='position:sticky']",
];
const NAVIGATION_SELECTORS = [
"header",
"footer",
"nav",
"aside",
".header",
".top",
".navbar",
"#header",
".footer",
".bottom",
"#footer",
".sidebar",
".side",
".aside",
"#sidebar",
".modal",
".popup",
"#modal",
".overlay",
".ad",
".ads",
".advert",
"#ad",
".lang-selector",
".language",
"#language-selector",
".social",
".social-media",
".social-links",
"#social",
".menu",
".navigation",
"#nav",
".breadcrumbs",
"#breadcrumbs",
".share",
"#share",
".widget",
"#widget",
".cookie",
"#cookie",
];
const FORCE_INCLUDE_SELECTORS = [
"#main",
"#content",
"#main-content",
"#article",
"#post",
"#page-content",
"main",
"article",
"[role='main']",
".main-content",
".content",
".post-content",
".article-content",
".entry-content",
".page-content",
".article-body",
".post-body",
".story-content",
".blog-content",
];
const AD_SELECTORS = [
"ins.adsbygoogle",
".google-ad",
".adsense",
"[data-ad]",
"[data-ads]",
"[data-ad-slot]",
"[data-ad-client]",
".ad-container",
".ad-wrapper",
".advertisement",
".sponsored-content",
"img[width='1'][height='1']",
"img[src*='pixel']",
"img[src*='tracking']",
"img[src*='analytics']",
];
function getLinkDensity(element: Element): number {
const text = element.textContent || "";
const textLength = text.trim().length;
if (textLength === 0) {
return 1;
}
let linkLength = 0;
element.querySelectorAll("a").forEach((link) => {
linkLength += (link.textContent || "").trim().length;
});
return linkLength / textLength;
}
function getContentScore(element: Element): number {
let score = 0;
const text = element.textContent || "";
const textLength = text.trim().length;
score += Math.min(textLength / 100, 50);
score += element.querySelectorAll("p").length * 3;
score += element.querySelectorAll("h1, h2, h3, h4, h5, h6").length * 2;
score += element.querySelectorAll("img").length;
score -= element.querySelectorAll("a").length * 0.5;
score -= element.querySelectorAll("li").length * 0.2;
const linkDensity = getLinkDensity(element);
if (linkDensity > 0.5) {
score -= 30;
} else if (linkDensity > 0.3) {
score -= 15;
}
const className = typeof element.className === "string" ? element.className : "";
const classAndId = `${className} ${element.id || ""}`;
if (/article|content|post|body|main|entry/i.test(classAndId)) {
score += 25;
}
if (/comment|sidebar|footer|nav|menu|header|widget|ad/i.test(classAndId)) {
score -= 25;
}
return score;
}
function looksLikeNavigation(element: Element): boolean {
const linkDensity = getLinkDensity(element);
if (linkDensity > 0.5) {
return true;
}
const listItems = element.querySelectorAll("li");
const links = element.querySelectorAll("a");
return listItems.length > 5 && links.length > listItems.length * 0.8;
}
function removeElements(document: Document, selectors: string[]): void {
for (const selector of selectors) {
try {
document.querySelectorAll(selector).forEach((element) => element.remove());
} catch {
// Ignore unsupported selectors.
}
}
}
function removeWithProtection(
document: Document,
selectorsToRemove: string[],
protectedSelectors: string[],
): void {
for (const selector of selectorsToRemove) {
try {
document.querySelectorAll(selector).forEach((element) => {
const isProtected = protectedSelectors.some((protectedSelector) => {
try {
return element.matches(protectedSelector);
} catch {
return false;
}
});
if (isProtected) {
return;
}
const containsProtected = protectedSelectors.some((protectedSelector) => {
try {
return element.querySelector(protectedSelector) !== null;
} catch {
return false;
}
});
if (containsProtected) {
return;
}
element.remove();
});
} catch {
// Ignore unsupported selectors.
}
}
}
function isValidContent(element: Element | null): element is Element {
if (!element) {
return false;
}
const text = element.textContent || "";
if (text.trim().length < 100) {
return false;
}
return !looksLikeNavigation(element);
}
function findMainContent(document: Document): Element | null {
const main = document.querySelector("main");
if (isValidContent(main) && getLinkDensity(main) < 0.4) {
return main;
}
const roleMain = document.querySelector('[role="main"]');
if (isValidContent(roleMain) && getLinkDensity(roleMain) < 0.4) {
return roleMain;
}
const articles = document.querySelectorAll("article");
if (articles.length === 1 && isValidContent(articles[0] ?? null)) {
return articles[0] ?? null;
}
const contentSelectors = [
"#content",
"#main-content",
"#main",
".content",
".main-content",
".post-content",
".article-content",
".entry-content",
".page-content",
".article-body",
".post-body",
".story-content",
".blog-content",
];
for (const selector of contentSelectors) {
try {
const element = document.querySelector(selector);
if (isValidContent(element) && getLinkDensity(element) < 0.4) {
return element;
}
} catch {
// Ignore invalid selectors.
}
}
const candidates: Array<{ element: Element; score: number }> = [];
document.querySelectorAll("div, section, article").forEach((element) => {
const text = element.textContent || "";
if (text.trim().length < 200) {
return;
}
const score = getContentScore(element);
if (score > 0) {
candidates.push({ element, score });
}
});
candidates.sort((left, right) => right.score - left.score);
if ((candidates[0]?.score ?? 0) > 20) {
return candidates[0]?.element ?? null;
}
return null;
}
function removeBase64ImagesFromDocument(document: Document): void {
document.querySelectorAll("img[src^='data:']").forEach((element) => element.remove());
document.querySelectorAll("[style*='data:image']").forEach((element) => {
const style = element.getAttribute("style");
if (!style) {
return;
}
const cleanedStyle = style.replace(
/background(-image)?:\s*url\([^)]*data:image[^)]*\)[^;]*;?/gi,
"",
);
if (cleanedStyle.trim()) {
element.setAttribute("style", cleanedStyle);
} else {
element.removeAttribute("style");
}
});
document
.querySelectorAll("source[src^='data:'], source[srcset*='data:']")
.forEach((element) => element.remove());
}
function makeAbsoluteUrl(value: string, baseUrl: string): string | null {
try {
return new URL(value, baseUrl).toString();
} catch {
return null;
}
}
function convertRelativeUrls(document: Document, baseUrl: string): void {
document.querySelectorAll("[src]").forEach((element) => {
const src = element.getAttribute("src");
if (!src || src.startsWith("http") || src.startsWith("//") || src.startsWith("data:")) {
return;
}
const absolute = makeAbsoluteUrl(src, baseUrl);
if (absolute) {
element.setAttribute("src", absolute);
}
});
document.querySelectorAll("[href]").forEach((element) => {
const href = element.getAttribute("href");
if (
!href ||
href.startsWith("http") ||
href.startsWith("//") ||
href.startsWith("#") ||
href.startsWith("mailto:") ||
href.startsWith("tel:") ||
href.startsWith("javascript:")
) {
return;
}
const absolute = makeAbsoluteUrl(href, baseUrl);
if (absolute) {
element.setAttribute("href", absolute);
}
});
}
function removeComments(document: Document): void {
const walker = document.createTreeWalker(document, document.defaultView?.NodeFilter.SHOW_COMMENT ?? 128);
const comments: Comment[] = [];
while (walker.nextNode()) {
comments.push(walker.currentNode as Comment);
}
comments.forEach((comment) => comment.parentNode?.removeChild(comment));
}
export function cleanHtml(
html: string,
baseUrl: string,
options: CleanHtmlOptions = {},
): string {
const {
removeAds = true,
removeBase64Images = true,
onlyMainContent = true,
includeSelectors,
excludeSelectors,
} = options;
const dom = new JSDOM(html, { url: baseUrl });
const { document } = dom.window;
removeElements(document, ALWAYS_REMOVE_SELECTORS);
removeElements(document, OVERLAY_SELECTORS);
if (removeAds) {
removeElements(document, AD_SELECTORS);
}
if (excludeSelectors?.length) {
removeElements(document, excludeSelectors);
}
if (onlyMainContent) {
removeWithProtection(document, NAVIGATION_SELECTORS, FORCE_INCLUDE_SELECTORS);
const mainContent = findMainContent(document);
if (mainContent && document.body) {
const clone = mainContent.cloneNode(true);
document.body.innerHTML = "";
document.body.appendChild(clone);
}
}
if (includeSelectors?.length && document.body) {
const matchedElements: Element[] = [];
for (const selector of includeSelectors) {
try {
document.querySelectorAll(selector).forEach((element) => {
matchedElements.push(element.cloneNode(true) as Element);
});
} catch {
// Ignore invalid selectors.
}
}
if (matchedElements.length > 0) {
document.body.innerHTML = "";
matchedElements.forEach((element) => document.body?.appendChild(element));
}
}
if (removeBase64Images) {
removeBase64ImagesFromDocument(document);
}
removeComments(document);
convertRelativeUrls(document, baseUrl);
return document.documentElement.outerHTML || html;
}
@@ -0,0 +1,83 @@
import { Readability } from "@mozilla/readability";
import { JSDOM } from "jsdom";
import type { ExtractedDocument } from "./document";
function getMetaContent(document: Document, selectors: string[]): string | undefined {
for (const selector of selectors) {
const value = document.querySelector(selector)?.getAttribute("content")?.trim();
if (value) {
return value;
}
}
return undefined;
}
export function extractDocumentFromHtml(input: {
url: string;
html: string;
adapter?: string;
}): ExtractedDocument {
const dom = new JSDOM(input.html, { url: input.url });
const document = dom.window.document;
const canonicalUrl =
document.querySelector('link[rel="canonical"]')?.getAttribute("href")?.trim() ??
getMetaContent(document, ['meta[property="og:url"]']);
const siteName = getMetaContent(document, [
'meta[property="og:site_name"]',
'meta[name="application-name"]',
]);
const metadataAuthor = getMetaContent(document, [
'meta[name="author"]',
'meta[property="article:author"]',
'meta[name="twitter:creator"]',
]);
const publishedAt = getMetaContent(document, [
'meta[property="article:published_time"]',
'meta[name="pubdate"]',
'meta[name="date"]',
'meta[itemprop="datePublished"]',
]);
const article = new Readability(document).parse();
const title =
article?.title?.trim() ||
getMetaContent(document, ['meta[property="og:title"]']) ||
document.title.trim() ||
undefined;
const summary =
article?.excerpt?.trim() ||
getMetaContent(document, [
'meta[name="description"]',
'meta[property="og:description"]',
'meta[name="twitter:description"]',
]);
const contentHtml =
article?.content?.trim() ||
document.querySelector("main")?.innerHTML?.trim() ||
document.body?.innerHTML?.trim() ||
"";
const author = article?.byline?.trim() || metadataAuthor;
return {
url: input.url,
canonicalUrl,
title,
author,
siteName,
publishedAt,
summary,
adapter: input.adapter ?? "generic",
metadata: {
language: document.documentElement.lang || undefined,
},
content: contentHtml ? [{ type: "html", html: contentHtml }] : [],
};
}
@@ -0,0 +1,758 @@
import { Readability } from "@mozilla/readability";
import { Defuddle } from "defuddle/node";
import { JSDOM, VirtualConsole } from "jsdom";
import TurndownService from "turndown";
import { gfm } from "turndown-plugin-gfm";
import { collectMediaFromMarkdown } from "../media/markdown-media";
import type { MediaAsset } from "../media/types";
import { cleanHtml } from "./html-cleaner";
export interface HtmlConversionMetadata {
url: string;
canonicalUrl?: string;
siteName?: string;
title?: string;
summary?: string;
author?: string;
publishedAt?: string;
coverImage?: string;
language?: string;
capturedAt: string;
}
export interface ConvertHtmlToMarkdownOptions {
enableRemoteMarkdownFallback?: boolean;
preserveBase64Images?: boolean;
}
export interface HtmlToMarkdownResult {
metadata: HtmlConversionMetadata;
markdown: string;
rawHtml: string;
cleanedHtml: string;
media: MediaAsset[];
conversionMethod: string;
fallbackReason?: string;
}
type JsonObject = Record<string, unknown>;
const MIN_CONTENT_LENGTH = 120;
const DEFUDDLE_API_ORIGIN = "https://defuddle.md";
const LOCAL_FALLBACK_SCORE_DELTA = 120;
const REMOTE_FALLBACK_SCORE_DELTA = 20;
const LOW_QUALITY_MARKERS = [
/Join The Conversation/i,
/One Community\. Many Voices/i,
/Read our community guidelines/i,
/Create a free account to share your thoughts/i,
/Become a Forbes Member/i,
/Subscribe to trusted journalism/i,
/\bComments\b/i,
];
const ARTICLE_TYPES = new Set([
"Article",
"NewsArticle",
"BlogPosting",
"WebPage",
"ReportageNewsArticle",
]);
const turndown = new TurndownService({
headingStyle: "atx",
bulletListMarker: "-",
codeBlockStyle: "fenced",
}) as TurndownService & {
remove(selectors: string[]): void;
addRule(
key: string,
rule: {
filter: string | ((node: Node) => boolean);
replacement: (content: string) => string;
},
): void;
};
turndown.use(gfm);
turndown.remove(["script", "style", "iframe", "noscript", "template", "svg", "path"]);
turndown.addRule("collapseFigure", {
filter: "figure",
replacement(content: string) {
return `\n\n${content.trim()}\n\n`;
},
});
turndown.addRule("dropInvisibleAnchors", {
filter(node: Node) {
return (
node.nodeName === "A" &&
!(node as Element).textContent?.trim() &&
!(node as Element).querySelector("img, video, picture, source")
);
},
replacement() {
return "";
},
});
function pickString(...values: unknown[]): string | undefined {
for (const value of values) {
if (typeof value !== "string") {
continue;
}
const trimmed = value.trim();
if (trimmed) {
return trimmed;
}
}
return undefined;
}
function normalizeMarkdown(markdown: string): string {
return markdown
.replace(/\r\n/g, "\n")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
}
function stripWrappingQuotes(value: string): string {
const trimmed = value.trim();
if (
(trimmed.startsWith('"') && trimmed.endsWith('"')) ||
(trimmed.startsWith("'") && trimmed.endsWith("'"))
) {
return trimmed.slice(1, -1).trim();
}
return trimmed;
}
function stripMarkdownFrontmatter(markdown: string): string {
return markdown.replace(/^\uFEFF?---\n[\s\S]*?\n---(?:\n|$)/, "").trim();
}
function cleanMarkdownTitle(value: string): string | undefined {
const cleaned = stripWrappingQuotes(
value
.replace(/\s+#+\s*$/, "")
.replace(/!\[[^\]]*\]\([^)]+\)/g, "")
.replace(/\[([^\]]+)\]\([^)]+\)/g, "$1")
.replace(/[*_`~]/g, "")
.trim(),
);
return cleaned || undefined;
}
export function extractTitleFromMarkdownDocument(markdown: string): string | undefined {
const normalized = markdown.replace(/\r\n/g, "\n").trim();
if (!normalized) {
return undefined;
}
const frontmatterMatch = normalized.match(/^\uFEFF?---\n([\s\S]*?)\n---(?:\n|$)/);
if (frontmatterMatch) {
for (const line of frontmatterMatch[1].split("\n")) {
const match = line.match(/^title:\s*(.+?)\s*$/i);
if (!match) {
continue;
}
const title = cleanMarkdownTitle(match[1]);
if (title) {
return title;
}
}
}
const body = stripMarkdownFrontmatter(normalized);
const headingMatch = body.match(/^#{1,6}\s+(.+)$/m);
if (!headingMatch) {
return undefined;
}
return cleanMarkdownTitle(headingMatch[1]);
}
function trimKnownBoilerplate(markdown: string): string {
const normalized = normalizeMarkdown(markdown);
const lines = normalized.split("\n");
while (lines.length > 0) {
const lastLine = lines[lines.length - 1]?.trim();
if (!lastLine) {
lines.pop();
continue;
}
if (/^继续滑动看下一个$/.test(lastLine) || /^轻触阅读原文$/.test(lastLine)) {
lines.pop();
continue;
}
break;
}
return normalizeMarkdown(lines.join("\n"));
}
function buildDefuddleApiUrl(targetUrl: string): string {
return `${DEFUDDLE_API_ORIGIN}/${encodeURIComponent(targetUrl)}`;
}
async function fetchDefuddleApiMarkdown(
targetUrl: string,
): Promise<{ markdown: string; title?: string }> {
const response = await fetch(buildDefuddleApiUrl(targetUrl), {
headers: {
accept: "text/markdown,text/plain;q=0.9,*/*;q=0.1",
},
redirect: "follow",
});
if (!response.ok) {
throw new Error(`defuddle.md returned ${response.status} ${response.statusText}`);
}
const rawMarkdown = (await response.text()).replace(/\r\n/g, "\n").trim();
if (!rawMarkdown) {
throw new Error("defuddle.md returned empty markdown");
}
const title = extractTitleFromMarkdownDocument(rawMarkdown);
const markdown = trimKnownBoilerplate(stripMarkdownFrontmatter(rawMarkdown));
if (!markdown) {
throw new Error("defuddle.md returned empty markdown");
}
return {
markdown,
title,
};
}
function sanitizeHtmlFragment(html: string): string {
const dom = new JSDOM(`<div id="__root">${html}</div>`);
const root = dom.window.document.querySelector("#__root");
if (!root) {
return html;
}
for (const selector of ["script", "style", "iframe", "noscript", "template", "svg", "path"]) {
root.querySelectorAll(selector).forEach((element) => element.remove());
}
return root.innerHTML;
}
function extractTextFromHtml(html: string): string {
const dom = new JSDOM(`<!doctype html><html><body>${html}</body></html>`);
const { document } = dom.window;
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
document.querySelectorAll(selector).forEach((element) => element.remove());
}
return document.body?.textContent?.replace(/\s+/g, " ").trim() ?? "";
}
function getMetaContent(document: Document, names: string[]): string | undefined {
for (const name of names) {
const element =
document.querySelector(`meta[name="${name}"]`) ??
document.querySelector(`meta[property="${name}"]`);
const content = element?.getAttribute("content")?.trim();
if (content) {
return content;
}
}
return undefined;
}
function normalizeLanguageTag(value: string | null | undefined): string | undefined {
if (!value) {
return undefined;
}
const trimmed = value.trim();
if (!trimmed) {
return undefined;
}
const primary = trimmed.split(/[,\s;]/, 1)[0]?.trim();
if (!primary) {
return undefined;
}
return primary.replace(/_/g, "-");
}
function flattenJsonLdItems(data: unknown): JsonObject[] {
if (!data || typeof data !== "object") {
return [];
}
if (Array.isArray(data)) {
return data.flatMap(flattenJsonLdItems);
}
const item = data as JsonObject;
if (Array.isArray(item["@graph"])) {
return (item["@graph"] as unknown[]).flatMap(flattenJsonLdItems);
}
return [item];
}
function parseJsonLdScripts(document: Document): JsonObject[] {
const results: JsonObject[] = [];
document.querySelectorAll("script[type='application/ld+json']").forEach((script) => {
try {
const data = JSON.parse(script.textContent ?? "");
results.push(...flattenJsonLdItems(data));
} catch {
// Ignore malformed json-ld blocks.
}
});
return results;
}
function extractAuthorFromJsonLd(authorData: unknown): string | undefined {
if (typeof authorData === "string") {
return authorData.trim() || undefined;
}
if (!authorData || typeof authorData !== "object") {
return undefined;
}
if (Array.isArray(authorData)) {
return authorData
.map((author) => extractAuthorFromJsonLd(author))
.filter((value): value is string => Boolean(value))
.join(", ") || undefined;
}
const author = authorData as JsonObject;
return pickString(author.name);
}
function extractPrimaryJsonLdMeta(document: Document): Partial<HtmlConversionMetadata> {
for (const item of parseJsonLdScripts(document)) {
const type = Array.isArray(item["@type"]) ? item["@type"][0] : item["@type"];
if (typeof type !== "string" || !ARTICLE_TYPES.has(type)) {
continue;
}
return {
title: pickString(item.headline, item.name),
summary: pickString(item.description),
author: extractAuthorFromJsonLd(item.author),
publishedAt: pickString(item.datePublished, item.dateCreated),
coverImage: pickString(
item.image,
(item.image as JsonObject | undefined)?.url,
Array.isArray(item.image) ? item.image[0] : undefined,
),
};
}
return {};
}
function extractPageMetadata(
html: string,
url: string,
capturedAt: string,
): HtmlConversionMetadata {
const dom = new JSDOM(html, { url });
const { document } = dom.window;
const jsonLd = extractPrimaryJsonLdMeta(document);
return {
url,
canonicalUrl:
document.querySelector('link[rel="canonical"]')?.getAttribute("href")?.trim() ??
getMetaContent(document, ["og:url"]),
siteName: pickString(
getMetaContent(document, ["og:site_name"]),
document.querySelector('meta[name="application-name"]')?.getAttribute("content"),
),
title: pickString(
getMetaContent(document, ["og:title", "twitter:title"]),
jsonLd.title,
document.querySelector("h1")?.textContent,
document.title,
),
summary: pickString(
getMetaContent(document, ["description", "og:description", "twitter:description"]),
jsonLd.summary,
),
author: pickString(
getMetaContent(document, ["author", "article:author", "twitter:creator"]),
jsonLd.author,
),
publishedAt: pickString(
document.querySelector("time[datetime]")?.getAttribute("datetime"),
getMetaContent(document, ["article:published_time", "datePublished", "publishdate", "date"]),
jsonLd.publishedAt,
),
coverImage: pickString(
getMetaContent(document, ["og:image", "twitter:image", "twitter:image:src"]),
jsonLd.coverImage,
),
language: pickString(
normalizeLanguageTag(document.documentElement.getAttribute("lang")),
normalizeLanguageTag(
pickString(
getMetaContent(document, ["language", "content-language", "og:locale"]),
document.querySelector("meta[http-equiv='content-language']")?.getAttribute("content"),
),
),
),
capturedAt,
};
}
function isMarkdownUsable(markdown: string, html: string): boolean {
const normalized = normalizeMarkdown(markdown);
if (!normalized) {
return false;
}
const htmlTextLength = extractTextFromHtml(html).length;
if (htmlTextLength < MIN_CONTENT_LENGTH) {
return true;
}
if (normalized.length >= 80) {
return true;
}
return normalized.length >= Math.min(200, Math.floor(htmlTextLength * 0.2));
}
function countMarkerHits(markdown: string, markers: RegExp[]): number {
let hits = 0;
for (const marker of markers) {
if (marker.test(markdown)) {
hits += 1;
}
}
return hits;
}
function countUsefulParagraphs(markdown: string): number {
const paragraphs = normalizeMarkdown(markdown).split(/\n{2,}/);
let count = 0;
for (const paragraph of paragraphs) {
const trimmed = paragraph.trim();
if (!trimmed) {
continue;
}
if (/^!?\[[^\]]*\]\([^)]+\)$/.test(trimmed)) {
continue;
}
if (/^#{1,6}\s+/.test(trimmed)) {
continue;
}
if ((trimmed.match(/\b[\p{L}\p{N}']+\b/gu) || []).length < 8) {
continue;
}
count += 1;
}
return count;
}
function scoreMarkdownQuality(markdown: string): number {
const normalized = normalizeMarkdown(markdown);
const wordCount = (normalized.match(/\b[\p{L}\p{N}']+\b/gu) || []).length;
const usefulParagraphs = countUsefulParagraphs(normalized);
const headingCount = (normalized.match(/^#{1,6}\s+/gm) || []).length;
const markerHits = countMarkerHits(normalized, LOW_QUALITY_MARKERS);
return Math.min(wordCount, 4000) + usefulParagraphs * 40 + headingCount * 10 - markerHits * 180;
}
function shouldCompareWithFallback(markdown: string): boolean {
const normalized = normalizeMarkdown(markdown);
return countMarkerHits(normalized, LOW_QUALITY_MARKERS) > 0 || countUsefulParagraphs(normalized) < 6;
}
function hasMeaningfulMarkdownStructure(markdown: string): boolean {
const normalized = normalizeMarkdown(markdown);
if (!normalized) {
return false;
}
return (
countUsefulParagraphs(normalized) > 0 ||
/^#{1,6}\s+/m.test(normalized) ||
/^[-*]\s+/m.test(normalized) ||
/^\d+\.\s+/m.test(normalized) ||
/!\[[^\]]*\]\([^)]+\)/.test(normalized)
);
}
function shouldTryRemoteMarkdownFallback(
markdown: string,
html: string,
options: ConvertHtmlToMarkdownOptions,
): boolean {
if (!options.enableRemoteMarkdownFallback) {
return false;
}
return !isMarkdownUsable(markdown, html) || shouldCompareWithFallback(markdown);
}
function shouldPreferRemoteMarkdown(
current: HtmlToMarkdownResult,
remote: HtmlToMarkdownResult,
html: string,
): boolean {
if (!isMarkdownUsable(current.markdown, html)) {
return true;
}
if (!hasMeaningfulMarkdownStructure(current.markdown) && hasMeaningfulMarkdownStructure(remote.markdown)) {
return true;
}
return scoreMarkdownQuality(remote.markdown) > scoreMarkdownQuality(current.markdown) + REMOTE_FALLBACK_SCORE_DELTA;
}
function buildRemoteFallbackReason(current: HtmlToMarkdownResult, html: string): string {
if (!isMarkdownUsable(current.markdown, html)) {
return current.fallbackReason
? `Used defuddle.md markdown fallback after local extraction failed: ${current.fallbackReason}`
: "Used defuddle.md markdown fallback after local extraction returned empty or incomplete markdown";
}
return "defuddle.md produced higher-quality markdown than local extraction";
}
async function tryDefuddleConversion(
html: string,
url: string,
baseMetadata: HtmlConversionMetadata,
): Promise<{ ok: true; result: HtmlToMarkdownResult } | { ok: false; reason: string }> {
try {
const virtualConsole = new VirtualConsole();
virtualConsole.on("jsdomError", (error: Error & { type?: string }) => {
if (error.type === "css parsing" || /Could not parse CSS stylesheet/i.test(error.message)) {
return;
}
});
const dom = new JSDOM(html, { url, virtualConsole });
const result = await Defuddle(dom, url, { markdown: true });
const markdown = trimKnownBoilerplate(result.content || "");
if (!isMarkdownUsable(markdown, html)) {
return { ok: false, reason: "Defuddle returned empty or incomplete markdown" };
}
const metadata: HtmlConversionMetadata = {
...baseMetadata,
title: pickString(result.title, baseMetadata.title),
summary: pickString(result.description, baseMetadata.summary),
author: pickString(result.author, baseMetadata.author),
publishedAt: pickString(result.published, baseMetadata.publishedAt),
coverImage: pickString(result.image, baseMetadata.coverImage),
language: pickString(result.language, baseMetadata.language),
};
return {
ok: true,
result: {
metadata,
markdown,
rawHtml: html,
cleanedHtml: html,
media: collectMediaFromMarkdown(markdown).concat(
metadata.coverImage
? [{ url: metadata.coverImage, kind: "image", role: "cover" as const }]
: [],
),
conversionMethod: "defuddle",
},
};
} catch (error) {
return {
ok: false,
reason: error instanceof Error ? error.message : String(error),
};
}
}
async function tryDefuddleApiConversion(
html: string,
url: string,
baseMetadata: HtmlConversionMetadata,
): Promise<{ ok: true; result: HtmlToMarkdownResult } | { ok: false; reason: string }> {
try {
const result = await fetchDefuddleApiMarkdown(url);
const markdown = result.markdown;
if (!isMarkdownUsable(markdown, html) && scoreMarkdownQuality(markdown) < 80) {
return { ok: false, reason: "defuddle.md returned empty or incomplete markdown" };
}
const metadata: HtmlConversionMetadata = {
...baseMetadata,
title: pickString(result.title, baseMetadata.title),
};
return {
ok: true,
result: {
metadata,
markdown,
rawHtml: html,
cleanedHtml: html,
media: collectMediaFromMarkdown(markdown).concat(
metadata.coverImage
? [{ url: metadata.coverImage, kind: "image", role: "cover" as const }]
: [],
),
conversionMethod: "defuddle-api",
},
};
} catch (error) {
return {
ok: false,
reason: error instanceof Error ? error.message : String(error),
};
}
}
function convertHtmlFragmentToMarkdown(html: string): string {
if (!html.trim()) {
return "";
}
try {
return turndown.turndown(sanitizeHtmlFragment(html));
} catch {
return "";
}
}
function fallbackPlainText(html: string): string {
return trimKnownBoilerplate(extractTextFromHtml(html));
}
function convertWithReadability(
rawHtml: string,
cleanedHtml: string,
url: string,
baseMetadata: HtmlConversionMetadata,
): HtmlToMarkdownResult {
const dom = new JSDOM(cleanedHtml, { url });
const document = dom.window.document;
const article = new Readability(document).parse();
const contentHtml =
article?.content?.trim() ??
document.querySelector("main")?.innerHTML?.trim() ??
document.body?.innerHTML?.trim() ??
"";
let markdown = contentHtml ? convertHtmlFragmentToMarkdown(contentHtml) : "";
if (!markdown) {
markdown = fallbackPlainText(cleanedHtml);
}
const metadata: HtmlConversionMetadata = {
...baseMetadata,
title: pickString(article?.title, baseMetadata.title),
summary: pickString(article?.excerpt, baseMetadata.summary),
author: pickString(article?.byline, baseMetadata.author),
};
const media = collectMediaFromMarkdown(markdown);
if (metadata.coverImage) {
media.unshift({
url: metadata.coverImage,
kind: "image",
role: "cover",
});
}
return {
metadata,
markdown: trimKnownBoilerplate(markdown),
rawHtml,
cleanedHtml,
media,
conversionMethod: article?.content ? "legacy:readability" : "legacy:body",
};
}
export async function convertHtmlToMarkdown(
html: string,
url: string,
options: ConvertHtmlToMarkdownOptions = {},
): Promise<HtmlToMarkdownResult> {
const capturedAt = new Date().toISOString();
const baseMetadata = extractPageMetadata(html, url, capturedAt);
let cleanedHtml = html;
try {
cleanedHtml = cleanHtml(html, url, {
removeBase64Images: !options.preserveBase64Images,
});
} catch {
cleanedHtml = html;
}
let selectedResult: HtmlToMarkdownResult;
const defuddleResult = await tryDefuddleConversion(cleanedHtml, url, baseMetadata);
if (defuddleResult.ok) {
if (shouldCompareWithFallback(defuddleResult.result.markdown)) {
const fallbackResult = convertWithReadability(html, cleanedHtml, url, baseMetadata);
if (
scoreMarkdownQuality(fallbackResult.markdown) >
scoreMarkdownQuality(defuddleResult.result.markdown) + LOCAL_FALLBACK_SCORE_DELTA
) {
selectedResult = {
...fallbackResult,
fallbackReason: "Readability/Turndown produced higher-quality markdown than Defuddle",
};
} else {
selectedResult = {
...defuddleResult.result,
rawHtml: html,
cleanedHtml,
};
}
} else {
selectedResult = {
...defuddleResult.result,
rawHtml: html,
cleanedHtml,
};
}
} else {
selectedResult = {
...convertWithReadability(html, cleanedHtml, url, baseMetadata),
fallbackReason: defuddleResult.reason,
};
}
if (!shouldTryRemoteMarkdownFallback(selectedResult.markdown, cleanedHtml, options)) {
return selectedResult;
}
const remoteDefuddleResult = await tryDefuddleApiConversion(cleanedHtml, url, baseMetadata);
if (!remoteDefuddleResult.ok || !shouldPreferRemoteMarkdown(selectedResult, remoteDefuddleResult.result, cleanedHtml)) {
return selectedResult;
}
return {
...remoteDefuddleResult.result,
rawHtml: html,
cleanedHtml,
fallbackReason: buildRemoteFallbackReason(selectedResult, cleanedHtml),
};
}
@@ -0,0 +1,169 @@
import TurndownService from "turndown";
import { gfm } from "turndown-plugin-gfm";
import { normalizeMarkdownMediaLinks } from "../media/markdown-media";
import type { ContentBlock, ExtractedDocument } from "./document";
const turndownService = new TurndownService({
codeBlockStyle: "fenced",
headingStyle: "atx",
bulletListMarker: "-",
});
turndownService.use(gfm);
function renderBlock(block: ContentBlock): string {
switch (block.type) {
case "paragraph":
return block.text.trim();
case "heading":
return `${"#".repeat(Math.min(Math.max(block.depth, 1), 6))} ${block.text.trim()}`;
case "list":
return block.items
.map((item, index) => (block.ordered ? `${index + 1}. ${item.trim()}` : `- ${item.trim()}`))
.join("\n");
case "quote":
return block.text
.split("\n")
.map((line) => `> ${line}`)
.join("\n");
case "code":
return `\`\`\`${block.language ?? ""}\n${block.code.trimEnd()}\n\`\`\``;
case "image":
return `![${block.alt ?? ""}](${block.url})`;
case "html":
return turndownService.turndown(block.html).trim();
case "markdown":
return block.markdown.trim();
}
}
function isDefinedValue(value: unknown): boolean {
return value !== undefined && value !== null && value !== "";
}
function renderFrontmatterValue(value: unknown): string {
if (typeof value === "string") {
if (value.includes("\n")) {
return `|-\n${value
.replace(/\r\n/g, "\n")
.split("\n")
.map((line) => ` ${line}`)
.join("\n")}`;
}
return JSON.stringify(value);
}
if (typeof value === "number" || typeof value === "boolean") {
return String(value);
}
return JSON.stringify(value);
}
function renderFrontmatter(document: ExtractedDocument): string {
const fields = new Map<string, unknown>();
const preferredOrder = [
"title",
"url",
"requestedUrl",
"author",
"authorName",
"authorUsername",
"authorUrl",
"coverImage",
"siteName",
"publishedAt",
"summary",
"adapter",
];
fields.set("title", document.title);
fields.set("url", document.canonicalUrl ?? document.url);
fields.set("requestedUrl", document.requestedUrl ?? document.url);
fields.set("author", document.author);
fields.set("siteName", document.siteName);
fields.set("publishedAt", document.publishedAt);
fields.set("summary", document.summary);
fields.set("adapter", document.adapter);
for (const [key, value] of Object.entries(document.metadata ?? {})) {
if (!fields.has(key)) {
fields.set(key, value);
}
}
const orderedKeys = [
...preferredOrder.filter((key) => fields.has(key)),
...Array.from(fields.keys()).filter((key) => !preferredOrder.includes(key)).sort(),
];
const lines = orderedKeys
.map((key) => [key, fields.get(key)] as const)
.filter(([, value]) => isDefinedValue(value))
.map(([key, value]) => `${key}: ${renderFrontmatterValue(value)}`);
if (lines.length === 0) {
return "";
}
return `---\n${lines.join("\n")}\n---`;
}
function cleanMarkdown(markdown: string): string {
return normalizeMarkdownMediaLinks(markdown.replace(/\n{3,}/g, "\n\n").trim());
}
function normalizeComparableTitle(value: string): string {
return value
.trim()
.toLowerCase()
.replace(/^>\s*/, "")
.replace(/^#+\s+/, "")
.replace(/(?:\.{3}|…)\s*$/, "");
}
function bodyStartsWithTitle(body: string, title: string): boolean {
const firstMeaningfulLine = body
.replace(/\r\n/g, "\n")
.split("\n")
.map((line) => line.trim())
.find((line) => line && !/^!?\[[^\]]*\]\([^)]+\)$/.test(line));
if (!firstMeaningfulLine) {
return false;
}
const comparableTitle = normalizeComparableTitle(title);
const comparableFirstLine = normalizeComparableTitle(firstMeaningfulLine);
if (!comparableTitle || !comparableFirstLine) {
return false;
}
return (
comparableFirstLine === comparableTitle ||
comparableFirstLine.startsWith(comparableTitle) ||
comparableTitle.startsWith(comparableFirstLine)
);
}
export function renderMarkdown(document: ExtractedDocument): string {
const sections: string[] = [];
const frontmatter = renderFrontmatter(document);
if (frontmatter) {
sections.push(frontmatter);
}
const body = document.content
.map((block) => renderBlock(block))
.filter(Boolean)
.join("\n\n");
if (document.title && !bodyStartsWithTitle(body, document.title)) {
sections.push(`# ${document.title}`);
}
if (body) {
sections.push(body);
}
return cleanMarkdown(sections.join("\n\n"));
}

Some files were not shown because too many files have changed in this diff Show More