Compare commits

...

43 Commits

Author SHA1 Message Date
Jim Liu 宝玉 c1e1526c84 chore: release v1.63.0 2026-03-13 00:24:32 -05:00
Jim Liu 宝玉 0279fa403d feat(baoyu-url-to-markdown): add defuddle.md API fallback, YouTube transcripts, and modular converter architecture 2026-03-13 00:22:03 -05:00
Jim Liu 宝玉 994e47d1be docs: fix Claude Code marketplace repo casing 2026-03-12 12:46:25 -05:00
Jim Liu 宝玉 6a71d5080d chore: release v1.62.0 2026-03-12 01:51:18 -05:00
Jim Liu 宝玉 1e6e6637ac feat(baoyu-infographic): support flexible aspect ratios
Accept custom W:H ratios (e.g., 3:4, 4:3, 2.35:1) in addition to
named presets (landscape/portrait/square).
2026-03-12 01:50:36 -05:00
Jim Liu 宝玉 873b60aee5 docs(wechat): replace credential-like placeholders 2026-03-12 00:53:33 -05:00
Jim Liu 宝玉 b88ac59133 fix: set strict mode on plugins to prevent duplicated slash commands
Closes #79
2026-03-11 23:29:27 -05:00
Jim Liu 宝玉 d889c04084 chore: release v1.61.0 2026-03-11 23:24:43 -05:00
Jim Liu 宝玉 5276fae6bd feat(baoyu-post-to-wechat): add multi-account support
Support multiple WeChat Official Accounts via EXTEND.md accounts block.
Each account gets isolated Chrome profile, credential resolution chain
(inline → prefixed env → unprefixed env), and --account CLI arg.
2026-03-11 23:24:02 -05:00
Jim Liu 宝玉 c0941f8089 fix: exclude out/dist/build dirs and bun.lockb from skill release files 2026-03-11 22:55:47 -05:00
Jim Liu 宝玉 42b8b1fc99 fix: use proper MIME types in skill publish to fix ClawhHub rejection 2026-03-11 22:46:06 -05:00
Jim Liu 宝玉 53c788eb3b chore: release v1.60.0 2026-03-11 22:22:03 -05:00
Jim Liu 宝玉 c79066e96e fix(baoyu-post-to-x): add missing fs import in x-article 2026-03-11 22:18:11 -05:00
Jim Liu 宝玉 05dba5c320 refactor: publish skills directly from synced vendor 2026-03-11 21:36:19 -05:00
Jim Liu 宝玉 270a9af804 refactor: simplify CLAUDE.md, move detailed docs to docs/
CLAUDE.md reduced from 621 to 76 lines (↓88%). Detailed guidelines
moved to on-demand reference files under docs/.
2026-03-11 21:01:59 -05:00
Jim Liu 宝玉 3bba18c1fe build: commit vendored shared skill packages 2026-03-11 20:45:25 -05:00
Jim Liu 宝玉 069c5dc7d7 refactor: unify skill cdp and release artifacts 2026-03-11 19:38:59 -05:00
Jim Liu 宝玉 00bf946403 支持复用已有 Chrome CDP 实例,修复端口检测顺序问题 2026-03-11 17:24:18 -05:00
Jim Liu 宝玉 1cb54420e0 chore: release v1.59.1 2026-03-11 17:08:33 -05:00
Jim Liu 宝玉 6363bd83e2 refactor(scripts): replace clawhub CLI with local sync-clawhub.mjs 2026-03-11 16:45:57 -05:00
Jim Liu 宝玉 7b8247544d refactor(baoyu-post-to-weibo): add entry point guard to md-to-html.ts 2026-03-11 16:45:56 -05:00
Jim Liu 宝玉 a9576ebc67 fix(baoyu-post-to-x): remove disable-blink-features Chrome flag 2026-03-11 16:45:55 -05:00
Jim Liu 宝玉 79ca378229 fix(baoyu-translate): improve annotation density and style preset passing
- Add short text (<5 sentences) annotation reduction rule to translator's notes
- Explicitly pass resolved style preset to 02-prompt.md assembly in all modes
2026-03-11 16:45:51 -05:00
Jim Liu 宝玉 e1a1fe23cb Merge pull request #77 from jackL1020/claude/claude-md-mmlh0po6953zhhbf-D8rFo
docs: Update CLAUDE.md with v1.59.0 features and skill registry
2026-03-11 13:40:27 -05:00
Claude 3d85a7e663 docs: fix marketplace.json path (it's in .claude-plugin/, not root)
https://claude.ai/code/session_01DREj1A7KKhNVE1RXjmg9ZH
2026-03-11 03:28:56 +00:00
Claude 10aabb39f8 docs: update CLAUDE.md to reflect v1.59.0 codebase state
- Add 8 missing skills to architecture tree (baoyu-post-to-weibo,
  baoyu-infographic, baoyu-image-gen, baoyu-url-to-markdown,
  baoyu-format-markdown, baoyu-markdown-to-html, baoyu-translate)
- Fix marketplace.json path (root, not .claude-plugin/)
- Rename SKILL_DIR → baseDir in Script Directory template (v1.57.0 change)
- Add ClawHub/OpenClaw publishing section with sync-clawhub.sh usage
- Add openclaw frontmatter template and requirement for new skills
- Document baoyu-image-gen batch parallel generation capability
- Update Project Overview to mention all AI providers
- Expand Key Dependencies and Choosing a Category tables
- Add image-gen API key requirement note

https://claude.ai/code/session_01DREj1A7KKhNVE1RXjmg9ZH
2026-03-11 03:28:14 +00:00
Jim Liu 宝玉 661c74cce4 docs: add contributor credit to v1.59.0 changelog 2026-03-09 00:58:14 -05:00
Jim Liu 宝玉 82445f2a39 chore: release v1.59.0 2026-03-09 00:55:52 -05:00
Jim Liu 宝玉 df9ce95c46 fix: restore Google as default provider when multiple keys available
EXTEND.md default_provider still takes priority over auto-detect.
Priority: CLI --provider > EXTEND.md default_provider > auto-detect (Google first).
2026-03-09 00:52:53 -05:00
Jim Liu 宝玉 88b433d565 docs: improve skill documentation clarity
- Fix Provider Selection: default to Replicate when multiple keys available (matches code)
- Add Replicate column to Quality Presets table (normal→1K, 2k→2K)
- Add Replicate aspect ratio behavior (match_input_image when --ref without --ar)
- Remove stale Google Imagen reference from Aspect Ratios
- Add batch file format example with JSON schema to SKILL.md
- Note that batch paths resolve relative to batch file directory
- Move batch execution strategy in article-illustrator before numbered steps
- Fix translate image-language reminder to use standard markdown syntax
  with note to match article's own image syntax convention
2026-03-09 00:43:05 -05:00
Jim Liu 宝玉 5acef7151b feat: add batch parallel image generation and provider-level throttling
- Add --batchfile and --jobs flags for multi-image parallel generation
  with per-provider concurrency control and rate limiting
- Refactor main.ts into prepareSingleTask/prepareBatchTasks/runBatchTasks
  with worker pool pattern and up to 3 retries per image
- Fix Replicate provider: use image_input array (nano-banana-pro schema),
  add match_input_image aspect ratio, add quality-to-resolution mapping
- Improve OpenAI error message for missing API key (Codex auth hint)
- Expand non-retryable error detection (4xx codes, disabled models)
- Add batch config to EXTEND.md schema (max_workers, provider_limits)
- Add build-batch.ts for article-illustrator batch workflow integration
- Add image-language awareness pass to baoyu-translate

Co-authored-by: 敖氏 <aoshi@MacBook-Air.local>
2026-03-09 00:07:45 -05:00
Jim Liu 宝玉 e4d4ec8334 chore: release v1.58.0 2026-03-08 22:30:46 -05:00
Jim Liu 宝玉 4935dc8b1a fix: surface agent-browser startup errors 2026-03-08 22:25:34 -05:00
Jim Liu 宝玉 c43ed57ffc Merge pull request #69 from liby/feat/xdg-config-path
feat: add XDG config path support for EXTEND.md
2026-03-08 22:19:52 -05:00
Jim Liu 宝玉 d96368492e Merge pull request #66 from luojiyin1987/fix/harden-command-exec-and-js-escaping
fix: harden command execution and JS literal escaping
2026-03-08 22:10:45 -05:00
Jim Liu 宝玉 cfc647fd38 Merge pull request #68 from luojiyin1987/docs/sync-readme-and-claude-deps
docs: sync README utility skills and CLAUDE dependency notes
2026-03-08 22:04:40 -05:00
Bryan Lee a702513487 feat: add XDG config path support for EXTEND.md
Add $XDG_CONFIG_HOME/baoyu-skills/<skill>/EXTEND.md as second priority
path between project-level and legacy $HOME/.baoyu-skills/ paths.

Priority order:
1. .baoyu-skills/<skill>/EXTEND.md (project)
2. $XDG_CONFIG_HOME/baoyu-skills/<skill>/EXTEND.md (XDG, new)
3. $HOME/.baoyu-skills/<skill>/EXTEND.md (legacy fallback)

Updated: CLAUDE.md template, 16 SKILL.md files, 2 workflow.md
references, and 2 extend-config.ts runtime loaders.
2026-03-09 04:54:26 +08:00
luojiyin 6b2b28b421 docs: fix CLAUDE dependency statement 2026-03-08 21:21:54 +08:00
luojiyin 0df63c3a0b docs: add markdown-to-html to README.zh utility skills 2026-03-08 21:21:54 +08:00
luojiyin 7834d632e0 docs: add markdown-to-html to README utility skills 2026-03-08 21:21:47 +08:00
luojiyin ca9a0a1404 fix: harden wechat agent-browser command and eval handling 2026-03-08 20:34:43 +08:00
luojiyin 366e7b5403 fix: use execFileSync for google curl requests 2026-03-08 20:34:39 +08:00
luojiyin 709e026be1 fix: use spawnSync for autocorrect command 2026-03-08 20:34:35 +08:00
101 changed files with 8434 additions and 3307 deletions
+4 -4
View File
@@ -6,14 +6,14 @@
}, },
"metadata": { "metadata": {
"description": "Skills shared by Baoyu for improving daily work efficiency", "description": "Skills shared by Baoyu for improving daily work efficiency",
"version": "1.57.0" "version": "1.63.0"
}, },
"plugins": [ "plugins": [
{ {
"name": "content-skills", "name": "content-skills",
"description": "Content generation and publishing skills", "description": "Content generation and publishing skills",
"source": "./", "source": "./",
"strict": false, "strict": true,
"skills": [ "skills": [
"./skills/baoyu-xhs-images", "./skills/baoyu-xhs-images",
"./skills/baoyu-post-to-x", "./skills/baoyu-post-to-x",
@@ -30,7 +30,7 @@
"name": "ai-generation-skills", "name": "ai-generation-skills",
"description": "AI-powered generation backends", "description": "AI-powered generation backends",
"source": "./", "source": "./",
"strict": false, "strict": true,
"skills": [ "skills": [
"./skills/baoyu-danger-gemini-web", "./skills/baoyu-danger-gemini-web",
"./skills/baoyu-image-gen" "./skills/baoyu-image-gen"
@@ -40,7 +40,7 @@
"name": "utility-skills", "name": "utility-skills",
"description": "Utility tools for content processing", "description": "Utility tools for content processing",
"source": "./", "source": "./",
"strict": false, "strict": true,
"skills": [ "skills": [
"./skills/baoyu-danger-x-to-markdown", "./skills/baoyu-danger-x-to-markdown",
"./skills/baoyu-compress-image", "./skills/baoyu-compress-image",
+29
View File
@@ -35,6 +35,7 @@ Just run `/release-skills` - auto-detects your project configuration.
### Step 1: Detect Project Configuration ### Step 1: Detect Project Configuration
1. Check for `.releaserc.yml` (optional config override) 1. Check for `.releaserc.yml` (optional config override)
- If present, inspect whether it defines release hooks
2. Auto-detect version file by scanning (priority order): 2. Auto-detect version file by scanning (priority order):
- `package.json` (Node.js) - `package.json` (Node.js)
- `pyproject.toml` (Python) - `pyproject.toml` (Python)
@@ -48,6 +49,34 @@ Just run `/release-skills` - auto-detects your project configuration.
4. Identify language of each changelog by filename suffix 4. Identify language of each changelog by filename suffix
5. Display detected configuration 5. Display detected configuration
**Project Hook Contract**:
If `.releaserc.yml` defines `release.hooks`, keep the release workflow generic and delegate project-specific packaging/publishing to those hooks.
Supported hooks:
| Hook | Purpose | Expected Responsibility |
|------|---------|-------------------------|
| `prepare_artifact` | Make one target releasable | Validate the target is self-contained, sync/embed local dependencies, optionally stage extra files |
| `publish_artifact` | Publish one releasable target | Upload the prepared target (or a staged directory if the project uses one), attach version/changelog/tags |
Supported placeholders:
| Placeholder | Meaning |
|-------------|---------|
| `{project_root}` | Absolute path to repository root |
| `{target}` | Absolute path to the module/skill being released |
| `{artifact_dir}` | Absolute path to a temporary staging directory for this target, when the project uses one |
| `{version}` | Version selected by the release workflow |
| `{dry_run}` | `true` or `false` |
| `{release_notes_file}` | Absolute path to a UTF-8 file containing release notes/changelog text |
Execution rules:
- Keep the skill generic: do not hardcode registry/package-manager/project layout details into this SKILL.
- If `prepare_artifact` exists, run it once per target before publish-related checks that need the final releasable target state.
- Write release notes to a temp file and pass that file path to `publish_artifact`; do not inline multiline changelog text into shell commands.
- If hooks are absent, fall back to the default project-agnostic release workflow.
**Language Detection Rules**: **Language Detection Rules**:
Changelog files follow the pattern `CHANGELOG_{LANG}.md` or `CHANGELOG.{lang}.md`, where `{lang}` / `{LANG}` is a language or region code. Changelog files follow the pattern `CHANGELOG_{LANG}.md` or `CHANGELOG.{lang}.md`, where `{lang}` / `{LANG}` is a language or region code.
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
REPO_ROOT=$(git rev-parse --show-toplevel)
cd "$REPO_ROOT"
node scripts/sync-shared-skill-packages.mjs --repo-root "$REPO_ROOT" --enforce-clean
+2
View File
@@ -164,3 +164,5 @@ posts/
# ClawHub local state (current and legacy directory names from the official CLI) # ClawHub local state (current and legacy directory names from the official CLI)
.clawhub/ .clawhub/
.clawdhub/ .clawdhub/
.release-artifacts/
.worktrees/
+6
View File
@@ -0,0 +1,6 @@
release:
target_globs:
- skills/*
hooks:
prepare_artifact: node scripts/sync-shared-skill-packages.mjs --repo-root "{project_root}" --target "{target}"
publish_artifact: node scripts/publish-skill.mjs --skill-dir "{target}" --version "{version}" --changelog-file "{release_notes_file}" --dry-run "{dry_run}"
+86
View File
@@ -2,6 +2,92 @@
English | [中文](./CHANGELOG.zh.md) English | [中文](./CHANGELOG.zh.md)
## 1.63.0 - 2026-03-13
### Features
- `baoyu-url-to-markdown`: add hosted `defuddle.md` API fallback when local browser capture fails
- `baoyu-url-to-markdown`: extract YouTube transcript/caption text into markdown output
- `baoyu-url-to-markdown`: materialize shadow DOM content for better web-component page conversion
- `baoyu-url-to-markdown`: include language hint in markdown front matter when available
### Refactor
- `baoyu-url-to-markdown`: split monolithic converter into defuddle, legacy, and shared modules
### Documentation
- Fix Claude Code marketplace repo casing in READMEs
## 1.62.0 - 2026-03-12
### Features
- `baoyu-infographic`: support flexible aspect ratios with custom W:H values (e.g., 3:4, 4:3, 2.35:1) in addition to named presets
### Fixes
- Set strict mode on plugins to prevent duplicated slash commands
### Documentation
- `baoyu-post-to-wechat`: replace credential-like placeholders
## 1.61.0 - 2026-03-11
### Features
- `baoyu-post-to-wechat`: add multi-account support with `--account` CLI arg, EXTEND.md accounts block, isolated Chrome profiles, and credential resolution chain
### Fixes
- Exclude `out/dist/build` dirs and `bun.lockb` from skill release files
- Use proper MIME types in skill publish to fix ClawhHub rejection
## 1.60.0 - 2026-03-11
### Features
- `baoyu-url-to-markdown`: support reusing existing Chrome CDP instances and fix port detection order
### Fixes
- `baoyu-post-to-x`: add missing `fs` import in x-article
### Refactor
- Unify all CDP skills to use shared `baoyu-chrome-cdp` package with vendored copies
- Simplify CLAUDE.md, move detailed documentation to `docs/` directory
- Publish skills directly from synced vendor, removing separate artifact preparation step
## 1.59.1 - 2026-03-11
### Fixes
- `baoyu-translate`: improve short text annotation density rule and add explicit style preset passing to 02-prompt.md
- `baoyu-post-to-x`: remove `--disable-blink-features=AutomationControlled` Chrome flag
### Refactor
- `baoyu-post-to-weibo`: add entry point guard to md-to-html.ts for module import compatibility
- Replace clawhub CLI with local sync-clawhub.mjs script
### Documentation
- Update CLAUDE.md to reflect v1.59.0 codebase state (by @jackL1020)
## 1.59.0 - 2026-03-09
### Features
- `baoyu-image-gen`: add batch parallel image generation and provider-level throttling (by @SeamoonAO)
### Fixes
- `baoyu-image-gen`: restore Google as default provider when multiple keys available
### Documentation
- Improve skill documentation clarity (by @SeamoonAO)
## 1.58.0 - 2026-03-08
### Features
- Add XDG config path support for EXTEND.md (by @liby)
### Fixes
- `baoyu-post-to-wechat`: surface agent-browser startup errors
- `baoyu-post-to-wechat`: harden agent-browser command and eval handling (by @luojiyin1987)
- `baoyu-image-gen`: use execFileSync for google curl requests (by @luojiyin1987)
- `baoyu-format-markdown`: use spawnSync for autocorrect command (by @luojiyin1987)
### Documentation
- Fix CLAUDE dependency statement (by @luojiyin1987)
- Add markdown-to-html to README utility skills (by @luojiyin1987)
## 1.57.0 - 2026-03-08 ## 1.57.0 - 2026-03-08
### Features ### Features
+86
View File
@@ -2,6 +2,92 @@
[English](./CHANGELOG.md) | 中文 [English](./CHANGELOG.md) | 中文
## 1.63.0 - 2026-03-13
### 新功能
- `baoyu-url-to-markdown`:本地浏览器抓取失败时自动回退到 `defuddle.md` 托管 API
- `baoyu-url-to-markdown`:将 YouTube 字幕/文字记录提取到 Markdown 输出中
- `baoyu-url-to-markdown`:转换前展开 Shadow DOM 内容,提升 Web Component 页面的转换质量
- `baoyu-url-to-markdown`Markdown front matter 中包含语言标识(如有)
### 重构
- `baoyu-url-to-markdown`:将单体转换器拆分为 defuddle、legacy 和 shared 三个模块
### 文档
- 修复 README 中 Claude Code marketplace 仓库名大小写
## 1.62.0 - 2026-03-12
### 新功能
- `baoyu-infographic`:支持灵活宽高比,可使用自定义 W:H 值(如 3:4、4:3、2.35:1),同时保留预设名称
### 修复
- 设置插件严格模式,防止重复注册斜杠命令
### 文档
- `baoyu-post-to-wechat`:替换类似凭证的占位符
## 1.61.0 - 2026-03-11
### 新功能
- `baoyu-post-to-wechat`:新增多账号支持,通过 `--account` 参数选择账号,EXTEND.md 支持 accounts 配置块,每个账号独立 Chrome 配置目录和凭证解析链
### 修复
- 排除 `out/dist/build` 目录和 `bun.lockb` 文件,避免打包到技能发布文件中
- 修复技能发布时 MIME 类型不正确导致 ClawhHub 拒绝的问题
## 1.60.0 - 2026-03-11
### 新功能
- `baoyu-url-to-markdown`:支持复用已有 Chrome CDP 实例,修复端口检测顺序问题
### 修复
- `baoyu-post-to-x`:补充 x-article 缺失的 `fs` 导入
### 重构
- 统一所有 CDP 技能使用共享 `baoyu-chrome-cdp` 包,各技能内置 vendor 副本
- 精简 CLAUDE.md,将详细文档移至 `docs/` 目录
- 从 synced vendor 直接发布技能,移除单独的 artifact 准备步骤
## 1.59.1 - 2026-03-11
### 修复
- `baoyu-translate`:改进短文本注释密度规则,补充风格预设到 02-prompt.md 的显式传递
- `baoyu-post-to-x`:移除 `--disable-blink-features=AutomationControlled` Chrome 启动参数
### 重构
- `baoyu-post-to-weibo`:为 md-to-html.ts 添加入口守卫,支持模块导入
- 使用本地 sync-clawhub.mjs 脚本替代 clawhub CLI
### 文档
- 更新 CLAUDE.md 以反映 v1.59.0 代码库状态 (by @jackL1020)
## 1.59.0 - 2026-03-09
### 新功能
- `baoyu-image-gen`:新增批量并行图片生成和提供商级别限流 (by @SeamoonAO)
### 修复
- `baoyu-image-gen`:修复多个 API key 可用时恢复 Google 为默认提供商
### 文档
- 改进技能文档清晰度 (by @SeamoonAO)
## 1.58.0 - 2026-03-08
### 新功能
- 新增 EXTEND.md 的 XDG 配置路径支持 (by @liby)
### 修复
- `baoyu-post-to-wechat`:暴露 agent-browser 启动错误信息
- `baoyu-post-to-wechat`:加固 agent-browser 命令和 eval 处理 (by @luojiyin1987)
- `baoyu-image-gen`:使用 execFileSync 替代 shell 执行 Google curl 请求 (by @luojiyin1987)
- `baoyu-format-markdown`:使用 spawnSync 替代 shell 执行 autocorrect 命令 (by @luojiyin1987)
### 文档
- 修正 CLAUDE 依赖说明 (by @luojiyin1987)
- 将 markdown-to-html 添加到 README 工具技能列表 (by @luojiyin1987)
## 1.57.0 - 2026-03-08 ## 1.57.0 - 2026-03-08
### 新功能 ### 新功能
+37 -488
View File
@@ -1,527 +1,76 @@
# CLAUDE.md # CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Claude Code marketplace plugin providing AI-powered content generation skills. Version: **1.63.0**.
## Project Overview
Claude Code marketplace plugin providing AI-powered content generation skills. Skills use Gemini Web API (reverse-engineered) for text/image generation and Chrome CDP for browser automation.
## Architecture ## Architecture
Skills are organized into three plugin categories in `marketplace.json`: Skills organized into three categories in `.claude-plugin/marketplace.json` (defines plugin metadata, version, and skill paths):
```
skills/
├── [content-skills] # Content generation and publishing
│ ├── baoyu-xhs-images/ # Xiaohongshu infographic series (1-10 images)
│ ├── baoyu-cover-image/ # Article cover images (2.35:1 aspect)
│ ├── baoyu-slide-deck/ # Presentation slides with outlines
│ ├── baoyu-article-illustrator/ # Smart illustration placement
│ ├── baoyu-comic/ # Knowledge comics (Logicomix/Ohmsha style)
│ ├── baoyu-post-to-x/ # X/Twitter posting automation
│ └── baoyu-post-to-wechat/ # WeChat Official Account posting
├── [ai-generation-skills] # AI-powered generation backends
│ └── baoyu-danger-gemini-web/ # Gemini API wrapper (text + image gen)
└── [utility-skills] # Utility tools for content processing
├── baoyu-danger-x-to-markdown/ # X/Twitter content to markdown
└── baoyu-compress-image/ # Image compression
```
**Plugin Categories**:
| Category | Description | | Category | Description |
|----------|-------------| |----------|-------------|
| `content-skills` | Skills that generate or publish content (images, slides, comics, posts) | | `content-skills` | Generate or publish content (images, slides, comics, posts) |
| `ai-generation-skills` | Backend skills providing AI generation capabilities | | `ai-generation-skills` | AI generation backends |
| `utility-skills` | Helper tools for content processing (conversion, compression) | | `utility-skills` | Content processing (conversion, compression, translation) |
Each skill contains: Each skill contains `SKILL.md` (YAML front matter + docs), optional `scripts/`, `references/`, `prompts/`.
- `SKILL.md` - YAML front matter (name, description) + documentation
- `scripts/` - TypeScript implementations Top-level `scripts/` contains repo maintenance utilities (sync, hooks, publish).
- `prompts/system.md` - AI generation guidelines (optional)
## Running Skills ## Running Skills
All scripts are TypeScript, executed via Bun runtime (no build step). TypeScript via Bun (no build step). Detect runtime once per session:
### Runtime Detection (`${BUN_X}`)
Before running any script, the agent MUST detect the runtime **once per session** and set `${BUN_X}`:
```bash ```bash
# Detect runtime (run once, reuse result) if command -v bun &>/dev/null; then BUN_X="bun"
if command -v bun &>/dev/null; then elif command -v npx &>/dev/null; then BUN_X="npx -y bun"
BUN_X="bun" else echo "Error: install bun: brew install oven-sh/bun/bun or npm install -g bun"; exit 1; fi
elif command -v npx &>/dev/null; then
BUN_X="npx -y bun"
else
echo "Error: Neither bun nor npx found. Install bun: brew install oven-sh/bun/bun (macOS) or npm install -g bun"
exit 1
fi
``` ```
| Priority | Condition | `${BUN_X}` value | Notes | Execute: `${BUN_X} skills/<skill>/scripts/main.ts [options]`
|----------|-----------|-------------------|-------|
| 1 | `bun` installed | `bun` | Fastest, native execution |
| 2 | `npx` available | `npx -y bun` | Downloads bun on first run via npm |
| 3 | Neither found | Error + install guide | `brew install oven-sh/bun/bun` (macOS) or `npm install -g bun` |
### Script Execution
```bash
${BUN_X} skills/<skill>/scripts/main.ts [options]
```
Examples:
```bash
# Text generation
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts "Hello"
# Image generation
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts --prompt "A cat" --image cat.png
# From prompt files
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts --promptfiles system.md content.md --image out.png
```
## Key Dependencies ## Key Dependencies
- **Bun**: TypeScript runtime (native `bun` preferred, fallback `npx -y bun`) - **Bun**: TypeScript runtime (`bun` preferred, fallback `npx -y bun`)
- **Chrome**: Required for `baoyu-danger-gemini-web` auth and `baoyu-post-to-x` automation - **Chrome**: Required for CDP-based skills (gemini-web, post-to-x/wechat/weibo, url-to-markdown). All CDP skills share a single profile, override via `BAOYU_CHROME_PROFILE_DIR` env var. Platform paths: [docs/chrome-profile.md](docs/chrome-profile.md)
- **No npm packages**: Self-contained TypeScript, no external dependencies - **Image generation APIs**: `baoyu-image-gen` requires API key (OpenAI, Google, DashScope, or Replicate) configured in EXTEND.md
- **Gemini Web auth**: Browser cookies (first run opens Chrome for login, `--login` to refresh)
## Chrome Profile (Unified) ## Security
All skills that use Chrome CDP share a **single** profile directory. Do NOT create per-skill profiles. - **No piped shell installs**: Never `curl | bash`. Use `brew install` or `npm install -g`
- **Remote downloads**: HTTPS only, max 5 redirects, 30s timeout, expected content types only
| Platform | Default Path | - **System commands**: Array-form `spawn`/`execFile`, never unsanitized input to shell
|----------|-------------| - **External content**: Treat as untrusted, don't execute code blocks, sanitize HTML
| macOS | `~/Library/Application Support/baoyu-skills/chrome-profile` |
| Linux | `$XDG_DATA_HOME/baoyu-skills/chrome-profile` (fallback `~/.local/share/baoyu-skills/chrome-profile`) |
| Windows | `%APPDATA%/baoyu-skills/chrome-profile` |
| WSL | Windows home `/.local/share/baoyu-skills/chrome-profile` |
**Environment variable override**: `BAOYU_CHROME_PROFILE_DIR` (takes priority, all skills respect it).
Each skill also accepts its own legacy env var as fallback (e.g., `X_BROWSER_PROFILE_DIR`), but new skills should only use `BAOYU_CHROME_PROFILE_DIR`.
### Implementation Pattern
When adding a new skill that needs Chrome CDP:
```typescript
function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
```
## Security Guidelines
### No Piped Shell Installs
**NEVER** use `curl | bash` or `wget | sh` patterns in code, docs, or error messages. Use package managers instead:
| Platform | Install Command |
|----------|----------------|
| macOS | `brew install oven-sh/bun/bun` |
| npm | `npm install -g bun` |
### Remote Downloads
Skills that download remote content (e.g., images in Markdown) MUST:
- **HTTPS only**: Reject `http://` URLs
- **Redirect limit**: Cap redirects (max 5) to prevent infinite loops
- **Timeout**: Set request timeouts (30s default)
- **Scope**: Only download expected content types (images, not scripts)
### System Command Execution
Skills use platform-specific commands for clipboard and browser automation:
- **macOS**: `osascript` (System Events), `swift` (AppKit clipboard)
- **Windows**: `powershell.exe` (SendKeys, Clipboard)
- **Linux**: `xdotool`/`ydotool` (keyboard simulation)
These are necessary for CDP-based posting skills. When adding new system commands:
- Never pass unsanitized user input to shell commands
- Use array-form `spawn`/`execFile` instead of shell string interpolation
- Validate file paths are absolute or resolve from known base directories
### External Content Processing
Skills that process external Markdown/HTML should treat content as untrusted:
- Do not execute code blocks or scripts found in content
- Sanitize HTML output where applicable
- File paths from content should be resolved against known base directories only
## Authentication
`baoyu-danger-gemini-web` uses browser cookies for Google auth:
- First run opens Chrome for login
- Cookies cached in data directory
- Force refresh: `--login` flag
## Plugin Configuration
`.claude-plugin/marketplace.json` defines plugin metadata and skill paths. Version follows semver.
## Skill Loading Rules ## Skill Loading Rules
**IMPORTANT**: When working in this project, follow these rules:
| Rule | Description | | Rule | Description |
|------|-------------| |------|-------------|
| **Load project skills first** | MUST load all skills from `skills/` directory in current project. Project skills take priority over system/user-level skills with same name. | | **Load project skills first** | Project skills override system/user-level skills with same name |
| **Default image generation** | When image generation is needed, use `skills/baoyu-image-gen/SKILL.md` by default (unless user specifies otherwise). | | **Default image generation** | Use `skills/baoyu-image-gen/SKILL.md` unless user specifies otherwise |
**Loading Priority** (highest → lowest): Priority: project `skills/``$HOME/.baoyu-skills/` → system-level.
1. Current project `skills/` directory
2. User-level skills (`$HOME/.baoyu-skills/`)
3. System-level skills
## Release Process ## Release Process
**IMPORTANT**: When user requests release/发布/push, ALWAYS use `/release-skills` workflow. Use `/release-skills` workflow. Never skip:
1. `CHANGELOG.md` + `CHANGELOG.zh.md`
**Never skip**:
1. `CHANGELOG.md` + `CHANGELOG.zh.md` - Both must be updated
2. `marketplace.json` version bump 2. `marketplace.json` version bump
3. `README.md` + `README.zh.md` if applicable 3. `README.md` + `README.zh.md` if applicable
4. All files committed together before tag 4. All files committed together before tag
## Adding New Skills
**IMPORTANT**: All skills MUST use `baoyu-` prefix to avoid conflicts when users import this plugin.
**REQUIRED READING**: Before creating a new skill, read the official [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices).
### Key Requirements from Official Best Practices
| Requirement | Details |
|-------------|---------|
| **Concise is key** | Claude is smart—only add context it doesn't have. Challenge each token. |
| **name field** | Max 64 chars, lowercase letters/numbers/hyphens only, no "anthropic"/"claude" |
| **description field** | Max 1024 chars, non-empty, MUST be third person, include what + when to use |
| **SKILL.md body** | Keep under 500 lines; use separate files for additional content |
| **Naming convention** | Gerund form preferred (e.g., `processing-pdfs`), but `baoyu-` prefix required here |
| **References** | Keep one level deep from SKILL.md; avoid nested references |
| **No time-sensitive info** | Avoid dates/versions that become outdated |
### Steps
1. Create `skills/baoyu-<name>/SKILL.md` with YAML front matter
- Directory name: `baoyu-<name>`
- SKILL.md `name` field: `baoyu-<name>`
2. Add TypeScript in `skills/baoyu-<name>/scripts/`
3. Add prompt templates in `skills/baoyu-<name>/prompts/` if needed
4. **Choose the appropriate category** and register in `marketplace.json`:
- `content-skills`: For content generation/publishing (images, slides, posts)
- `ai-generation-skills`: For AI backend capabilities
- `utility-skills`: For helper tools (conversion, compression)
- If none fit, create a new category with descriptive name
5. **Add Script Directory section** to SKILL.md (see template below)
### Choosing a Category
| If your skill... | Use category |
|------------------|--------------|
| Generates visual content (images, slides, comics) | `content-skills` |
| Publishes to platforms (X, WeChat, etc.) | `content-skills` |
| Provides AI generation backend | `ai-generation-skills` |
| Converts or processes content | `utility-skills` |
| Compresses or optimizes files | `utility-skills` |
**Creating a new category**: If the skill doesn't fit existing categories, add a new plugin object to `marketplace.json` with:
- `name`: Descriptive kebab-case name (e.g., `analytics-skills`)
- `description`: Brief description of the category
- `skills`: Array with the skill path
### Writing Effective Descriptions
**MUST write in third person** (not "I can help you" or "You can use this"):
```yaml
# Good
description: Generates Xiaohongshu infographic series from content. Use when user asks for "小红书图片", "XHS images", or "RedNote infographics".
# Bad
description: I can help you create Xiaohongshu images
description: You can use this to generate XHS content
```
Include both **what** the skill does and **when** to use it (triggers/keywords).
### Script Directory Template
Every SKILL.md with scripts MUST include this section after Usage:
```markdown
## Script Directory
**Important**: All scripts are located in the `scripts/` subdirectory of this skill.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `${SKILL_DIR}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
|--------|---------|
| `scripts/main.ts` | Main entry point |
| `scripts/other.ts` | Other functionality |
```
When referencing scripts in workflow sections, use `${BUN_X} ${SKILL_DIR}/scripts/<name>.ts` so agents can resolve the correct runtime and path.
### Progressive Disclosure
For skills with extensive content, use separate reference files:
```
skills/baoyu-example/
├── SKILL.md # Main instructions (<500 lines)
├── references/
│ ├── styles.md # Style definitions (loaded as needed)
│ └── examples.md # Usage examples (loaded as needed)
└── scripts/
└── main.ts # Executable script
```
In SKILL.md, link to reference files (one level deep only):
```markdown
**Available styles**: See [references/styles.md](references/styles.md)
**Examples**: See [references/examples.md](references/examples.md)
```
## Code Style ## Code Style
- TypeScript throughout, no comments TypeScript, no comments, async/await, short variable names, type-safe interfaces.
- Async/await patterns
- Short variable names
- Type-safe interfaces
## Image Generation Guidelines ## Adding New Skills
Skills that require image generation MUST delegate to available image generation skills rather than implementing their own. All skills MUST use `baoyu-` prefix. Details: [docs/creating-skills.md](docs/creating-skills.md)
### Image Generation Skill Selection ## Reference Docs
**Default**: Use `skills/baoyu-image-gen/SKILL.md` (unless user specifies otherwise). | Topic | File |
|-------|------|
1. Read `skills/baoyu-image-gen/SKILL.md` for parameters and capabilities | Image generation guidelines | [docs/image-generation.md](docs/image-generation.md) |
2. If user explicitly requests a different skill, check `skills/` directory for alternatives | Chrome profile platform paths | [docs/chrome-profile.md](docs/chrome-profile.md) |
3. Only ask user to choose when they haven't specified and multiple viable options exist | Comic style maintenance | [docs/comic-style-maintenance.md](docs/comic-style-maintenance.md) |
| ClawHub/OpenClaw publishing | [docs/publishing.md](docs/publishing.md) |
### Generation Flow Template
Use this template when implementing image generation in skills:
```markdown
### Step N: Generate Images
**Skill Selection**:
1. Check available image generation skills (e.g., `baoyu-danger-gemini-web`)
2. Read selected skill's SKILL.md for parameter reference
3. If multiple skills available, ask user to choose
**Generation Flow**:
1. Call selected image generation skill with:
- Prompt file path (or inline prompt)
- Output image path
- Any skill-specific parameters (refer to skill's SKILL.md)
2. Generate images sequentially (one at a time)
3. After each image, output progress: "Generated X/N"
4. On failure, auto-retry once before reporting error
```
### Output Path Convention
Each session creates an independent directory. Even the same source file generates a new directory per session.
**Output Directory**:
```
<skill-suffix>/<topic-slug>/
```
- `<skill-suffix>`: Skill name suffix (e.g., `xhs-images`, `cover-image`, `slide-deck`, `comic`)
- `<topic-slug>`: Generated from content topic (2-4 words, kebab-case)
- Example: `xhs-images/ai-future-trends/`
**Slug Generation**:
1. Extract main topic from content (2-4 words, kebab-case)
2. Example: "Introduction to Machine Learning" → `intro-machine-learning`
**Conflict Resolution**:
If `<skill-suffix>/<topic-slug>/` already exists:
- Append timestamp: `<topic-slug>-YYYYMMDD-HHMMSS`
- Example: `ai-future` exists → `ai-future-20260118-143052`
**Source Files**:
- Copy all sources to `<skill-suffix>/<topic-slug>/` with naming: `source-{slug}.{ext}`
- Multiple sources supported: text, images, files from conversation
- Examples:
- `source-article.md` (main text content)
- `source-reference.png` (image from conversation)
- `source-data.csv` (additional file)
- Original source files remain unchanged
### Image Naming Convention
Image filenames MUST include meaningful slugs for readability:
**Format**: `NN-{type}-[slug].png`
- `NN`: Two-digit sequence number (01, 02, ...)
- `{type}`: Image type (cover, content, page, slide, illustration, etc.)
- `[slug]`: Descriptive kebab-case slug derived from content
**Examples**:
```
01-cover-ai-future.png
02-content-key-benefits.png
03-page-enigma-machine.png
04-slide-architecture-overview.png
```
**Slug Rules**:
- Derived from image purpose or content (kebab-case)
- Must be unique within the output directory
- 2-5 words, concise but descriptive
- When content changes significantly, update slug accordingly
### Best Practices
- Always read the image generation skill's SKILL.md before calling
- Pass parameters exactly as documented in the skill
- Handle failures gracefully with retry logic
- Provide clear progress feedback to user
## Style Maintenance (baoyu-comic)
When adding, updating, or deleting styles for `baoyu-comic`, follow this workflow:
### Adding a New Style
1. **Create style definition**: `skills/baoyu-comic/references/styles/<style-name>.md`
2. **Update SKILL.md**:
- Add style to `--style` options table
- Add auto-selection entry if applicable
3. **Generate showcase image**:
```bash
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts \
--prompt "A single comic book page in <style-name> style showing [appropriate scene]. Features: [style characteristics from style definition]. 3:4 portrait aspect ratio comic page." \
--image screenshots/comic-styles/<style-name>.png
```
4. **Compress to WebP**:
```bash
${BUN_X} skills/baoyu-compress-image/scripts/main.ts screenshots/comic-styles/<style-name>.png
```
5. **Update both READMEs** (`README.md` and `README.zh.md`):
- Add style to `--style` options
- Add row to style description table
- Add image to style preview grid
### Updating an Existing Style
1. **Update style definition**: `skills/baoyu-comic/references/styles/<style-name>.md`
2. **Regenerate showcase image** (if visual characteristics changed):
- Follow steps 3-4 from "Adding a New Style"
3. **Update READMEs** if description changed
### Deleting a Style
1. **Delete style definition**: `skills/baoyu-comic/references/styles/<style-name>.md`
2. **Delete showcase image**: `screenshots/comic-styles/<style-name>.webp`
3. **Update SKILL.md**:
- Remove from `--style` options
- Remove auto-selection entry
4. **Update both READMEs**:
- Remove from `--style` options
- Remove from style description table
- Remove from style preview grid
### Style Preview Grid Format
READMEs use 3-column tables for style previews:
```markdown
| | | |
|:---:|:---:|:---:|
| ![style1](./screenshots/comic-styles/style1.webp) | ![style2](./screenshots/comic-styles/style2.webp) | ![style3](./screenshots/comic-styles/style3.webp) |
| style1 | style2 | style3 |
```
## Extension Support
Every SKILL.md MUST include two parts for extension support:
### 1. Load Preferences Section (in Step 1 or as "Preferences" section)
For skills with workflows, add as Step 1.1. For utility skills, add as "Preferences (EXTEND.md)" section before Usage:
```markdown
**1.1 Load Preferences (EXTEND.md)**
Check EXTEND.md existence (priority order):
\`\`\`bash
# macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/<skill-name>/EXTEND.md && echo "project"
test -f "$HOME/.baoyu-skills/<skill-name>/EXTEND.md" && echo "user"
\`\`\`
\`\`\`powershell
# PowerShell (Windows)
if (Test-Path .baoyu-skills/<skill-name>/EXTEND.md) { "project" }
if (Test-Path "$HOME/.baoyu-skills/<skill-name>/EXTEND.md") { "user" }
\`\`\`
┌────────────────────────────────────────────┬───────────────────┐
│ Path │ Location │
├────────────────────────────────────────────┼───────────────────┤
│ .baoyu-skills/<skill-name>/EXTEND.md │ Project directory │
├────────────────────────────────────────────┼───────────────────┤
│ $HOME/.baoyu-skills/<skill-name>/EXTEND.md │ User home │
└────────────────────────────────────────────┴───────────────────┘
┌───────────┬───────────────────────────────────────────────────────────────────────────┐
│ Result │ Action │
├───────────┼───────────────────────────────────────────────────────────────────────────┤
│ Found │ Read, parse, display summary │
├───────────┼───────────────────────────────────────────────────────────────────────────┤
│ Not found │ Ask user with AskUserQuestion (see references/config/first-time-setup.md) │
└───────────┴───────────────────────────────────────────────────────────────────────────┘
**EXTEND.md Supports**: [List supported configuration options for this skill]
Schema: `references/config/preferences-schema.md`
```
### 2. Extension Support Section (at the end)
Simplified section that references the preferences section:
```markdown
## Extension Support
Custom configurations via EXTEND.md. See **Step 1.1** for paths and supported options.
```
Or for utility skills without workflow steps:
```markdown
## Extension Support
Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.
```
**Notes**:
- Replace `<skill-name>` with actual skill name (e.g., `baoyu-cover-image`)
- Use `$HOME` instead of `~` for cross-platform compatibility (macOS/Linux/WSL/PowerShell)
- Use `test -f` (Bash) or `Test-Path` (PowerShell) for explicit file existence check
- ASCII tables for clear visual formatting
+61 -4
View File
@@ -43,7 +43,7 @@ Publishing to ClawHub releases the published skill under `MIT-0`, per ClawHub's
Run the following command in Claude Code: Run the following command in Claude Code:
```bash ```bash
/plugin marketplace add jimliu/baoyu-skills /plugin marketplace add JimLiu/baoyu-skills
``` ```
### Install Skills ### Install Skills
@@ -76,7 +76,7 @@ Simply tell Claude Code:
|--------|-------------|--------| |--------|-------------|--------|
| **content-skills** | Content generation and publishing | [xhs-images](#baoyu-xhs-images), [infographic](#baoyu-infographic), [cover-image](#baoyu-cover-image), [slide-deck](#baoyu-slide-deck), [comic](#baoyu-comic), [article-illustrator](#baoyu-article-illustrator), [post-to-x](#baoyu-post-to-x), [post-to-wechat](#baoyu-post-to-wechat), [post-to-weibo](#baoyu-post-to-weibo) | | **content-skills** | Content generation and publishing | [xhs-images](#baoyu-xhs-images), [infographic](#baoyu-infographic), [cover-image](#baoyu-cover-image), [slide-deck](#baoyu-slide-deck), [comic](#baoyu-comic), [article-illustrator](#baoyu-article-illustrator), [post-to-x](#baoyu-post-to-x), [post-to-wechat](#baoyu-post-to-wechat), [post-to-weibo](#baoyu-post-to-weibo) |
| **ai-generation-skills** | AI-powered generation backends | [image-gen](#baoyu-image-gen), [danger-gemini-web](#baoyu-danger-gemini-web) | | **ai-generation-skills** | AI-powered generation backends | [image-gen](#baoyu-image-gen), [danger-gemini-web](#baoyu-danger-gemini-web) |
| **utility-skills** | Utility tools for content processing | [url-to-markdown](#baoyu-url-to-markdown), [danger-x-to-markdown](#baoyu-danger-x-to-markdown), [compress-image](#baoyu-compress-image), [format-markdown](#baoyu-format-markdown), [translate](#baoyu-translate) | | **utility-skills** | Utility tools for content processing | [url-to-markdown](#baoyu-url-to-markdown), [danger-x-to-markdown](#baoyu-danger-x-to-markdown), [compress-image](#baoyu-compress-image), [format-markdown](#baoyu-format-markdown), [markdown-to-html](#baoyu-markdown-to-html), [translate](#baoyu-translate) |
## Update Skills ## Update Skills
@@ -169,8 +169,9 @@ Generate professional infographics with 20 layout types and 17 visual styles. An
# Specify both # Specify both
/baoyu-infographic path/to/content.md --layout funnel --style corporate-memphis /baoyu-infographic path/to/content.md --layout funnel --style corporate-memphis
# With aspect ratio # With aspect ratio (named preset or custom W:H)
/baoyu-infographic path/to/content.md --aspect portrait /baoyu-infographic path/to/content.md --aspect portrait
/baoyu-infographic path/to/content.md --aspect 3:4
``` ```
**Options**: **Options**:
@@ -178,7 +179,7 @@ Generate professional infographics with 20 layout types and 17 visual styles. An
|--------|-------------| |--------|-------------|
| `--layout <name>` | Information layout (20 options) | | `--layout <name>` | Information layout (20 options) |
| `--style <name>` | Visual style (17 options, default: craft-handmade) | | `--style <name>` | Visual style (17 options, default: craft-handmade) |
| `--aspect <ratio>` | landscape (16:9), portrait (9:16), square (1:1) | | `--aspect <ratio>` | Named: landscape (16:9), portrait (9:16), square (1:1). Custom: any W:H ratio (e.g., 3:4, 4:3, 2.35:1) |
| `--lang <code>` | Output language (en, zh, ja, etc.) | | `--lang <code>` | Output language (en, zh, ja, etc.) |
**Layouts** (information structure): **Layouts** (information structure):
@@ -581,6 +582,47 @@ To obtain credentials:
**Browser Method** (no API setup needed): Requires Google Chrome. First run opens browser for QR code login (session preserved). **Browser Method** (no API setup needed): Requires Google Chrome. First run opens browser for QR code login (session preserved).
**Multi-Account Support**: Manage multiple WeChat Official Accounts via `EXTEND.md`:
```bash
mkdir -p .baoyu-skills/baoyu-post-to-wechat
```
Create `.baoyu-skills/baoyu-post-to-wechat/EXTEND.md`:
```yaml
# Global settings (shared across all accounts)
default_theme: default
default_color: blue
# Account list
accounts:
- name: My Tech Blog
alias: tech-blog
default: false
default_publish_method: api
default_author: Author Name
need_open_comment: 1
only_fans_can_comment: 0
app_id: your_wechat_app_id
app_secret: your_wechat_app_secret
- name: AI Newsletter
alias: ai-news
default_publish_method: browser
default_author: AI Newsletter
need_open_comment: 1
only_fans_can_comment: 0
```
| Accounts configured | Behavior |
|---------------------|----------|
| No `accounts` block | Single-account mode (backward compatible) |
| 1 account | Auto-select, no prompt |
| 2+ accounts | Prompt to select, or use `--account <alias>` |
| 1 account has `default: true` | Pre-selected as default |
Each account gets an isolated Chrome profile for independent login sessions (browser method). API credentials can be set inline in EXTEND.md or via `.env` with alias-prefixed keys (e.g., `WECHAT_TECH_BLOG_APP_ID`).
#### baoyu-post-to-weibo #### baoyu-post-to-weibo
Post content to Weibo (微博). Supports regular posts with text, images, and videos, and headline articles (头条文章) with Markdown input. Uses real Chrome with CDP to bypass anti-automation. Post content to Weibo (微博). Supports regular posts with text, images, and videos, and headline articles (头条文章) with Markdown input. Uses real Chrome with CDP to bypass anti-automation.
@@ -798,6 +840,21 @@ Format plain text or markdown files with proper frontmatter, titles, summaries,
| Code/commands | `` `inline` `` or ` ```block``` ` | | Code/commands | `` `inline` `` or ` ```block``` ` |
| Quotes | `>` blockquote | | Quotes | `>` blockquote |
#### baoyu-markdown-to-html
Convert markdown files into styled HTML with WeChat-compatible themes, syntax highlighting, and optional bottom citations for external links.
```bash
# Basic conversion
/baoyu-markdown-to-html article.md
# Theme + color
/baoyu-markdown-to-html article.md --theme grace --color red
# Convert ordinary external links to bottom citations
/baoyu-markdown-to-html article.md --cite
```
#### baoyu-translate #### baoyu-translate
Translate articles and documents between languages with three modes: quick (direct), normal (analysis-informed), and refined (full publication-quality workflow with review and polish). Translate articles and documents between languages with three modes: quick (direct), normal (analysis-informed), and refined (full publication-quality workflow with review and polish).
+61 -4
View File
@@ -43,7 +43,7 @@ clawhub install baoyu-markdown-to-html
在 Claude Code 中运行: 在 Claude Code 中运行:
```bash ```bash
/plugin marketplace add jimliu/baoyu-skills /plugin marketplace add JimLiu/baoyu-skills
``` ```
### 安装技能 ### 安装技能
@@ -76,7 +76,7 @@ clawhub install baoyu-markdown-to-html
|------|------|----------| |------|------|----------|
| **content-skills** | 内容生成和发布 | [xhs-images](#baoyu-xhs-images), [infographic](#baoyu-infographic), [cover-image](#baoyu-cover-image), [slide-deck](#baoyu-slide-deck), [comic](#baoyu-comic), [article-illustrator](#baoyu-article-illustrator), [post-to-x](#baoyu-post-to-x), [post-to-wechat](#baoyu-post-to-wechat), [post-to-weibo](#baoyu-post-to-weibo) | | **content-skills** | 内容生成和发布 | [xhs-images](#baoyu-xhs-images), [infographic](#baoyu-infographic), [cover-image](#baoyu-cover-image), [slide-deck](#baoyu-slide-deck), [comic](#baoyu-comic), [article-illustrator](#baoyu-article-illustrator), [post-to-x](#baoyu-post-to-x), [post-to-wechat](#baoyu-post-to-wechat), [post-to-weibo](#baoyu-post-to-weibo) |
| **ai-generation-skills** | AI 生成后端 | [image-gen](#baoyu-image-gen), [danger-gemini-web](#baoyu-danger-gemini-web) | | **ai-generation-skills** | AI 生成后端 | [image-gen](#baoyu-image-gen), [danger-gemini-web](#baoyu-danger-gemini-web) |
| **utility-skills** | 内容处理工具 | [url-to-markdown](#baoyu-url-to-markdown), [danger-x-to-markdown](#baoyu-danger-x-to-markdown), [compress-image](#baoyu-compress-image), [format-markdown](#baoyu-format-markdown), [translate](#baoyu-translate) | | **utility-skills** | 内容处理工具 | [url-to-markdown](#baoyu-url-to-markdown), [danger-x-to-markdown](#baoyu-danger-x-to-markdown), [compress-image](#baoyu-compress-image), [format-markdown](#baoyu-format-markdown), [markdown-to-html](#baoyu-markdown-to-html), [translate](#baoyu-translate) |
## 更新技能 ## 更新技能
@@ -169,8 +169,9 @@ clawhub install baoyu-markdown-to-html
# 同时指定布局和风格 # 同时指定布局和风格
/baoyu-infographic path/to/content.md --layout funnel --style corporate-memphis /baoyu-infographic path/to/content.md --layout funnel --style corporate-memphis
# 指定比例 # 指定比例(预设名称或自定义 W:H
/baoyu-infographic path/to/content.md --aspect portrait /baoyu-infographic path/to/content.md --aspect portrait
/baoyu-infographic path/to/content.md --aspect 3:4
``` ```
**选项** **选项**
@@ -178,7 +179,7 @@ clawhub install baoyu-markdown-to-html
|------|------| |------|------|
| `--layout <name>` | 信息布局(20 种选项) | | `--layout <name>` | 信息布局(20 种选项) |
| `--style <name>` | 视觉风格(17 种选项,默认:craft-handmade | | `--style <name>` | 视觉风格(17 种选项,默认:craft-handmade |
| `--aspect <ratio>` | landscape (16:9)、portrait (9:16)、square (1:1) | | `--aspect <ratio>` | 预设:landscape (16:9)、portrait (9:16)、square (1:1)。自定义:任意 W:H 比例(如 3:4、4:3、2.35:1 |
| `--lang <code>` | 输出语言(en、zh、ja 等) | | `--lang <code>` | 输出语言(en、zh、ja 等) |
**布局**(信息结构): **布局**(信息结构):
@@ -581,6 +582,47 @@ WECHAT_APP_SECRET=你的AppSecret
**浏览器方式**(无需 API 配置):需已安装 Google Chrome,首次运行需扫码登录(登录状态会保存) **浏览器方式**(无需 API 配置):需已安装 Google Chrome,首次运行需扫码登录(登录状态会保存)
**多账号支持**:通过 `EXTEND.md` 管理多个微信公众号:
```bash
mkdir -p .baoyu-skills/baoyu-post-to-wechat
```
创建 `.baoyu-skills/baoyu-post-to-wechat/EXTEND.md`
```yaml
# 全局设置(所有账号共享)
default_theme: default
default_color: blue
# 账号列表
accounts:
- name: 宝玉的技术分享
alias: baoyu
default: false
default_publish_method: api
default_author: 宝玉
need_open_comment: 1
only_fans_can_comment: 0
app_id: 你的微信AppID
app_secret: 你的微信AppSecret
- name: AI 工具集
alias: ai-tools
default_publish_method: browser
default_author: AI 工具集
need_open_comment: 1
only_fans_can_comment: 0
```
| 账号配置情况 | 行为 |
|-------------|------|
| 无 `accounts` 块 | 单账号模式(向后兼容) |
| 1 个账号 | 自动选择,无需提示 |
| 2+ 个账号 | 提示选择,或使用 `--account <别名>` |
| 某账号设置 `default: true` | 预选为默认账号 |
每个账号拥有独立的 Chrome 配置目录,保证浏览器方式下的登录会话互不干扰。API 凭证可在 EXTEND.md 中直接配置,也可通过 `.env` 文件使用别名前缀的环境变量(如 `WECHAT_BAOYU_APP_ID`)。
#### baoyu-post-to-weibo #### baoyu-post-to-weibo
发布内容到微博。支持文字、图片、视频发布和头条文章(长篇 Markdown)。使用真实 Chrome + CDP 绕过反自动化检测。 发布内容到微博。支持文字、图片、视频发布和头条文章(长篇 Markdown)。使用真实 Chrome + CDP 绕过反自动化检测。
@@ -798,6 +840,21 @@ AI 驱动的生成后端。
| 代码/命令 | `` `行内` `` 或 ` ```代码块``` ` | | 代码/命令 | `` `行内` `` 或 ` ```代码块``` ` |
| 引用 | `>` 引用块 | | 引用 | `>` 引用块 |
#### baoyu-markdown-to-html
将 Markdown 文件转换为样式化 HTML,支持微信公众号兼容主题、代码高亮,以及可选的外链底部引用。
```bash
# 基础转换
/baoyu-markdown-to-html article.md
# 主题 + 颜色
/baoyu-markdown-to-html article.md --theme grace --color red
# 将普通外链转换为文末引用
/baoyu-markdown-to-html article.md --cite
```
#### baoyu-translate #### baoyu-translate
三模式翻译技能:快速(直接翻译)、标准(分析后翻译)、精翻(完整出版级工作流,含审校与润色)。 三模式翻译技能:快速(直接翻译)、标准(分析后翻译)、精翻(完整出版级工作流,含审校与润色)。
+27
View File
@@ -0,0 +1,27 @@
# Chrome Profile
All CDP skills share a single profile directory. Do NOT create per-skill profiles.
Override: `BAOYU_CHROME_PROFILE_DIR` env var (takes priority over all defaults).
| Platform | Default Path |
|----------|-------------|
| macOS | `~/Library/Application Support/baoyu-skills/chrome-profile` |
| Linux | `$XDG_DATA_HOME/baoyu-skills/chrome-profile` (fallback `~/.local/share/`) |
| Windows | `%APPDATA%/baoyu-skills/chrome-profile` |
| WSL | Windows home `/.local/share/baoyu-skills/chrome-profile` |
New skills: use `BAOYU_CHROME_PROFILE_DIR` only (not per-skill env vars like `X_BROWSER_PROFILE_DIR`).
## Implementation Pattern
```typescript
function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
```
+35
View File
@@ -0,0 +1,35 @@
# Style Maintenance (baoyu-comic)
## Adding a New Style
1. Create style definition: `skills/baoyu-comic/references/styles/<style-name>.md`
2. Update SKILL.md: add to `--style` options table + auto-selection entry
3. Generate showcase image:
```bash
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts \
--prompt "A single comic book page in <style-name> style showing [scene]. Features: [characteristics]. 3:4 portrait aspect ratio comic page." \
--image screenshots/comic-styles/<style-name>.png
```
4. Compress: `${BUN_X} skills/baoyu-compress-image/scripts/main.ts screenshots/comic-styles/<style-name>.png`
5. Update both READMEs (`README.md` + `README.zh.md`): add style to options, description table, preview grid
## Updating an Existing Style
1. Update style definition in `references/styles/`
2. Regenerate showcase image if visual characteristics changed (steps 3-4 above)
3. Update READMEs if description changed
## Deleting a Style
1. Delete style definition + showcase image (`.webp`)
2. Remove from SKILL.md `--style` options + auto-selection
3. Remove from both READMEs (options, description table, preview grid)
## Style Preview Grid Format
```markdown
| | | |
|:---:|:---:|:---:|
| ![style1](./screenshots/comic-styles/style1.webp) | ![style2](./screenshots/comic-styles/style2.webp) | ![style3](./screenshots/comic-styles/style3.webp) |
| style1 | style2 | style3 |
```
+135
View File
@@ -0,0 +1,135 @@
# Creating New Skills
**REQUIRED READING**: [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)
## Key Requirements
| Requirement | Details |
|-------------|---------|
| **Prefix** | All skills MUST use `baoyu-` prefix |
| **name field** | Max 64 chars, lowercase letters/numbers/hyphens only, no "anthropic"/"claude" |
| **description** | Max 1024 chars, third person, include what + when to use |
| **SKILL.md body** | Keep under 500 lines; use `references/` for additional content |
| **References** | One level deep from SKILL.md; avoid nested references |
## SKILL.md Frontmatter Template
```yaml
---
name: baoyu-<name>
description: <Third-person description. What it does + when to use it.>
version: <semver matching marketplace.json>
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-<name>
requires: # include only if skill has scripts
anyBins:
- bun
- npx
---
```
## Steps
1. Create `skills/baoyu-<name>/SKILL.md` with YAML front matter
2. Add TypeScript in `skills/baoyu-<name>/scripts/` (if applicable)
3. Add prompt templates in `skills/baoyu-<name>/prompts/` if needed
4. Register in `marketplace.json` under appropriate category
5. Add Script Directory section to SKILL.md if skill has scripts
6. Add openclaw metadata to frontmatter
## Category Selection
| If your skill... | Use category |
|------------------|--------------|
| Generates visual content (images, slides, comics) | `content-skills` |
| Publishes to platforms (X, WeChat, Weibo) | `content-skills` |
| Provides AI generation backend | `ai-generation-skills` |
| Converts or processes content | `utility-skills` |
New category: add plugin object to `marketplace.json` with `name`, `description`, `skills[]`.
## Writing Descriptions
**MUST write in third person**:
```yaml
# Good
description: Generates Xiaohongshu infographic series from content. Use when user asks for "小红书图片", "XHS images".
# Bad
description: I can help you create Xiaohongshu images
```
## Script Directory Template
Every SKILL.md with scripts MUST include:
```markdown
## Script Directory
**Important**: All scripts are located in the `scripts/` subdirectory of this skill.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `{baseDir}`
2. Script path = `{baseDir}/scripts/<script-name>.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `{baseDir}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
|--------|---------|
| `scripts/main.ts` | Main entry point |
```
## Progressive Disclosure
For skills with extensive content:
```
skills/baoyu-example/
├── SKILL.md # Main instructions (<500 lines)
├── references/
│ ├── styles.md # Loaded as needed
│ └── examples.md # Loaded as needed
└── scripts/
└── main.ts
```
Link from SKILL.md (one level deep only):
```markdown
**Available styles**: See [references/styles.md](references/styles.md)
```
## Extension Support (EXTEND.md)
Every SKILL.md MUST include EXTEND.md loading. Add as Step 1.1 (workflow skills) or "Preferences" section (utility skills):
```markdown
**1.1 Load Preferences (EXTEND.md)**
Check EXTEND.md existence (priority order):
\`\`\`bash
test -f .baoyu-skills/<skill-name>/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/<skill-name>/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/<skill-name>/EXTEND.md" && echo "user"
\`\`\`
| Path | Location |
|------|----------|
| `.baoyu-skills/<skill-name>/EXTEND.md` | Project directory |
| `$XDG_CONFIG_HOME/baoyu-skills/<skill-name>/EXTEND.md` | XDG config (~/.config) |
| `$HOME/.baoyu-skills/<skill-name>/EXTEND.md` | User home (legacy) |
| Result | Action |
|--------|--------|
| Found | Read, parse, display summary |
| Not found | Ask user with AskUserQuestion |
```
End of SKILL.md should include:
```markdown
## Extension Support
Custom configurations via EXTEND.md. See **Step 1.1** for paths and supported options.
```
+53
View File
@@ -0,0 +1,53 @@
# Image Generation Guidelines
Skills that require image generation MUST delegate to available image generation skills.
## Skill Selection
**Default**: `skills/baoyu-image-gen/SKILL.md` (unless user specifies otherwise).
1. Read skill's SKILL.md for parameters and capabilities
2. If user requests different skill, check `skills/` for alternatives
3. Only ask user when multiple viable options exist
## Generation Flow Template
```markdown
### Step N: Generate Images
**Skill Selection**:
1. Check available skills (`baoyu-image-gen` default, or `baoyu-danger-gemini-web`)
2. Read selected skill's SKILL.md for parameters
3. If multiple skills available, ask user to choose
**Generation Flow**:
1. Call skill with prompt, output path, and skill-specific parameters
2. Generate sequentially by default (batch parallel only when user has multiple prompts)
3. Output progress: "Generated X/N"
4. On failure, auto-retry once before reporting error
```
**Batch Parallel** (`baoyu-image-gen` only): concurrent workers with per-provider throttling via `batch.max_workers` in EXTEND.md.
## Output Path Convention
**Output Directory**: `<skill-suffix>/<topic-slug>/`
- `<skill-suffix>`: e.g., `xhs-images`, `cover-image`, `slide-deck`, `comic`
- `<topic-slug>`: 2-4 words, kebab-case from content topic
- Conflict: append timestamp `<topic-slug>-YYYYMMDD-HHMMSS`
**Source Files**: Copy to output dir as `source-{slug}.{ext}`
## Image Naming Convention
**Format**: `NN-{type}-[slug].png`
- `NN`: Two-digit sequence (01, 02, ...)
- `{type}`: cover, content, page, slide, illustration, etc.
- `[slug]`: 2-5 word kebab-case descriptor, unique within directory
Examples:
```
01-cover-ai-future.png
02-content-key-benefits.png
03-slide-architecture-overview.png
```
+39
View File
@@ -0,0 +1,39 @@
# ClawHub / OpenClaw Publishing
## OpenClaw Metadata
Skills include `metadata.openclaw` in YAML front matter:
```yaml
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#<skill-name>
requires: # only for skills with scripts
anyBins:
- bun
- npx
```
## Publishing Commands
```bash
bash scripts/sync-clawhub.sh # sync all skills
bash scripts/sync-clawhub.sh <skill> # sync one skill
```
Release hooks are configured via `.releaserc.yml`. This repo does not stage a separate release directory: release prep only syncs `packages/` into each skill's committed `scripts/vendor/`, and publish reads the skill directory directly.
## Shared Workspace Packages
`packages/` is the **only** source of truth. Never edit `skills/*/scripts/vendor/` directly.
Current package: `baoyu-chrome-cdp` (Chrome CDP utilities), consumed by 6 skills (`baoyu-danger-gemini-web`, `baoyu-danger-x-to-markdown`, `baoyu-post-to-wechat`, `baoyu-post-to-weibo`, `baoyu-post-to-x`, `baoyu-url-to-markdown`).
**How it works**: Sync script copies packages into each consuming skill's `vendor/` directory and rewrites dependency specs to `file:./vendor/<name>`. Vendor copies are committed to git, making skills self-contained.
**Update workflow**:
1. Edit package under `packages/`
2. Run `node scripts/sync-shared-skill-packages.mjs`
3. Commit synced `vendor/`, `package.json`, and `bun.lock` together
**Git hook**: Run `node scripts/install-git-hooks.mjs` once to enable the `pre-push` hook. It re-syncs and blocks push if vendor copies are stale (`--enforce-clean`).
+9
View File
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
+408
View File
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env node
import { spawnSync } from "node:child_process";
import path from "node:path";
async function main() {
const repoRoot = path.resolve(process.cwd());
const hooksPath = path.join(repoRoot, ".githooks");
const result = spawnSync("git", ["config", "core.hooksPath", hooksPath], {
cwd: repoRoot,
stdio: "inherit",
});
if (result.status !== 0) {
throw new Error("Failed to configure core.hooksPath");
}
console.log(`Configured git hooks path: ${hooksPath}`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
+75
View File
@@ -0,0 +1,75 @@
import fs from "node:fs/promises";
import path from "node:path";
const PACKAGE_DEPENDENCY_SECTIONS = [
"dependencies",
"optionalDependencies",
"peerDependencies",
"devDependencies",
];
const SKIPPED_DIRS = new Set([".git", ".clawhub", ".clawdhub", "node_modules", "out", "dist", "build"]);
const SKIPPED_FILES = new Set([".DS_Store", "bun.lockb"]);
export async function listReleaseFiles(root) {
const resolvedRoot = path.resolve(root);
const files = [];
async function walk(folder) {
const entries = await fs.readdir(folder, { withFileTypes: true });
for (const entry of entries) {
if (entry.isDirectory() && SKIPPED_DIRS.has(entry.name)) continue;
if (entry.isFile() && SKIPPED_FILES.has(entry.name)) continue;
const fullPath = path.join(folder, entry.name);
if (entry.isDirectory()) {
await walk(fullPath);
continue;
}
if (!entry.isFile()) continue;
const relPath = path.relative(resolvedRoot, fullPath).split(path.sep).join("/");
const bytes = await fs.readFile(fullPath);
files.push({ relPath, bytes });
}
}
await walk(resolvedRoot);
files.sort((left, right) => left.relPath.localeCompare(right.relPath));
return files;
}
export async function validateSelfContainedRelease(root) {
const files = await listReleaseFiles(root);
for (const file of files.filter((entry) => path.posix.basename(entry.relPath) === "package.json")) {
const packageDir = path.resolve(root, fromPosixRel(path.posix.dirname(file.relPath)));
const packageJson = JSON.parse(file.bytes.toString("utf8"));
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
if (!dependencies || typeof dependencies !== "object") continue;
for (const [name, spec] of Object.entries(dependencies)) {
if (typeof spec !== "string" || !spec.startsWith("file:")) continue;
const targetDir = path.resolve(packageDir, spec.slice(5));
if (!isWithinRoot(root, targetDir)) {
throw new Error(
`Release target is not self-contained: ${file.relPath} depends on ${name} via ${spec}`,
);
}
await fs.access(targetDir).catch(() => {
throw new Error(`Missing local dependency for release: ${file.relPath} -> ${spec}`);
});
}
}
}
}
function fromPosixRel(relPath) {
return relPath === "." ? "." : relPath.split("/").join(path.sep);
}
function isWithinRoot(root, target) {
const resolvedRoot = path.resolve(root);
const relative = path.relative(resolvedRoot, path.resolve(target));
return relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative));
}
+280
View File
@@ -0,0 +1,280 @@
import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
const PACKAGE_DEPENDENCY_SECTIONS = [
"dependencies",
"optionalDependencies",
"peerDependencies",
"devDependencies",
];
const SKIPPED_DIRS = new Set([".git", ".clawhub", ".clawdhub", "node_modules"]);
const SKIPPED_FILES = new Set([".DS_Store"]);
export async function syncSharedSkillPackages(repoRoot, options = {}) {
const root = path.resolve(repoRoot);
const workspacePackages = await discoverWorkspacePackages(root);
const targetConsumerDirs = normalizeTargetConsumerDirs(root, options.targets ?? []);
const consumers = await discoverSkillScriptPackages(root, targetConsumerDirs);
const runtime = options.install === false ? null : resolveBunRuntime();
const managedPaths = new Set();
const packageDirs = [];
for (const consumer of consumers) {
const result = await syncConsumerPackage({
consumer,
root,
workspacePackages,
runtime,
});
if (!result) continue;
packageDirs.push(consumer.dir);
for (const managedPath of result.managedPaths) {
managedPaths.add(managedPath);
}
}
return {
packageDirs,
managedPaths: [...managedPaths].sort(),
};
}
function normalizeTargetConsumerDirs(repoRoot, targets) {
if (!targets || targets.length === 0) return null;
const consumerDirs = new Set();
for (const target of targets) {
if (!target) continue;
const resolvedTarget = path.resolve(repoRoot, target);
if (path.basename(resolvedTarget) === "scripts") {
consumerDirs.add(resolvedTarget);
continue;
}
consumerDirs.add(path.join(resolvedTarget, "scripts"));
}
return consumerDirs;
}
export function ensureManagedPathsClean(repoRoot, managedPaths) {
if (managedPaths.length === 0) return;
const result = spawnSync("git", ["status", "--porcelain", "--", ...managedPaths], {
cwd: repoRoot,
encoding: "utf8",
stdio: ["ignore", "pipe", "pipe"],
});
if (result.status !== 0) {
throw new Error(result.stderr.trim() || "Failed to inspect git status for managed paths");
}
const output = result.stdout.trim();
if (!output) return;
throw new Error(
[
"Shared skill package sync produced uncommitted managed changes.",
"Review and commit these files before pushing:",
output,
].join("\n"),
);
}
async function syncConsumerPackage({ consumer, root, workspacePackages, runtime }) {
const packageJsonPath = path.join(consumer.dir, "package.json");
const packageJson = JSON.parse(await fs.readFile(packageJsonPath, "utf8"));
const localDeps = collectLocalDependencies(packageJson, workspacePackages);
if (localDeps.length === 0) {
return null;
}
const vendorRoot = path.join(consumer.dir, "vendor");
await fs.rm(vendorRoot, { recursive: true, force: true });
for (const name of localDeps) {
const sourceDir = workspacePackages.get(name);
if (!sourceDir) continue;
await syncPackageTree({
sourceDir,
targetDir: path.join(vendorRoot, name),
workspacePackages,
});
}
rewriteLocalDependencySpecs(packageJson, localDeps);
await writeJson(packageJsonPath, packageJson);
if (runtime) {
runInstall(runtime, consumer.dir);
}
const managedPaths = [
path.relative(root, packageJsonPath).split(path.sep).join("/"),
path.relative(root, path.join(consumer.dir, "bun.lock")).split(path.sep).join("/"),
path.relative(root, vendorRoot).split(path.sep).join("/"),
];
return { managedPaths };
}
async function syncPackageTree({ sourceDir, targetDir, workspacePackages }) {
await fs.rm(targetDir, { recursive: true, force: true });
await fs.mkdir(targetDir, { recursive: true });
const sourcePackageJsonPath = path.join(sourceDir, "package.json");
const packageJson = JSON.parse(await fs.readFile(sourcePackageJsonPath, "utf8"));
const localDeps = collectLocalDependencies(packageJson, workspacePackages);
const entries = await fs.readdir(sourceDir, { withFileTypes: true });
for (const entry of entries) {
if (SKIPPED_DIRS.has(entry.name) || SKIPPED_FILES.has(entry.name)) continue;
const sourcePath = path.join(sourceDir, entry.name);
const targetPath = path.join(targetDir, entry.name);
if (entry.isDirectory()) {
await copyDirectory(sourcePath, targetPath);
continue;
}
if (!entry.isFile() || entry.name === "package.json") continue;
await fs.mkdir(path.dirname(targetPath), { recursive: true });
await fs.copyFile(sourcePath, targetPath);
}
for (const name of localDeps) {
const nestedSourceDir = workspacePackages.get(name);
if (!nestedSourceDir) continue;
await syncPackageTree({
sourceDir: nestedSourceDir,
targetDir: path.join(targetDir, "vendor", name),
workspacePackages,
});
}
rewriteLocalDependencySpecs(packageJson, localDeps);
await writeJson(path.join(targetDir, "package.json"), packageJson);
}
async function copyDirectory(sourceDir, targetDir) {
await fs.mkdir(targetDir, { recursive: true });
const entries = await fs.readdir(sourceDir, { withFileTypes: true });
for (const entry of entries) {
if (SKIPPED_DIRS.has(entry.name) || SKIPPED_FILES.has(entry.name)) continue;
const sourcePath = path.join(sourceDir, entry.name);
const targetPath = path.join(targetDir, entry.name);
if (entry.isDirectory()) {
await copyDirectory(sourcePath, targetPath);
continue;
}
if (!entry.isFile()) continue;
await fs.mkdir(path.dirname(targetPath), { recursive: true });
await fs.copyFile(sourcePath, targetPath);
}
}
async function discoverWorkspacePackages(repoRoot) {
const packagesRoot = path.join(repoRoot, "packages");
const map = new Map();
if (!existsSync(packagesRoot)) return map;
const entries = await fs.readdir(packagesRoot, { withFileTypes: true });
for (const entry of entries) {
if (!entry.isDirectory()) continue;
const packageJsonPath = path.join(packagesRoot, entry.name, "package.json");
if (!existsSync(packageJsonPath)) continue;
const packageJson = JSON.parse(await fs.readFile(packageJsonPath, "utf8"));
if (!packageJson.name) continue;
map.set(packageJson.name, path.join(packagesRoot, entry.name));
}
return map;
}
async function discoverSkillScriptPackages(repoRoot, targetConsumerDirs = null) {
const skillsRoot = path.join(repoRoot, "skills");
const consumers = [];
const skillEntries = await fs.readdir(skillsRoot, { withFileTypes: true });
for (const entry of skillEntries) {
if (!entry.isDirectory()) continue;
const scriptsDir = path.join(skillsRoot, entry.name, "scripts");
if (targetConsumerDirs && !targetConsumerDirs.has(path.resolve(scriptsDir))) continue;
const packageJsonPath = path.join(scriptsDir, "package.json");
if (!existsSync(packageJsonPath)) continue;
consumers.push({ dir: scriptsDir, packageJsonPath });
}
return consumers.sort((left, right) => left.dir.localeCompare(right.dir));
}
function collectLocalDependencies(packageJson, workspacePackages) {
const localDeps = [];
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
if (!dependencies || typeof dependencies !== "object") continue;
for (const name of Object.keys(dependencies)) {
if (!workspacePackages.has(name)) continue;
localDeps.push(name);
}
}
return [...new Set(localDeps)].sort();
}
function rewriteLocalDependencySpecs(packageJson, localDeps) {
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
if (!dependencies || typeof dependencies !== "object") continue;
for (const name of localDeps) {
if (!(name in dependencies)) continue;
dependencies[name] = `file:./vendor/${name}`;
}
}
}
async function writeJson(filePath, value) {
await fs.mkdir(path.dirname(filePath), { recursive: true });
await fs.writeFile(filePath, `${JSON.stringify(value, null, 2)}\n`);
}
function resolveBunRuntime() {
if (commandExists("bun")) {
return { command: "bun", args: [] };
}
if (commandExists("npx")) {
return { command: "npx", args: ["-y", "bun"] };
}
throw new Error(
"Neither bun nor npx is installed. Install bun with `brew install oven-sh/bun/bun` or `npm install -g bun`.",
);
}
function commandExists(command) {
const result = spawnSync("sh", ["-lc", `command -v ${command}`], {
stdio: "ignore",
});
return result.status === 0;
}
function runInstall(runtime, cwd) {
const result = spawnSync(runtime.command, [...runtime.args, "install"], {
cwd,
stdio: "inherit",
});
if (result.status !== 0) {
throw new Error(`Failed to refresh Bun dependencies in ${cwd}`);
}
}
+311
View File
@@ -0,0 +1,311 @@
#!/usr/bin/env node
import fs from "node:fs/promises";
import { existsSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { listReleaseFiles, validateSelfContainedRelease } from "./lib/release-files.mjs";
const DEFAULT_REGISTRY = "https://clawhub.ai";
async function main() {
const options = parseArgs(process.argv.slice(2));
if (!options.skillDir || !options.version) {
throw new Error("--skill-dir and --version are required");
}
const skillDir = path.resolve(options.skillDir);
const skill = buildSkillEntry(skillDir, options.slug, options.displayName);
const changelog = options.changelogFile
? await fs.readFile(path.resolve(options.changelogFile), "utf8")
: "";
await validateSelfContainedRelease(skillDir);
const files = await listReleaseFiles(skillDir);
if (files.length === 0) {
throw new Error(`Skill directory is empty: ${skillDir}`);
}
if (options.dryRun) {
console.log(`Dry run: would publish ${skill.slug}@${options.version}`);
console.log(`Skill: ${skillDir}`);
console.log(`Files: ${files.length}`);
return;
}
const config = await readClawhubConfig();
const registry = (
options.registry ||
process.env.CLAWHUB_REGISTRY ||
process.env.CLAWDHUB_REGISTRY ||
config.registry ||
DEFAULT_REGISTRY
).replace(/\/+$/, "");
if (!config.token) {
throw new Error("Not logged in. Run: clawhub login");
}
await apiJson(registry, config.token, "/api/v1/whoami");
const tags = options.tags
.split(",")
.map((tag) => tag.trim())
.filter(Boolean);
await publishSkill({
registry,
token: config.token,
skill,
files,
version: options.version,
changelog,
tags,
});
}
function parseArgs(argv) {
const options = {
skillDir: "",
version: "",
changelogFile: "",
registry: "",
tags: "latest",
dryRun: false,
slug: "",
displayName: "",
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--skill-dir") {
options.skillDir = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--version") {
options.version = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--changelog-file") {
options.changelogFile = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--registry") {
options.registry = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--tags") {
options.tags = argv[index + 1] ?? "latest";
index += 1;
continue;
}
if (arg === "--slug") {
options.slug = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--display-name") {
options.displayName = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--dry-run") {
const next = argv[index + 1];
if (next && !next.startsWith("-")) {
options.dryRun = parseBoolean(next);
index += 1;
} else {
options.dryRun = true;
}
continue;
}
if (arg === "-h" || arg === "--help") {
printUsage();
process.exit(0);
}
throw new Error(`Unknown argument: ${arg}`);
}
return options;
}
function printUsage() {
console.log(`Usage: publish-skill.mjs --skill-dir <dir> --version <semver> [options]
Options:
--skill-dir <dir> Skill directory to publish
--version <semver> Version to publish
--changelog-file <file> Release notes file
--registry <url> Override registry base URL
--tags <tags> Comma-separated tags (default: latest)
--slug <value> Override slug
--display-name <value> Override display name
--dry-run Print publish plan without network calls
-h, --help Show help`);
}
function buildSkillEntry(folder, slugOverride, displayNameOverride) {
const base = path.basename(folder);
return {
folder,
slug: slugOverride || sanitizeSlug(base),
displayName: displayNameOverride || titleCase(base),
};
}
async function readClawhubConfig() {
const configPath = getConfigPath();
try {
return JSON.parse(await fs.readFile(configPath, "utf8"));
} catch {
return {};
}
}
function getConfigPath() {
const override =
process.env.CLAWHUB_CONFIG_PATH?.trim() || process.env.CLAWDHUB_CONFIG_PATH?.trim();
if (override) {
return path.resolve(override);
}
const home = os.homedir();
if (process.platform === "darwin") {
const clawhub = path.join(home, "Library", "Application Support", "clawhub", "config.json");
const clawdhub = path.join(home, "Library", "Application Support", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const xdg = process.env.XDG_CONFIG_HOME;
if (xdg) {
const clawhub = path.join(xdg, "clawhub", "config.json");
const clawdhub = path.join(xdg, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
if (process.platform === "win32" && process.env.APPDATA) {
const clawhub = path.join(process.env.APPDATA, "clawhub", "config.json");
const clawdhub = path.join(process.env.APPDATA, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const clawhub = path.join(home, ".config", "clawhub", "config.json");
const clawdhub = path.join(home, ".config", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
function pathForExistingConfig(primary, legacy) {
if (existsSync(primary)) return path.resolve(primary);
if (existsSync(legacy)) return path.resolve(legacy);
return path.resolve(primary);
}
async function publishSkill({ registry, token, skill, files, version, changelog, tags }) {
const form = new FormData();
form.set(
"payload",
JSON.stringify({
slug: skill.slug,
displayName: skill.displayName,
version,
changelog,
tags,
acceptLicenseTerms: true,
}),
);
for (const file of files) {
form.append("files", new Blob([file.bytes], { type: mimeType(file.relPath) }), file.relPath);
}
const response = await fetch(`${registry}/api/v1/skills`, {
method: "POST",
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
body: form,
});
const text = await response.text();
if (!response.ok) {
throw new Error(text || `Publish failed for ${skill.slug} (HTTP ${response.status})`);
}
const result = text ? JSON.parse(text) : {};
console.log(`OK. Published ${skill.slug}@${version}${result.versionId ? ` (${result.versionId})` : ""}`);
}
async function apiJson(registry, token, requestPath) {
const response = await fetch(`${registry}${requestPath}`, {
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
});
const text = await response.text();
let body = null;
try {
body = text ? JSON.parse(text) : null;
} catch {
body = { message: text };
}
if (response.status < 200 || response.status >= 300) {
throw new Error(body?.message || `HTTP ${response.status}`);
}
return body;
}
function sanitizeSlug(value) {
return value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+/, "")
.replace(/-+$/, "")
.replace(/--+/g, "-");
}
function titleCase(value) {
return value
.trim()
.replace(/[-_]+/g, " ")
.replace(/\s+/g, " ")
.replace(/\b\w/g, (char) => char.toUpperCase());
}
const MIME_MAP = {
".md": "text/markdown",
".ts": "text/plain",
".js": "text/javascript",
".mjs": "text/javascript",
".json": "application/json",
".yml": "text/yaml",
".yaml": "text/yaml",
".txt": "text/plain",
".html": "text/html",
".css": "text/css",
".xml": "text/xml",
".svg": "image/svg+xml",
};
function mimeType(relPath) {
const ext = path.extname(relPath).toLowerCase();
return MIME_MAP[ext] || "text/plain";
}
function parseBoolean(value) {
return ["1", "true", "yes", "on"].includes(String(value).trim().toLowerCase());
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
+528
View File
@@ -0,0 +1,528 @@
#!/usr/bin/env node
import crypto from "node:crypto";
import fs from "node:fs/promises";
import { existsSync } from "node:fs";
import path from "node:path";
import os from "node:os";
const DEFAULT_REGISTRY = "https://clawhub.ai";
const TEXT_EXTENSIONS = new Set([
"md",
"mdx",
"txt",
"json",
"json5",
"yaml",
"yml",
"toml",
"js",
"cjs",
"mjs",
"ts",
"tsx",
"jsx",
"py",
"sh",
"rb",
"go",
"rs",
"swift",
"kt",
"java",
"cs",
"cpp",
"c",
"h",
"hpp",
"sql",
"csv",
"ini",
"cfg",
"env",
"xml",
"html",
"css",
"scss",
"sass",
"svg",
]);
async function main() {
const options = parseArgs(process.argv.slice(2));
const config = await readClawhubConfig();
const registry = (
process.env.CLAWHUB_REGISTRY ||
process.env.CLAWDHUB_REGISTRY ||
config.registry ||
DEFAULT_REGISTRY
).replace(/\/+$/, "");
if (!config.token) {
throw new Error("Not logged in. Run: clawhub login");
}
await apiJson(registry, config.token, "/api/v1/whoami");
const roots = options.roots.length > 0 ? options.roots : [path.resolve("skills")];
const skills = await findSkills(roots);
if (skills.length === 0) {
throw new Error("No skills found.");
}
console.log("ClawHub sync");
console.log(`Roots with skills: ${roots.join(", ")}`);
const locals = await mapWithConcurrency(skills, options.concurrency, async (skill) => {
const files = await listTextFiles(skill.folder);
const fingerprint = buildFingerprint(files);
return {
...skill,
fileCount: files.length,
fingerprint,
};
});
const candidates = await mapWithConcurrency(locals, options.concurrency, async (skill) => {
const query = new URLSearchParams({
slug: skill.slug,
hash: skill.fingerprint,
});
const { status, body } = await apiJsonWithStatus(
registry,
config.token,
`/api/v1/resolve?${query.toString()}`
);
if (status === 404) {
return {
...skill,
status: "new",
latestVersion: null,
matchVersion: null,
};
}
if (status !== 200) {
throw new Error(body?.message || `Resolve failed for ${skill.slug} (HTTP ${status})`);
}
const latestVersion = body?.latestVersion?.version ?? null;
const matchVersion = body?.match?.version ?? null;
if (!latestVersion) {
return {
...skill,
status: "new",
latestVersion: null,
matchVersion: null,
};
}
return {
...skill,
status: matchVersion ? "synced" : "update",
latestVersion,
matchVersion,
};
});
const actionable = candidates.filter((candidate) => candidate.status !== "synced");
if (actionable.length === 0) {
console.log("Nothing to sync.");
return;
}
console.log("");
console.log("To sync");
for (const candidate of actionable) {
console.log(`- ${formatCandidate(candidate, options.bump)}`);
}
if (options.dryRun) {
console.log("");
console.log(`Dry run: would upload ${actionable.length} skill(s).`);
return;
}
const tags = options.tags
.split(",")
.map((tag) => tag.trim())
.filter(Boolean);
for (const candidate of actionable) {
const version =
candidate.status === "new"
? "1.0.0"
: bumpSemver(candidate.latestVersion, options.bump);
console.log(`Publishing ${candidate.slug}@${version}`);
const files = await listTextFiles(candidate.folder);
await publishSkill({
registry,
token: config.token,
skill: candidate,
files,
version,
changelog: options.changelog,
tags,
});
}
console.log("");
console.log(`Uploaded ${actionable.length} skill(s).`);
}
function parseArgs(argv) {
const options = {
roots: [],
dryRun: false,
bump: "patch",
changelog: "",
tags: "latest",
concurrency: 4,
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--dry-run") {
options.dryRun = true;
continue;
}
if (arg === "--all") {
continue;
}
if (arg === "--root") {
const value = argv[index + 1];
if (!value) throw new Error("--root requires a directory");
options.roots.push(path.resolve(value));
index += 1;
continue;
}
if (arg === "--bump") {
const value = argv[index + 1];
if (!["patch", "minor", "major"].includes(value)) {
throw new Error("--bump must be patch, minor, or major");
}
options.bump = value;
index += 1;
continue;
}
if (arg === "--changelog") {
const value = argv[index + 1];
if (value == null) throw new Error("--changelog requires text");
options.changelog = value;
index += 1;
continue;
}
if (arg === "--tags") {
const value = argv[index + 1];
if (value == null) throw new Error("--tags requires a value");
options.tags = value;
index += 1;
continue;
}
if (arg === "--concurrency") {
const value = Number(argv[index + 1]);
if (!Number.isInteger(value) || value < 1 || value > 32) {
throw new Error("--concurrency must be an integer between 1 and 32");
}
options.concurrency = value;
index += 1;
continue;
}
if (arg === "-h" || arg === "--help") {
printUsage();
process.exit(0);
}
throw new Error(`Unknown argument: ${arg}`);
}
return options;
}
function printUsage() {
console.log(`Usage: sync-clawhub.mjs [options]
Options:
--root <dir> Extra skill root (repeatable)
--all Accepted for compatibility
--dry-run Show what would be uploaded
--bump <type> patch | minor | major
--changelog <text> Changelog for updates
--tags <tags> Comma-separated tags
--concurrency <n> Registry check concurrency (1-32)
-h, --help Show help`);
}
async function readClawhubConfig() {
const configPath = getConfigPath();
try {
return JSON.parse(await fs.readFile(configPath, "utf8"));
} catch {
return {};
}
}
function getConfigPath() {
const override =
process.env.CLAWHUB_CONFIG_PATH?.trim() || process.env.CLAWDHUB_CONFIG_PATH?.trim();
if (override) {
return path.resolve(override);
}
const home = os.homedir();
if (process.platform === "darwin") {
const clawhub = path.join(home, "Library", "Application Support", "clawhub", "config.json");
const clawdhub = path.join(home, "Library", "Application Support", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const xdg = process.env.XDG_CONFIG_HOME;
if (xdg) {
const clawhub = path.join(xdg, "clawhub", "config.json");
const clawdhub = path.join(xdg, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
if (process.platform === "win32" && process.env.APPDATA) {
const clawhub = path.join(process.env.APPDATA, "clawhub", "config.json");
const clawdhub = path.join(process.env.APPDATA, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const clawhub = path.join(home, ".config", "clawhub", "config.json");
const clawdhub = path.join(home, ".config", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
function pathForExistingConfig(primary, legacy) {
if (existsSync(primary)) return path.resolve(primary);
if (existsSync(legacy)) return path.resolve(legacy);
return path.resolve(primary);
}
async function findSkills(roots) {
const deduped = new Map();
for (const root of roots) {
const folders = await findSkillFolders(root);
for (const folder of folders) {
deduped.set(folder.slug, folder);
}
}
return [...deduped.values()].sort((left, right) => left.slug.localeCompare(right.slug));
}
async function findSkillFolders(root) {
const stat = await safeStat(root);
if (!stat?.isDirectory()) return [];
if (await hasSkillMarker(root)) {
return [buildSkillEntry(root)];
}
const entries = await fs.readdir(root, { withFileTypes: true });
const found = [];
for (const entry of entries) {
if (!entry.isDirectory()) continue;
const folder = path.join(root, entry.name);
if (await hasSkillMarker(folder)) {
found.push(buildSkillEntry(folder));
}
}
return found;
}
function buildSkillEntry(folder) {
const base = path.basename(folder);
return {
folder,
slug: sanitizeSlug(base),
displayName: titleCase(base),
};
}
async function hasSkillMarker(folder) {
return Boolean(
(await safeStat(path.join(folder, "SKILL.md")))?.isFile() ||
(await safeStat(path.join(folder, "skill.md")))?.isFile()
);
}
async function listTextFiles(root) {
const files = [];
async function walk(folder) {
const entries = await fs.readdir(folder, { withFileTypes: true });
for (const entry of entries) {
if (entry.name.startsWith(".")) continue;
if (entry.name === "node_modules") continue;
if (entry.name === ".clawhub" || entry.name === ".clawdhub") continue;
const fullPath = path.join(folder, entry.name);
if (entry.isDirectory()) {
await walk(fullPath);
continue;
}
if (!entry.isFile()) continue;
const relPath = path.relative(root, fullPath).split(path.sep).join("/");
const ext = relPath.split(".").pop()?.toLowerCase() ?? "";
if (!TEXT_EXTENSIONS.has(ext)) continue;
const bytes = await fs.readFile(fullPath);
files.push({ relPath, bytes });
}
}
await walk(root);
files.sort((left, right) => left.relPath.localeCompare(right.relPath));
return files;
}
function buildFingerprint(files) {
const payload = files
.map((file) => `${file.relPath}:${sha256(file.bytes)}`)
.sort((left, right) => left.localeCompare(right))
.join("\n");
return crypto.createHash("sha256").update(payload).digest("hex");
}
function sha256(bytes) {
return crypto.createHash("sha256").update(bytes).digest("hex");
}
async function publishSkill({ registry, token, skill, files, version, changelog, tags }) {
const form = new FormData();
form.set(
"payload",
JSON.stringify({
slug: skill.slug,
displayName: skill.displayName,
version,
changelog,
tags,
acceptLicenseTerms: true,
})
);
for (const file of files) {
form.append("files", new Blob([file.bytes], { type: "text/plain" }), file.relPath);
}
const response = await fetch(`${registry}/api/v1/skills`, {
method: "POST",
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
body: form,
});
const text = await response.text();
if (!response.ok) {
throw new Error(text || `Publish failed for ${skill.slug} (HTTP ${response.status})`);
}
const result = text ? JSON.parse(text) : {};
console.log(`OK. Published ${skill.slug}@${version}${result.versionId ? ` (${result.versionId})` : ""}`);
}
async function apiJson(registry, token, requestPath) {
const { status, body } = await apiJsonWithStatus(registry, token, requestPath);
if (status < 200 || status >= 300) {
throw new Error(body?.message || `HTTP ${status}`);
}
return body;
}
async function apiJsonWithStatus(registry, token, requestPath) {
const response = await fetch(`${registry}${requestPath}`, {
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
});
const text = await response.text();
let body = null;
try {
body = text ? JSON.parse(text) : null;
} catch {
body = { message: text };
}
return { status: response.status, body };
}
async function mapWithConcurrency(items, limit, fn) {
const results = new Array(items.length);
let cursor = 0;
async function worker() {
while (cursor < items.length) {
const index = cursor;
cursor += 1;
results[index] = await fn(items[index], index);
}
}
const count = Math.min(Math.max(limit, 1), Math.max(items.length, 1));
await Promise.all(Array.from({ length: count }, () => worker()));
return results;
}
function formatCandidate(candidate, bump) {
if (candidate.status === "new") {
return `${candidate.slug} NEW (${candidate.fileCount} files)`;
}
return `${candidate.slug} UPDATE ${candidate.latestVersion} -> ${bumpSemver(
candidate.latestVersion,
bump
)} (${candidate.fileCount} files)`;
}
function bumpSemver(version, bump) {
const match = /^(\d+)\.(\d+)\.(\d+)$/.exec(version ?? "");
if (!match) {
throw new Error(`Invalid semver: ${version}`);
}
const major = Number(match[1]);
const minor = Number(match[2]);
const patch = Number(match[3]);
if (bump === "major") return `${major + 1}.0.0`;
if (bump === "minor") return `${major}.${minor + 1}.0`;
return `${major}.${minor}.${patch + 1}`;
}
function sanitizeSlug(value) {
return value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+/, "")
.replace(/-+$/, "")
.replace(/--+/g, "-");
}
function titleCase(value) {
return value
.trim()
.replace(/[-_]+/g, " ")
.replace(/\s+/g, " ")
.replace(/\b\w/g, (char) => char.toUpperCase());
}
async function safeStat(filePath) {
try {
return await fs.stat(filePath);
} catch {
return null;
}
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
+3 -8
View File
@@ -4,13 +4,8 @@ set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SKILLS_DIR="${ROOT_DIR}/skills" SKILLS_DIR="${ROOT_DIR}/skills"
if command -v clawhub >/dev/null 2>&1; then if ! command -v node >/dev/null 2>&1; then
CLAWHUB_CMD=(clawhub) echo "Error: node is required."
elif command -v npx >/dev/null 2>&1; then
CLAWHUB_CMD=(npx -y clawhub)
else
echo "Error: neither clawhub nor npx is available."
echo "See https://docs.openclaw.ai/zh-CN/tools/clawhub"
exit 1 exit 1
fi fi
@@ -18,4 +13,4 @@ if [ "$#" -eq 0 ]; then
set -- --all set -- --all
fi fi
exec "${CLAWHUB_CMD[@]}" sync --root "${SKILLS_DIR}" "$@" exec node "${ROOT_DIR}/scripts/sync-clawhub.mjs" --root "${SKILLS_DIR}" "$@"
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env node
import path from "node:path";
import {
ensureManagedPathsClean,
syncSharedSkillPackages,
} from "./lib/shared-skill-packages.mjs";
async function main() {
const options = parseArgs(process.argv.slice(2));
const repoRoot = path.resolve(options.repoRoot);
const result = await syncSharedSkillPackages(repoRoot, {
targets: options.targets,
});
if (options.enforceClean) {
ensureManagedPathsClean(repoRoot, result.managedPaths);
}
console.log(`Synced shared workspace packages into ${result.packageDirs.length} skill script package(s).`);
}
function parseArgs(argv) {
const options = {
repoRoot: process.cwd(),
enforceClean: false,
targets: [],
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--repo-root") {
options.repoRoot = argv[index + 1] ?? options.repoRoot;
index += 1;
continue;
}
if (arg === "--enforce-clean") {
options.enforceClean = true;
continue;
}
if (arg === "--target") {
options.targets.push(argv[index + 1] ?? "");
index += 1;
continue;
}
if (arg === "-h" || arg === "--help") {
printUsage();
process.exit(0);
}
throw new Error(`Unknown argument: ${arg}`);
}
return options;
}
function printUsage() {
console.log(`Usage: sync-shared-skill-packages.mjs [options]
Options:
--repo-root <dir> Repository root (default: current directory)
--target <dir> Sync only one skill directory (can be repeated)
--enforce-clean Fail if managed files change after sync
-h, --help Show help`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
@@ -55,12 +55,15 @@ See [references/styles.md](references/styles.md) for Core Styles, full gallery,
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-article-illustrator/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-article-illustrator/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-article-illustrator/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-article-illustrator/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-article-illustrator/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-article-illustrator/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md") { "user" }
``` ```
@@ -115,6 +118,8 @@ Full template: [references/workflow.md](references/workflow.md#step-4-generate-o
**BLOCKING: Prompt files MUST be saved before ANY image generation.** **BLOCKING: Prompt files MUST be saved before ANY image generation.**
**Execution strategy**: When multiple illustrations have saved prompt files and the task is now plain generation, prefer `baoyu-image-gen` batch mode (`build-batch.ts``--batchfile`) over spawning subagents. Use subagents only when each image still needs separate prompt iteration or creative exploration.
1. For each illustration, create a prompt file per [references/prompt-construction.md](references/prompt-construction.md) 1. For each illustration, create a prompt file per [references/prompt-construction.md](references/prompt-construction.md)
2. Save to `prompts/NN-{type}-{slug}.md` with YAML frontmatter 2. Save to `prompts/NN-{type}-{slug}.md` with YAML frontmatter
3. Prompts **MUST** use type-specific templates with structured sections (ZONES / LABELS / COLORS / STYLE / ASPECT) 3. Prompts **MUST** use type-specific templates with structured sections (ZONES / LABELS / COLORS / STYLE / ASPECT)
@@ -88,12 +88,15 @@ Check preferences and existing state, then ask ALL needed questions in ONE AskUs
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-article-illustrator/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-article-illustrator/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-article-illustrator/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-article-illustrator/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-article-illustrator/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-article-illustrator/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-article-illustrator/EXTEND.md") { "user" }
``` ```
@@ -312,6 +315,10 @@ Prompt Files:
**DO NOT** pass ad-hoc inline text to `--prompt` without first saving prompt files. The generation command should either use `--promptfiles prompts/NN-{type}-{slug}.md` or read the saved file content for `--prompt`. **DO NOT** pass ad-hoc inline text to `--prompt` without first saving prompt files. The generation command should either use `--promptfiles prompts/NN-{type}-{slug}.md` or read the saved file content for `--prompt`.
**Execution choice**:
- If multiple illustrations already have saved prompt files and the task is now plain generation, prefer `baoyu-image-gen` batch mode (`build-batch.ts` -> `main.ts --batchfile`)
- Use subagents only when each illustration still needs separate prompt rewriting, style exploration, or other per-image reasoning before generation
**CRITICAL - References in Frontmatter**: **CRITICAL - References in Frontmatter**:
- Only add `references` field if files ACTUALLY EXIST in `references/` directory - Only add `references` field if files ACTUALLY EXIST in `references/` directory
- If style/palette was extracted verbally (no file), append info to prompt BODY instead - If style/palette was extracted verbally (no file), append info to prompt BODY instead
@@ -0,0 +1,156 @@
import path from "node:path";
import process from "node:process";
import { readdir, readFile, writeFile } from "node:fs/promises";
type CliArgs = {
outlinePath: string | null;
promptsDir: string | null;
outputPath: string | null;
imagesDir: string | null;
provider: string;
model: string;
aspectRatio: string;
quality: string;
jobs: number | null;
help: boolean;
};
type OutlineEntry = {
index: number;
filename: string;
};
function printUsage(): void {
console.log(`Usage:
npx -y tsx scripts/build-batch.ts --outline outline.md --prompts prompts --output batch.json --images-dir attachments
Options:
--outline <path> Path to outline.md
--prompts <path> Path to prompts directory
--output <path> Path to output batch.json
--images-dir <path> Directory for generated images
--provider <name> Provider for baoyu-image-gen batch tasks (default: replicate)
--model <id> Model for baoyu-image-gen batch tasks (default: google/nano-banana-pro)
--ar <ratio> Aspect ratio for all tasks (default: 16:9)
--quality <level> Quality for all tasks (default: 2k)
--jobs <count> Recommended worker count metadata (optional)
-h, --help Show help`);
}
function parseArgs(argv: string[]): CliArgs {
const args: CliArgs = {
outlinePath: null,
promptsDir: null,
outputPath: null,
imagesDir: null,
provider: "replicate",
model: "google/nano-banana-pro",
aspectRatio: "16:9",
quality: "2k",
jobs: null,
help: false,
};
for (let i = 0; i < argv.length; i++) {
const current = argv[i]!;
if (current === "--outline") args.outlinePath = argv[++i] ?? null;
else if (current === "--prompts") args.promptsDir = argv[++i] ?? null;
else if (current === "--output") args.outputPath = argv[++i] ?? null;
else if (current === "--images-dir") args.imagesDir = argv[++i] ?? null;
else if (current === "--provider") args.provider = argv[++i] ?? args.provider;
else if (current === "--model") args.model = argv[++i] ?? args.model;
else if (current === "--ar") args.aspectRatio = argv[++i] ?? args.aspectRatio;
else if (current === "--quality") args.quality = argv[++i] ?? args.quality;
else if (current === "--jobs") {
const value = argv[++i];
args.jobs = value ? parseInt(value, 10) : null;
} else if (current === "--help" || current === "-h") {
args.help = true;
}
}
return args;
}
function parseOutline(content: string): OutlineEntry[] {
const entries: OutlineEntry[] = [];
const blocks = content.split(/^## Illustration\s+/m).slice(1);
for (const block of blocks) {
const indexMatch = block.match(/^(\d+)/);
const filenameMatch = block.match(/\*\*Filename\*\*:\s*(.+)/);
if (indexMatch && filenameMatch) {
entries.push({
index: parseInt(indexMatch[1]!, 10),
filename: filenameMatch[1]!.trim(),
});
}
}
return entries;
}
async function findPromptFile(promptsDir: string, entry: OutlineEntry): Promise<string | null> {
const files = await readdir(promptsDir);
const prefix = String(entry.index).padStart(2, "0");
const match = files.find((f) => f.startsWith(prefix) && f.endsWith(".md"));
return match ? path.join(promptsDir, match) : null;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
if (args.help) {
printUsage();
return;
}
if (!args.outlinePath) {
console.error("Error: --outline is required");
process.exit(1);
}
if (!args.promptsDir) {
console.error("Error: --prompts is required");
process.exit(1);
}
if (!args.outputPath) {
console.error("Error: --output is required");
process.exit(1);
}
const outlineContent = await readFile(args.outlinePath, "utf8");
const entries = parseOutline(outlineContent);
if (entries.length === 0) {
console.error("No illustration entries found in outline.");
process.exit(1);
}
const tasks = [];
for (const entry of entries) {
const promptFile = await findPromptFile(args.promptsDir, entry);
if (!promptFile) {
console.error(`Warning: No prompt file found for illustration ${entry.index}, skipping.`);
continue;
}
const imageDir = args.imagesDir ?? path.dirname(args.outputPath);
tasks.push({
id: `illustration-${String(entry.index).padStart(2, "0")}`,
promptFiles: [promptFile],
image: path.join(imageDir, entry.filename),
provider: args.provider,
model: args.model,
ar: args.aspectRatio,
quality: args.quality,
});
}
const output: Record<string, unknown> = { tasks };
if (args.jobs) output.jobs = args.jobs;
await writeFile(args.outputPath, JSON.stringify(output, null, 2) + "\n");
console.log(`Batch file written: ${args.outputPath} (${tasks.length} tasks)`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
@@ -39,12 +39,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-comic/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-comic/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-comic/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-comic/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-comic/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-comic/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-comic/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-comic/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-comic/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-comic/EXTEND.md") { "user" }
``` ```
+3
View File
@@ -30,12 +30,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-compress-image/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-compress-image/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-compress-image/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-compress-image/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-compress-image/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-compress-image/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-compress-image/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-compress-image/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-compress-image/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-compress-image/EXTEND.md") { "user" }
``` ```
+3
View File
@@ -134,12 +134,15 @@ Check EXTEND.md existence (priority: project → user):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-cover-image/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-cover-image/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-cover-image/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-cover-image/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-cover-image/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-cover-image/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-cover-image/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-cover-image/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-cover-image/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-cover-image/EXTEND.md") { "user" }
``` ```
+3
View File
@@ -57,12 +57,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-danger-gemini-web/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-danger-gemini-web/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-danger-gemini-web/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-danger-gemini-web/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-danger-gemini-web/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-danger-gemini-web/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-danger-gemini-web/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-danger-gemini-web/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-danger-gemini-web/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-danger-gemini-web/EXTEND.md") { "user" }
``` ```
@@ -0,0 +1,14 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-danger-gemini-web-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
},
},
},
"packages": {
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
}
}
@@ -1,183 +1,76 @@
import { spawn, type ChildProcess } from 'node:child_process';
import fs from 'node:fs';
import { mkdir } from 'node:fs/promises';
import net from 'node:net';
import process from 'node:process'; import process from 'node:process';
import {
CdpConnection,
findChromeExecutable as findChromeExecutableBase,
findExistingChromeDebugPort,
getFreePort,
killChrome,
launchChrome as launchChromeBase,
openPageSession,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
import { Endpoint, Headers } from '../constants.js'; import { Endpoint, Headers } from '../constants.js';
import { logger } from './logger.js'; import { logger } from './logger.js';
import { cookie_header, fetch_with_timeout, sleep } from './http.js'; import { cookie_header, fetch_with_timeout } from './http.js';
import { read_cookie_file, type CookieMap, write_cookie_file } from './cookie-file.js'; import { read_cookie_file, type CookieMap, write_cookie_file } from './cookie-file.js';
import { resolveGeminiWebChromeProfileDir, resolveGeminiWebCookiePath } from './paths.js'; import { resolveGeminiWebChromeProfileDir, resolveGeminiWebCookiePath } from './paths.js';
type CdpSendOptions = { sessionId?: string; timeoutMs?: number }; const GEMINI_APP_URL = 'https://gemini.google.com/app';
class CdpConnection { const CHROME_CANDIDATES_FULL: PlatformCandidates = {
private ws: WebSocket; darwin: [
private nextId = 0;
private pending = new Map<
number,
{ resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }
>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; result?: unknown; error?: { message?: string } };
if (msg.id) {
const p = this.pending.get(msg.id);
if (p) {
this.pending.delete(msg.id);
if (p.timer) clearTimeout(p.timer);
if (msg.error?.message) p.reject(new Error(msg.error.message));
else p.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, p] of this.pending.entries()) {
this.pending.delete(id);
if (p.timer) clearTimeout(p.timer);
p.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const t = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => {
clearTimeout(t);
resolve();
});
ws.addEventListener('error', () => {
clearTimeout(t);
reject(new Error('CDP connection failed.'));
});
});
return new CdpConnection(ws);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, opts?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const msg: Record<string, unknown> = { id, method };
if (params) msg.params = params;
if (opts?.sessionId) msg.sessionId = opts.sessionId;
const timeoutMs = opts?.timeoutMs ?? 15_000;
const out = await new Promise<unknown>((resolve, reject) => {
const t =
timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer: t });
this.ws.send(JSON.stringify(msg));
});
return out as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
async function get_free_port(): Promise<number> {
const fixed = parseInt(process.env.GEMINI_WEB_DEBUG_PORT || '', 10);
if (fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const srv = net.createServer();
srv.unref();
srv.on('error', reject);
srv.listen(0, '127.0.0.1', () => {
const addr = srv.address();
if (!addr || typeof addr === 'string') {
srv.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = addr.port;
srv.close((err) => (err ? reject(err) : resolve(port)));
});
});
}
function find_chrome_executable(): string | null {
const override = process.env.GEMINI_WEB_CHROME_PATH?.trim();
if (override && fs.existsSync(override)) return override;
const candidates: string[] = [];
switch (process.platform) {
case 'darwin':
candidates.push(
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary', '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta', '/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium', '/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge', '/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
); ],
break; win32: [
case 'win32': 'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
candidates.push( 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\\\Program Files\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe', 'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\\\Program Files (x86)\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe', 'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\\\Program Files\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe', ],
'C:\\\\Program Files (x86)\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe', default: [
);
break;
default:
candidates.push(
'/usr/bin/google-chrome', '/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable', '/usr/bin/google-chrome-stable',
'/usr/bin/chromium', '/usr/bin/chromium',
'/usr/bin/chromium-browser', '/usr/bin/chromium-browser',
'/snap/bin/chromium', '/snap/bin/chromium',
'/usr/bin/microsoft-edge', '/usr/bin/microsoft-edge',
); ],
break; };
async function get_free_port(): Promise<number> {
return await getFreePort('GEMINI_WEB_DEBUG_PORT');
} }
for (const p of candidates) { function find_chrome_executable(): string | null {
if (fs.existsSync(p)) return p; return findChromeExecutableBase({
} candidates: CHROME_CANDIDATES_FULL,
return null; envNames: ['GEMINI_WEB_CHROME_PATH'],
}) ?? null;
} }
async function wait_for_chrome_debug_port(port: number, timeoutMs: number): Promise<string> { async function find_existing_chrome_debug_port(profileDir: string): Promise<number | null> {
const start = Date.now(); return await findExistingChromeDebugPort({ profileDir });
while (Date.now() - start < timeoutMs) {
try {
const res = await fetch_with_timeout(`http://127.0.0.1:${port}/json/version`, { timeout_ms: 5_000 });
if (!res.ok) throw new Error(`status=${res.status}`);
const j = (await res.json()) as { webSocketDebuggerUrl?: string };
if (j.webSocketDebuggerUrl) return j.webSocketDebuggerUrl;
} catch {}
await sleep(200);
}
throw new Error('Chrome debug port not ready');
} }
async function launch_chrome(profileDir: string, port: number): Promise<ChildProcess> { async function launch_chrome(profileDir: string, port: number) {
const chrome = find_chrome_executable(); const chromePath = find_chrome_executable();
if (!chrome) throw new Error('Chrome executable not found.'); if (!chromePath) throw new Error('Chrome executable not found.');
const args = [ return await launchChromeBase({
`--remote-debugging-port=${port}`, chromePath,
`--user-data-dir=${profileDir}`, profileDir,
'--no-first-run', port,
'--no-default-browser-check', url: GEMINI_APP_URL,
'--disable-popup-blocking', extraArgs: ['--disable-popup-blocking'],
'https://gemini.google.com/app', });
];
return spawn(chrome, args, { stdio: 'ignore' });
} }
async function is_gemini_session_ready(cookies: CookieMap, verbose: boolean): Promise<boolean> { async function is_gemini_session_ready(cookies: CookieMap, verbose: boolean): Promise<boolean> {
@@ -209,27 +102,33 @@ async function fetch_google_cookies_via_cdp(
timeoutMs: number, timeoutMs: number,
verbose: boolean, verbose: boolean,
): Promise<CookieMap> { ): Promise<CookieMap> {
await mkdir(profileDir, { recursive: true }); const existingPort = await find_existing_chrome_debug_port(profileDir);
const reusing = existingPort !== null;
const port = await get_free_port(); const port = existingPort ?? await get_free_port();
const chrome = await launch_chrome(profileDir, port); const chrome = reusing ? null : await launch_chrome(profileDir, port);
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await wait_for_chrome_debug_port(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 15_000); cdp = await CdpConnection.connect(wsUrl, 15_000);
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', {
url: 'https://gemini.google.com/app',
newWindow: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId, flatten: true });
await cdp.send('Network.enable', {}, { sessionId });
if (verbose) { if (verbose) {
logger.info('Chrome opened. If needed, complete Google login in the window. Waiting for a valid Gemini session...'); logger.info(reusing
? `Reusing existing Chrome on port ${port}. Waiting for a valid Gemini session...`
: 'Chrome opened. If needed, complete Google login in the window. Waiting for a valid Gemini session...');
} }
const page = await openPageSession({
cdp,
reusing,
url: GEMINI_APP_URL,
matchTarget: (target) => target.type === 'page' && target.url.includes('gemini.google.com'),
enableNetwork: true,
});
const { sessionId } = page;
targetId = page.targetId;
const start = Date.now(); const start = Date.now();
let last: CookieMap = {}; let last: CookieMap = {};
@@ -240,14 +139,14 @@ async function fetch_google_cookies_via_cdp(
{ sessionId, timeoutMs: 10_000 }, { sessionId, timeoutMs: 10_000 },
); );
const m: CookieMap = {}; const cookieMap: CookieMap = {};
for (const c of cookies) { for (const cookie of cookies) {
if (c?.name && typeof c.value === 'string') m[c.name] = c.value; if (cookie?.name && typeof cookie.value === 'string') cookieMap[cookie.name] = cookie.value;
} }
last = m; last = cookieMap;
if (await is_gemini_session_ready(m, verbose)) { if (await is_gemini_session_ready(cookieMap, verbose)) {
return m; return cookieMap;
} }
await sleep(1000); await sleep(1000);
@@ -256,22 +155,19 @@ async function fetch_google_cookies_via_cdp(
throw new Error(`Timed out waiting for a valid Gemini session. Last keys: ${Object.keys(last).join(', ')}`); throw new Error(`Timed out waiting for a valid Gemini session. Last keys: ${Object.keys(last).join(', ')}`);
} finally { } finally {
if (cdp) { if (cdp) {
if (reusing && targetId) {
try {
await cdp.send('Target.closeTarget', { targetId }, { timeoutMs: 5_000 });
} catch {}
} else {
try { try {
await cdp.send('Browser.close', {}, { timeoutMs: 5_000 }); await cdp.send('Browser.close', {}, { timeoutMs: 5_000 });
} catch {} } catch {}
}
cdp.close(); cdp.close();
} }
try { if (chrome) killChrome(chrome);
chrome.kill('SIGTERM');
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill('SIGKILL');
} catch {}
}
}, 2_000).unref?.();
} }
} }
@@ -286,8 +182,8 @@ export async function load_browser_cookies(domain_name: string = '', verbose: bo
const cookies = await fetch_google_cookies_via_cdp(profileDir, 120_000, verbose); const cookies = await fetch_google_cookies_via_cdp(profileDir, 120_000, verbose);
const filtered: CookieMap = {}; const filtered: CookieMap = {};
for (const [k, v] of Object.entries(cookies)) { for (const [key, value] of Object.entries(cookies)) {
if (typeof v === 'string' && v.length > 0) filtered[k] = v; if (typeof value === 'string' && value.length > 0) filtered[key] = value;
} }
await write_cookie_file(filtered, resolveGeminiWebCookiePath(), 'cdp'); await write_cookie_file(filtered, resolveGeminiWebCookiePath(), 'cdp');
@@ -0,0 +1,8 @@
{
"name": "baoyu-danger-gemini-web-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
@@ -83,12 +83,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md") { "user" }
``` ```
@@ -0,0 +1,14 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-danger-x-to-markdown-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
},
},
},
"packages": {
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
}
}
@@ -1,7 +1,16 @@
import { spawn, type ChildProcess } from "node:child_process"; import {
import fs from "node:fs"; CdpConnection,
import { mkdir } from "node:fs/promises"; findChromeExecutable as findChromeExecutableBase,
import net from "node:net"; findExistingChromeDebugPort,
getFreePort,
killChrome,
launchChrome as launchChromeBase,
openPageSession,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from "baoyu-chrome-cdp";
import process from "node:process"; import process from "node:process";
import { read_cookie_file, write_cookie_file } from "./cookie-file.js"; import { read_cookie_file, write_cookie_file } from "./cookie-file.js";
@@ -9,201 +18,47 @@ import { resolveXToMarkdownCookiePath } from "./paths.js";
import { X_COOKIE_NAMES, X_REQUIRED_COOKIES, X_LOGIN_URL, X_USER_DATA_DIR } from "./constants.js"; import { X_COOKIE_NAMES, X_REQUIRED_COOKIES, X_LOGIN_URL, X_USER_DATA_DIR } from "./constants.js";
import type { CookieLike } from "./types.js"; import type { CookieLike } from "./types.js";
type CdpSendOptions = { sessionId?: string; timeoutMs?: number }; const CHROME_CANDIDATES_FULL: PlatformCandidates = {
darwin: [
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function fetchWithTimeout(
url: string,
init: RequestInit & { timeoutMs?: number } = {}
): Promise<Response> {
const { timeoutMs, ...rest } = init;
if (!timeoutMs || timeoutMs <= 0) return fetch(url, rest);
const ctl = new AbortController();
const t = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { ...rest, signal: ctl.signal });
} finally {
clearTimeout(t);
}
}
class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<
number,
{ resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }
>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener("message", (event) => {
try {
const data =
typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; result?: unknown; error?: { message?: string } };
if (msg.id) {
const p = this.pending.get(msg.id);
if (p) {
this.pending.delete(msg.id);
if (p.timer) clearTimeout(p.timer);
if (msg.error?.message) p.reject(new Error(msg.error.message));
else p.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, p] of this.pending.entries()) {
this.pending.delete(id);
if (p.timer) clearTimeout(p.timer);
p.reject(new Error("CDP connection closed."));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const t = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(t);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(t);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws);
}
async send<T = unknown>(
method: string,
params?: Record<string, unknown>,
opts?: CdpSendOptions
): Promise<T> {
const id = ++this.nextId;
const msg: Record<string, unknown> = { id, method };
if (params) msg.params = params;
if (opts?.sessionId) msg.sessionId = opts.sessionId;
const timeoutMs = opts?.timeoutMs ?? 15_000;
const out = await new Promise<unknown>((resolve, reject) => {
const t =
timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer: t });
this.ws.send(JSON.stringify(msg));
});
return out as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.X_DEBUG_PORT || "", 10);
if (fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const srv = net.createServer();
srv.unref();
srv.on("error", reject);
srv.listen(0, "127.0.0.1", () => {
const addr = srv.address();
if (!addr || typeof addr === "string") {
srv.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = addr.port;
srv.close((err) => (err ? reject(err) : resolve(port)));
});
});
}
function findChromeExecutable(): string | null {
const override = process.env.X_CHROME_PATH?.trim();
if (override && fs.existsSync(override)) return override;
const candidates: string[] = [];
switch (process.platform) {
case "darwin":
candidates.push(
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary", "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
"/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta", "/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta",
"/Applications/Chromium.app/Contents/MacOS/Chromium", "/Applications/Chromium.app/Contents/MacOS/Chromium",
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge" "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
); ],
break; win32: [
case "win32": "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
candidates.push( "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
"C:\\\\Program Files\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe", "C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe",
"C:\\\\Program Files (x86)\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe", "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
"C:\\\\Program Files\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe", ],
"C:\\\\Program Files (x86)\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe" default: [
);
break;
default:
candidates.push(
"/usr/bin/google-chrome", "/usr/bin/google-chrome",
"/usr/bin/google-chrome-stable", "/usr/bin/google-chrome-stable",
"/usr/bin/chromium", "/usr/bin/chromium",
"/usr/bin/chromium-browser", "/usr/bin/chromium-browser",
"/snap/bin/chromium", "/snap/bin/chromium",
"/usr/bin/microsoft-edge" "/usr/bin/microsoft-edge",
); ],
break; };
function findChromeExecutable(): string | null {
return findChromeExecutableBase({
candidates: CHROME_CANDIDATES_FULL,
envNames: ["X_CHROME_PATH"],
}) ?? null;
} }
for (const p of candidates) { async function launchChrome(profileDir: string, port: number) {
if (fs.existsSync(p)) return p; const chromePath = findChromeExecutable();
} if (!chromePath) throw new Error("Chrome executable not found.");
return null; return await launchChromeBase({
} chromePath,
profileDir,
async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> { port,
const start = Date.now(); url: X_LOGIN_URL,
while (Date.now() - start < timeoutMs) { extraArgs: ["--disable-popup-blocking"],
try { });
const res = await fetchWithTimeout(`http://127.0.0.1:${port}/json/version`, { timeoutMs: 5_000 });
if (!res.ok) throw new Error(`status=${res.status}`);
const j = (await res.json()) as { webSocketDebuggerUrl?: string };
if (j.webSocketDebuggerUrl) return j.webSocketDebuggerUrl;
} catch {}
await sleep(200);
}
throw new Error("Chrome debug port not ready");
}
async function launchChrome(profileDir: string, port: number): Promise<ChildProcess> {
const chrome = findChromeExecutable();
if (!chrome) throw new Error("Chrome executable not found.");
const args = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
"--no-first-run",
"--no-default-browser-check",
"--disable-popup-blocking",
X_LOGIN_URL,
];
return spawn(chrome, args, { stdio: "ignore" });
} }
async function fetchXCookiesViaCdp( async function fetchXCookiesViaCdp(
@@ -212,25 +67,33 @@ async function fetchXCookiesViaCdp(
verbose: boolean, verbose: boolean,
log?: (message: string) => void log?: (message: string) => void
): Promise<Record<string, string>> { ): Promise<Record<string, string>> {
await mkdir(profileDir, { recursive: true }); const existingPort = await findExistingChromeDebugPort({ profileDir });
const reusing = existingPort !== null;
const port = await getFreePort(); const port = existingPort ?? await getFreePort("X_DEBUG_PORT");
const chrome = await launchChrome(profileDir, port); const chrome = reusing ? null : await launchChrome(profileDir, port);
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 15_000); cdp = await CdpConnection.connect(wsUrl, 15_000);
const { targetId } = await cdp.send<{ targetId: string }>("Target.createTarget", { const page = await openPageSession({
cdp,
reusing,
url: X_LOGIN_URL, url: X_LOGIN_URL,
newWindow: true, matchTarget: (target) => target.type === "page" && (
target.url.includes("x.com") || target.url.includes("twitter.com")
),
enableNetwork: true,
}); });
const { sessionId } = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true }); const { sessionId } = page;
await cdp.send("Network.enable", {}, { sessionId }); targetId = page.targetId;
if (verbose) { if (verbose) {
log?.("[x-cookies] Chrome opened. If needed, complete X login in the window. Waiting for cookies..."); log?.(reusing
? `[x-cookies] Reusing existing Chrome on port ${port}. Waiting for cookies...`
: "[x-cookies] Chrome opened. If needed, complete X login in the window. Waiting for cookies...");
} }
const start = Date.now(); const start = Date.now();
@@ -255,22 +118,19 @@ async function fetchXCookiesViaCdp(
throw new Error(`Timed out waiting for X cookies. Last keys: ${Object.keys(last).join(", ")}`); throw new Error(`Timed out waiting for X cookies. Last keys: ${Object.keys(last).join(", ")}`);
} finally { } finally {
if (cdp) { if (cdp) {
if (reusing && targetId) {
try {
await cdp.send("Target.closeTarget", { targetId }, { timeoutMs: 5_000 });
} catch {}
} else {
try { try {
await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); await cdp.send("Browser.close", {}, { timeoutMs: 5_000 });
} catch {} } catch {}
}
cdp.close(); cdp.close();
} }
try { if (chrome) killChrome(chrome);
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
} }
} }
@@ -0,0 +1,8 @@
{
"name": "baoyu-danger-x-to-markdown-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
+3
View File
@@ -34,12 +34,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-format-markdown/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-format-markdown/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-format-markdown/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-format-markdown/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-format-markdown/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-format-markdown/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-format-markdown/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-format-markdown/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-format-markdown/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-format-markdown/EXTEND.md") { "user" }
``` ```
@@ -1,10 +1,10 @@
import { execSync } from "child_process"; import { spawnSync } from "node:child_process";
import process from "node:process";
export function applyAutocorrect(filePath: string): boolean { export function applyAutocorrect(filePath: string): boolean {
try { const npxCmd = process.platform === "win32" ? "npx.cmd" : "npx";
execSync(`npx autocorrect-node --fix "${filePath}"`, { stdio: "inherit" }); const result = spawnSync(npxCmd, ["autocorrect-node", "--fix", filePath], {
return true; stdio: "inherit",
} catch { });
return false; return result.status === 0;
}
} }
+77 -29
View File
@@ -1,6 +1,6 @@
--- ---
name: baoyu-image-gen name: baoyu-image-gen
description: AI image generation with OpenAI, Google, DashScope and Replicate APIs. Supports text-to-image, reference images, aspect ratios. Sequential by default; parallel generation available on request. Use when user asks to generate, create, or draw images. description: AI image generation with OpenAI, Google, DashScope and Replicate APIs. Supports text-to-image, reference images, aspect ratios, and batch generation from saved prompt files. Sequential by default; use batch parallel generation when the user already has multiple prompts or wants stable multi-image throughput. Use when user asks to generate, create, or draw images.
version: 1.56.1 version: 1.56.1
metadata: metadata:
openclaw: openclaw:
@@ -31,12 +31,15 @@ Check EXTEND.md existence (priority: project → user):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-image-gen/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-image-gen/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-image-gen/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-image-gen/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-image-gen/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-image-gen/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md") { "user" }
``` ```
@@ -52,7 +55,7 @@ if (Test-Path "$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md") { "user" }
| `.baoyu-skills/baoyu-image-gen/EXTEND.md` | Project directory | | `.baoyu-skills/baoyu-image-gen/EXTEND.md` | Project directory |
| `$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md` | User home | | `$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md` | User home |
**EXTEND.md Supports**: Default provider | Default quality | Default aspect ratio | Default image size | Default models **EXTEND.md Supports**: Default provider | Default quality | Default aspect ratio | Default image size | Default models | Batch worker cap | Provider-specific batch limits
Schema: `references/config/preferences-schema.md` Schema: `references/config/preferences-schema.md`
@@ -88,22 +91,57 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
# Replicate with specific model # Replicate with specific model
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana
# Batch mode with saved prompt files
${BUN_X} {baseDir}/scripts/main.ts --batchfile batch.json
# Batch mode with explicit worker count
${BUN_X} {baseDir}/scripts/main.ts --batchfile batch.json --jobs 4 --json
``` ```
### Batch File Format
```json
{
"jobs": 4,
"tasks": [
{
"id": "hero",
"promptFiles": ["prompts/hero.md"],
"image": "out/hero.png",
"provider": "replicate",
"model": "google/nano-banana-pro",
"ar": "16:9",
"quality": "2k"
},
{
"id": "diagram",
"promptFiles": ["prompts/diagram.md"],
"image": "out/diagram.png",
"ref": ["references/original.png"]
}
]
}
```
Paths in `promptFiles`, `image`, and `ref` are resolved relative to the batch file's directory. `jobs` is optional (overridden by CLI `--jobs`). Top-level array format (without `jobs` wrapper) is also accepted.
## Options ## Options
| Option | Description | | Option | Description |
|--------|-------------| |--------|-------------|
| `--prompt <text>`, `-p` | Prompt text | | `--prompt <text>`, `-p` | Prompt text |
| `--promptfiles <files...>` | Read prompt from files (concatenated) | | `--promptfiles <files...>` | Read prompt from files (concatenated) |
| `--image <path>` | Output image path (required) | | `--image <path>` | Output image path (required in single-image mode) |
| `--provider google\|openai\|dashscope\|replicate` | Force provider (default: google) | | `--batchfile <path>` | JSON batch file for multi-image generation |
| `--model <id>`, `-m` | Model ID (Google: `gemini-3-pro-image-preview`, `gemini-3.1-flash-image-preview`; OpenAI: `gpt-image-1.5`) | | `--jobs <count>` | Worker count for batch mode (default: auto, max from config, built-in default 10) |
| `--provider google\|openai\|dashscope\|replicate` | Force provider (default: auto-detect) |
| `--model <id>`, `-m` | Model ID (Google: `gemini-3-pro-image-preview`, `gemini-3.1-flash-image-preview`; OpenAI: `gpt-image-1.5`, `gpt-image-1`) |
| `--ar <ratio>` | Aspect ratio (e.g., `16:9`, `1:1`, `4:3`) | | `--ar <ratio>` | Aspect ratio (e.g., `16:9`, `1:1`, `4:3`) |
| `--size <WxH>` | Size (e.g., `1024x1024`) | | `--size <WxH>` | Size (e.g., `1024x1024`) |
| `--quality normal\|2k` | Quality preset (default: 2k) | | `--quality normal\|2k` | Quality preset (default: `2k`) |
| `--imageSize 1K\|2K\|4K` | Image size for Google (default: from quality) | | `--imageSize 1K\|2K\|4K` | Image size for Google (default: from quality) |
| `--ref <files...>` | Reference images. Supported by Google multimodal (`gemini-3-pro-image-preview`, `gemini-3-flash-preview`, `gemini-3.1-flash-image-preview`) and OpenAI edits (GPT Image models). If provider omitted: Google first, then OpenAI | | `--ref <files...>` | Reference images. Supported by Google multimodal, OpenAI GPT Image edits, and Replicate |
| `--n <count>` | Number of images | | `--n <count>` | Number of images |
| `--json` | JSON output | | `--json` | JSON output |
@@ -123,6 +161,9 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
| `GOOGLE_BASE_URL` | Custom Google endpoint | | `GOOGLE_BASE_URL` | Custom Google endpoint |
| `DASHSCOPE_BASE_URL` | Custom DashScope endpoint | | `DASHSCOPE_BASE_URL` | Custom DashScope endpoint |
| `REPLICATE_BASE_URL` | Custom Replicate endpoint | | `REPLICATE_BASE_URL` | Custom Replicate endpoint |
| `BAOYU_IMAGE_GEN_MAX_WORKERS` | Override batch worker cap |
| `BAOYU_IMAGE_GEN_<PROVIDER>_CONCURRENCY` | Override provider concurrency, e.g. `BAOYU_IMAGE_GEN_REPLICATE_CONCURRENCY` |
| `BAOYU_IMAGE_GEN_<PROVIDER>_START_INTERVAL_MS` | Override provider start gap, e.g. `BAOYU_IMAGE_GEN_REPLICATE_START_INTERVAL_MS` |
**Load Priority**: CLI args > EXTEND.md > env vars > `<cwd>/.baoyu-skills/.env` > `~/.baoyu-skills/.env` **Load Priority**: CLI args > EXTEND.md > env vars > `<cwd>/.baoyu-skills/.env` > `~/.baoyu-skills/.env`
@@ -167,10 +208,10 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
## Quality Presets ## Quality Presets
| Preset | Google imageSize | OpenAI Size | Use Case | | Preset | Google imageSize | OpenAI Size | Replicate resolution | Use Case |
|--------|------------------|-------------|----------| |--------|------------------|-------------|----------------------|----------|
| `normal` | 1K | 1024px | Quick previews | | `normal` | 1K | 1024px | 1K | Quick previews |
| `2k` (default) | 2K | 2048px | Covers, illustrations, infographics | | `2k` (default) | 2K | 2048px | 2K | Covers, illustrations, infographics |
**Google imageSize**: Can be overridden with `--imageSize 1K|2K|4K` **Google imageSize**: Can be overridden with `--imageSize 1K|2K|4K`
@@ -179,41 +220,48 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
Supported: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `2.35:1` Supported: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `2.35:1`
- Google multimodal: uses `imageConfig.aspectRatio` - Google multimodal: uses `imageConfig.aspectRatio`
- Google Imagen: uses `aspectRatio` parameter
- OpenAI: maps to closest supported size - OpenAI: maps to closest supported size
- Replicate: passes `aspect_ratio` to model; when `--ref` is provided without `--ar`, defaults to `match_input_image`
## Generation Mode ## Generation Mode
**Default**: Sequential generation (one image at a time). This ensures stable output and easier debugging. **Default**: Sequential generation.
**Parallel Generation**: Only use when user explicitly requests parallel/concurrent generation. **Batch Parallel Generation**: When `--batchfile` contains 2 or more pending tasks, the script automatically enables parallel generation.
| Mode | When to Use | | Mode | When to Use |
|------|-------------| |------|-------------|
| Sequential (default) | Normal usage, single images, small batches | | Sequential (default) | Normal usage, single images, small batches |
| Parallel | User explicitly requests, large batches (10+) | | Parallel batch | Batch mode with 2+ tasks |
**Parallel Settings** (when requested): Execution choice:
| Setting | Value | | Situation | Preferred approach | Why |
|---------|-------| |-----------|--------------------|-----|
| Recommended concurrency | 4 subagents | | One image, or 1-2 simple images | Sequential | Lower coordination overhead and easier debugging |
| Max concurrency | 8 subagents | | Multiple images already have saved prompt files | Batch (`--batchfile`) | Reuses finalized prompts, applies shared throttling/retries, and gives predictable throughput |
| Use case | Large batch generation when user requests parallel | | Each image still needs separate reasoning, prompt writing, or style exploration | Subagents | The work is still exploratory, so each image may need independent analysis before generation |
| Output comes from `baoyu-article-illustrator` with `outline.md` + `prompts/` | Batch (`build-batch.ts` -> `--batchfile`) | That workflow already produces prompt files, so direct batch execution is the intended path |
**Agent Implementation** (parallel mode only): Rule of thumb:
```
# Launch multiple generations in parallel using Task tool - Prefer batch over subagents once prompt files are already saved and the task is "generate all of these"
# Each Task runs as background subagent with run_in_background=true - Use subagents only when generation is coupled with per-image thinking, rewriting, or divergent creative exploration
# Collect results via TaskOutput when all complete
``` Parallel behavior:
- Default worker count is automatic, capped by config, built-in default 10
- Provider-specific throttling is applied only in batch mode, and the built-in defaults are tuned for faster throughput while still avoiding obvious RPM bursts
- You can override worker count with `--jobs <count>`
- Each image retries automatically up to 3 attempts
- Final output includes success count, failure count, and per-image failure reasons
## Error Handling ## Error Handling
- Missing API key → error with setup instructions - Missing API key → error with setup instructions
- Generation failure → auto-retry once - Generation failure → auto-retry up to 3 attempts per image
- Invalid aspect ratio → warning, proceed with default - Invalid aspect ratio → warning, proceed with default
- Reference images with unsupported provider/model → error with fix hint (switch to Google multimodal: `gemini-3-pro-image-preview`, `gemini-3.1-flash-image-preview`; or OpenAI GPT Image edits) - Reference images with unsupported provider/model → error with fix hint
## Extension Support ## Extension Support
@@ -21,9 +21,25 @@ default_image_size: null # 1K|2K|4K|null (Google only, overrides quality)
default_model: default_model:
google: null # e.g., "gemini-3-pro-image-preview", "gemini-3.1-flash-image-preview" google: null # e.g., "gemini-3-pro-image-preview", "gemini-3.1-flash-image-preview"
openai: null # e.g., "gpt-image-1.5" openai: null # e.g., "gpt-image-1.5", "gpt-image-1"
dashscope: null # e.g., "z-image-turbo" dashscope: null # e.g., "z-image-turbo"
replicate: null # e.g., "google/nano-banana-pro" replicate: null # e.g., "google/nano-banana-pro"
batch:
max_workers: 10
provider_limits:
replicate:
concurrency: 5
start_interval_ms: 700
google:
concurrency: 3
start_interval_ms: 1100
openai:
concurrency: 3
start_interval_ms: 1100
dashscope:
concurrency: 3
start_interval_ms: 1100
--- ---
``` ```
@@ -40,6 +56,9 @@ default_model:
| `default_model.openai` | string\|null | null | OpenAI default model | | `default_model.openai` | string\|null | null | OpenAI default model |
| `default_model.dashscope` | string\|null | null | DashScope default model | | `default_model.dashscope` | string\|null | null | DashScope default model |
| `default_model.replicate` | string\|null | null | Replicate default model | | `default_model.replicate` | string\|null | null | Replicate default model |
| `batch.max_workers` | int\|null | 10 | Batch worker cap |
| `batch.provider_limits.<provider>.concurrency` | int\|null | provider default | Max simultaneous requests per provider |
| `batch.provider_limits.<provider>.start_interval_ms` | int\|null | provider default | Minimum gap between request starts per provider |
## Examples ## Examples
@@ -65,5 +84,11 @@ default_model:
openai: "gpt-image-1.5" openai: "gpt-image-1.5"
dashscope: "z-image-turbo" dashscope: "z-image-turbo"
replicate: "google/nano-banana-pro" replicate: "google/nano-banana-pro"
batch:
max_workers: 10
provider_limits:
replicate:
concurrency: 5
start_interval_ms: 700
--- ---
``` ```
+560 -105
View File
@@ -2,34 +2,108 @@ import path from "node:path";
import process from "node:process"; import process from "node:process";
import { homedir } from "node:os"; import { homedir } from "node:os";
import { access, mkdir, readFile, writeFile } from "node:fs/promises"; import { access, mkdir, readFile, writeFile } from "node:fs/promises";
import type { CliArgs, Provider, ExtendConfig } from "./types"; import type {
BatchFile,
BatchTaskInput,
CliArgs,
ExtendConfig,
Provider,
} from "./types";
type ProviderModule = {
getDefaultModel: () => string;
generateImage: (prompt: string, model: string, args: CliArgs) => Promise<Uint8Array>;
};
type PreparedTask = {
id: string;
prompt: string;
args: CliArgs;
provider: Provider;
model: string;
outputPath: string;
providerModule: ProviderModule;
};
type TaskResult = {
id: string;
provider: Provider;
model: string;
outputPath: string;
success: boolean;
attempts: number;
error: string | null;
};
type ProviderRateLimit = {
concurrency: number;
startIntervalMs: number;
};
type LoadedBatchTasks = {
tasks: BatchTaskInput[];
jobs: number | null;
batchDir: string;
};
const MAX_ATTEMPTS = 3;
const DEFAULT_MAX_WORKERS = 10;
const POLL_WAIT_MS = 250;
const DEFAULT_PROVIDER_RATE_LIMITS: Record<Provider, ProviderRateLimit> = {
replicate: { concurrency: 5, startIntervalMs: 700 },
google: { concurrency: 3, startIntervalMs: 1100 },
openai: { concurrency: 3, startIntervalMs: 1100 },
dashscope: { concurrency: 3, startIntervalMs: 1100 },
};
function printUsage(): void { function printUsage(): void {
console.log(`Usage: console.log(`Usage:
npx -y bun scripts/main.ts --prompt "A cat" --image cat.png npx -y bun scripts/main.ts --prompt "A cat" --image cat.png
npx -y bun scripts/main.ts --prompt "A landscape" --image landscape.png --ar 16:9
npx -y bun scripts/main.ts --promptfiles system.md content.md --image out.png npx -y bun scripts/main.ts --promptfiles system.md content.md --image out.png
npx -y bun scripts/main.ts --batchfile batch.json
Options: Options:
-p, --prompt <text> Prompt text -p, --prompt <text> Prompt text
--promptfiles <files...> Read prompt from files (concatenated) --promptfiles <files...> Read prompt from files (concatenated)
--image <path> Output image path (required) --image <path> Output image path (required in single-image mode)
--batchfile <path> JSON batch file for multi-image generation
--jobs <count> Worker count for batch mode (default: auto, max from config, built-in default 10)
--provider google|openai|dashscope|replicate Force provider (auto-detect by default) --provider google|openai|dashscope|replicate Force provider (auto-detect by default)
-m, --model <id> Model ID -m, --model <id> Model ID
--ar <ratio> Aspect ratio (e.g., 16:9, 1:1, 4:3) --ar <ratio> Aspect ratio (e.g., 16:9, 1:1, 4:3)
--size <WxH> Size (e.g., 1024x1024) --size <WxH> Size (e.g., 1024x1024)
--quality normal|2k Quality preset (default: 2k) --quality normal|2k Quality preset (default: 2k)
--imageSize 1K|2K|4K Image size for Google (default: from quality) --imageSize 1K|2K|4K Image size for Google (default: from quality)
--ref <files...> Reference images (Google multimodal or OpenAI edits) --ref <files...> Reference images (Google multimodal, OpenAI GPT Image edits, or Replicate)
--n <count> Number of images (default: 1) --n <count> Number of images for the current task (default: 1)
--json JSON output --json JSON output
-h, --help Show help -h, --help Show help
Batch file format:
{
"jobs": 4,
"tasks": [
{
"id": "hero",
"promptFiles": ["prompts/hero.md"],
"image": "out/hero.png",
"provider": "replicate",
"model": "google/nano-banana-pro",
"ar": "16:9"
}
]
}
Behavior:
- Batch mode automatically runs in parallel when pending tasks >= 2
- Each image retries automatically up to 3 attempts
- Batch summary reports success count, failure count, and per-image errors
Environment variables: Environment variables:
OPENAI_API_KEY OpenAI API key OPENAI_API_KEY OpenAI API key
GOOGLE_API_KEY Google API key GOOGLE_API_KEY Google API key
GEMINI_API_KEY Gemini API key (alias for GOOGLE_API_KEY) GEMINI_API_KEY Gemini API key (alias for GOOGLE_API_KEY)
DASHSCOPE_API_KEY DashScope API key () DASHSCOPE_API_KEY DashScope API key
REPLICATE_API_TOKEN Replicate API token REPLICATE_API_TOKEN Replicate API token
OPENAI_IMAGE_MODEL Default OpenAI model (gpt-image-1.5) OPENAI_IMAGE_MODEL Default OpenAI model (gpt-image-1.5)
GOOGLE_IMAGE_MODEL Default Google model (gemini-3-pro-image-preview) GOOGLE_IMAGE_MODEL Default Google model (gemini-3-pro-image-preview)
@@ -40,6 +114,9 @@ Environment variables:
GOOGLE_BASE_URL Custom Google endpoint GOOGLE_BASE_URL Custom Google endpoint
DASHSCOPE_BASE_URL Custom DashScope endpoint DASHSCOPE_BASE_URL Custom DashScope endpoint
REPLICATE_BASE_URL Custom Replicate endpoint REPLICATE_BASE_URL Custom Replicate endpoint
BAOYU_IMAGE_GEN_MAX_WORKERS Override batch worker cap
BAOYU_IMAGE_GEN_<PROVIDER>_CONCURRENCY Override provider concurrency
BAOYU_IMAGE_GEN_<PROVIDER>_START_INTERVAL_MS Override provider start gap in ms
Env file load order: CLI args > EXTEND.md > process.env > <cwd>/.baoyu-skills/.env > ~/.baoyu-skills/.env`); Env file load order: CLI args > EXTEND.md > process.env > <cwd>/.baoyu-skills/.env > ~/.baoyu-skills/.env`);
} }
@@ -57,6 +134,8 @@ function parseArgs(argv: string[]): CliArgs {
imageSize: null, imageSize: null,
referenceImages: [], referenceImages: [],
n: 1, n: 1,
batchFile: null,
jobs: null,
json: false, json: false,
help: false, help: false,
}; };
@@ -110,9 +189,26 @@ function parseArgs(argv: string[]): CliArgs {
continue; continue;
} }
if (a === "--batchfile") {
const v = argv[++i];
if (!v) throw new Error("Missing value for --batchfile");
out.batchFile = v;
continue;
}
if (a === "--jobs") {
const v = argv[++i];
if (!v) throw new Error("Missing value for --jobs");
out.jobs = parseInt(v, 10);
if (isNaN(out.jobs) || out.jobs < 1) throw new Error(`Invalid worker count: ${v}`);
continue;
}
if (a === "--provider") { if (a === "--provider") {
const v = argv[++i]; const v = argv[++i];
if (v !== "google" && v !== "openai" && v !== "dashscope" && v !== "replicate") throw new Error(`Invalid provider: ${v}`); if (v !== "google" && v !== "openai" && v !== "dashscope" && v !== "replicate") {
throw new Error(`Invalid provider: ${v}`);
}
out.provider = v; out.provider = v;
continue; continue;
} }
@@ -228,9 +324,11 @@ function parseSimpleYaml(yaml: string): Partial<ExtendConfig> {
const config: Partial<ExtendConfig> = {}; const config: Partial<ExtendConfig> = {};
const lines = yaml.split("\n"); const lines = yaml.split("\n");
let currentKey: string | null = null; let currentKey: string | null = null;
let currentProvider: Provider | null = null;
for (const line of lines) { for (const line of lines) {
const trimmed = line.trim(); const trimmed = line.trim();
const indent = line.match(/^\s*/)?.[0].length ?? 0;
if (!trimmed || trimmed.startsWith("#")) continue; if (!trimmed || trimmed.startsWith("#")) continue;
if (trimmed.includes(":") && !trimmed.startsWith("-")) { if (trimmed.includes(":") && !trimmed.startsWith("-")) {
@@ -247,18 +345,57 @@ function parseSimpleYaml(yaml: string): Partial<ExtendConfig> {
} else if (key === "default_provider") { } else if (key === "default_provider") {
config.default_provider = value === "null" ? null : (value as Provider); config.default_provider = value === "null" ? null : (value as Provider);
} else if (key === "default_quality") { } else if (key === "default_quality") {
config.default_quality = value === "null" ? null : (value as "normal" | "2k"); config.default_quality = value === "null" ? null : value as "normal" | "2k";
} else if (key === "default_aspect_ratio") { } else if (key === "default_aspect_ratio") {
const cleaned = value.replace(/['"]/g, ""); const cleaned = value.replace(/['"]/g, "");
config.default_aspect_ratio = cleaned === "null" ? null : cleaned; config.default_aspect_ratio = cleaned === "null" ? null : cleaned;
} else if (key === "default_image_size") { } else if (key === "default_image_size") {
config.default_image_size = value === "null" ? null : (value as "1K" | "2K" | "4K"); config.default_image_size = value === "null" ? null : value as "1K" | "2K" | "4K";
} else if (key === "default_model") { } else if (key === "default_model") {
config.default_model = { google: null, openai: null, dashscope: null, replicate: null }; config.default_model = { google: null, openai: null, dashscope: null, replicate: null };
currentKey = "default_model"; currentKey = "default_model";
} else if (currentKey === "default_model" && (key === "google" || key === "openai" || key === "dashscope" || key === "replicate")) { currentProvider = null;
} else if (key === "batch") {
config.batch = {};
currentKey = "batch";
currentProvider = null;
} else if (currentKey === "batch" && indent >= 2 && key === "max_workers") {
config.batch ??= {};
config.batch.max_workers = value === "null" ? null : parseInt(value, 10);
} else if (currentKey === "batch" && indent >= 2 && key === "provider_limits") {
config.batch ??= {};
config.batch.provider_limits ??= {};
currentKey = "provider_limits";
currentProvider = null;
} else if (
currentKey === "provider_limits" &&
indent >= 4 &&
(key === "google" || key === "openai" || key === "dashscope" || key === "replicate")
) {
config.batch ??= {};
config.batch.provider_limits ??= {};
config.batch.provider_limits[key] ??= {};
currentProvider = key;
} else if (
currentKey === "default_model" &&
(key === "google" || key === "openai" || key === "dashscope" || key === "replicate")
) {
const cleaned = value.replace(/['"]/g, ""); const cleaned = value.replace(/['"]/g, "");
config.default_model![key] = cleaned === "null" ? null : cleaned; config.default_model![key] = cleaned === "null" ? null : cleaned;
} else if (
currentKey === "provider_limits" &&
currentProvider &&
indent >= 6 &&
(key === "concurrency" || key === "start_interval_ms")
) {
config.batch ??= {};
config.batch.provider_limits ??= {};
const providerLimit = (config.batch.provider_limits[currentProvider] ??= {});
if (key === "concurrency") {
providerLimit.concurrency = value === "null" ? null : parseInt(value, 10);
} else {
providerLimit.start_interval_ms = value === "null" ? null : parseInt(value, 10);
}
} }
} }
} }
@@ -280,7 +417,6 @@ async function loadExtendConfig(): Promise<Partial<ExtendConfig>> {
const content = await readFile(p, "utf8"); const content = await readFile(p, "utf8");
const yaml = extractYamlFrontMatter(content); const yaml = extractYamlFrontMatter(content);
if (!yaml) continue; if (!yaml) continue;
return parseSimpleYaml(yaml); return parseSimpleYaml(yaml);
} catch { } catch {
continue; continue;
@@ -300,6 +436,57 @@ function mergeConfig(args: CliArgs, extend: Partial<ExtendConfig>): CliArgs {
}; };
} }
function parsePositiveInt(value: string | undefined): number | null {
if (!value) return null;
const parsed = parseInt(value, 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
}
function parsePositiveBatchInt(value: unknown): number | null {
if (value === null || value === undefined) return null;
if (typeof value === "number") {
return Number.isInteger(value) && value > 0 ? value : null;
}
if (typeof value === "string") {
return parsePositiveInt(value);
}
return null;
}
function getConfiguredMaxWorkers(extendConfig: Partial<ExtendConfig>): number {
const envValue = parsePositiveInt(process.env.BAOYU_IMAGE_GEN_MAX_WORKERS);
const configValue = extendConfig.batch?.max_workers ?? null;
return Math.max(1, envValue ?? configValue ?? DEFAULT_MAX_WORKERS);
}
function getConfiguredProviderRateLimits(
extendConfig: Partial<ExtendConfig>
): Record<Provider, ProviderRateLimit> {
const configured: Record<Provider, ProviderRateLimit> = {
replicate: { ...DEFAULT_PROVIDER_RATE_LIMITS.replicate },
google: { ...DEFAULT_PROVIDER_RATE_LIMITS.google },
openai: { ...DEFAULT_PROVIDER_RATE_LIMITS.openai },
dashscope: { ...DEFAULT_PROVIDER_RATE_LIMITS.dashscope },
};
for (const provider of ["replicate", "google", "openai", "dashscope"] as Provider[]) {
const envPrefix = `BAOYU_IMAGE_GEN_${provider.toUpperCase()}`;
const extendLimit = extendConfig.batch?.provider_limits?.[provider];
configured[provider] = {
concurrency:
parsePositiveInt(process.env[`${envPrefix}_CONCURRENCY`]) ??
extendLimit?.concurrency ??
configured[provider].concurrency,
startIntervalMs:
parsePositiveInt(process.env[`${envPrefix}_START_INTERVAL_MS`]) ??
extendLimit?.start_interval_ms ??
configured[provider].startIntervalMs,
};
}
return configured;
}
async function readPromptFromFiles(files: string[]): Promise<string> { async function readPromptFromFiles(files: string[]): Promise<string> {
const parts: string[] = []; const parts: string[] = [];
for (const f of files) { for (const f of files) {
@@ -311,9 +498,12 @@ async function readPromptFromFiles(files: string[]): Promise<string> {
async function readPromptFromStdin(): Promise<string | null> { async function readPromptFromStdin(): Promise<string | null> {
if (process.stdin.isTTY) return null; if (process.stdin.isTTY) return null;
try { try {
const t = await Bun.stdin.text(); const chunks: Buffer[] = [];
const v = t.trim(); for await (const chunk of process.stdin) {
return v.length > 0 ? v : null; chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
}
const value = Buffer.concat(chunks).toString("utf8").trim();
return value.length > 0 ? value : null;
} catch { } catch {
return null; return null;
} }
@@ -327,7 +517,13 @@ function normalizeOutputImagePath(p: string): string {
} }
function detectProvider(args: CliArgs): Provider { function detectProvider(args: CliArgs): Provider {
if (args.referenceImages.length > 0 && args.provider && args.provider !== "google" && args.provider !== "openai" && args.provider !== "replicate") { if (
args.referenceImages.length > 0 &&
args.provider &&
args.provider !== "google" &&
args.provider !== "openai" &&
args.provider !== "replicate"
) {
throw new Error( throw new Error(
"Reference images require a ref-capable provider. Use --provider google (Gemini multimodal), --provider openai (GPT Image edits), or --provider replicate." "Reference images require a ref-capable provider. Use --provider google (Gemini multimodal), --provider openai (GPT Image edits), or --provider replicate."
); );
@@ -349,7 +545,12 @@ function detectProvider(args: CliArgs): Provider {
); );
} }
const available = [hasGoogle && "google", hasOpenai && "openai", hasDashscope && "dashscope", hasReplicate && "replicate"].filter(Boolean) as Provider[]; const available = [
hasGoogle && "google",
hasOpenai && "openai",
hasDashscope && "dashscope",
hasReplicate && "replicate",
].filter(Boolean) as Provider[];
if (available.length === 1) return available[0]!; if (available.length === 1) return available[0]!;
if (available.length > 1) return available[0]!; if (available.length > 1) return available[0]!;
@@ -371,11 +572,6 @@ async function validateReferenceImages(referenceImages: string[]): Promise<void>
} }
} }
type ProviderModule = {
getDefaultModel: () => string;
generateImage: (prompt: string, model: string, args: CliArgs) => Promise<Uint8Array>;
};
function isRetryableGenerationError(error: unknown): boolean { function isRetryableGenerationError(error: unknown): boolean {
const msg = error instanceof Error ? error.message : String(error); const msg = error instanceof Error ? error.message : String(error);
const nonRetryableMarkers = [ const nonRetryableMarkers = [
@@ -384,26 +580,353 @@ function isRetryableGenerationError(error: unknown): boolean {
"only supported", "only supported",
"No API key found", "No API key found",
"is required", "is required",
"Invalid ",
"Unexpected ",
"API error (400)",
"API error (401)",
"API error (402)",
"API error (403)",
"API error (404)",
"temporarily disabled",
]; ];
return !nonRetryableMarkers.some((marker) => msg.includes(marker)); return !nonRetryableMarkers.some((marker) => msg.includes(marker));
} }
async function loadProviderModule(provider: Provider): Promise<ProviderModule> { async function loadProviderModule(provider: Provider): Promise<ProviderModule> {
if (provider === "google") { if (provider === "google") return (await import("./providers/google")) as ProviderModule;
return (await import("./providers/google")) as ProviderModule; if (provider === "dashscope") return (await import("./providers/dashscope")) as ProviderModule;
} if (provider === "replicate") return (await import("./providers/replicate")) as ProviderModule;
if (provider === "dashscope") {
return (await import("./providers/dashscope")) as ProviderModule;
}
if (provider === "replicate") {
return (await import("./providers/replicate")) as ProviderModule;
}
return (await import("./providers/openai")) as ProviderModule; return (await import("./providers/openai")) as ProviderModule;
} }
async function loadPromptForArgs(args: CliArgs): Promise<string | null> {
let prompt: string | null = args.prompt;
if (!prompt && args.promptFiles.length > 0) {
prompt = await readPromptFromFiles(args.promptFiles);
}
return prompt;
}
function getModelForProvider(
provider: Provider,
requestedModel: string | null,
extendConfig: Partial<ExtendConfig>,
providerModule: ProviderModule
): string {
if (requestedModel) return requestedModel;
if (extendConfig.default_model) {
if (provider === "google" && extendConfig.default_model.google) return extendConfig.default_model.google;
if (provider === "openai" && extendConfig.default_model.openai) return extendConfig.default_model.openai;
if (provider === "dashscope" && extendConfig.default_model.dashscope) return extendConfig.default_model.dashscope;
if (provider === "replicate" && extendConfig.default_model.replicate) return extendConfig.default_model.replicate;
}
return providerModule.getDefaultModel();
}
async function prepareSingleTask(args: CliArgs, extendConfig: Partial<ExtendConfig>): Promise<PreparedTask> {
if (!args.quality) args.quality = "2k";
const prompt = (await loadPromptForArgs(args)) ?? (await readPromptFromStdin());
if (!prompt) throw new Error("Prompt is required");
if (!args.imagePath) throw new Error("--image is required");
if (args.referenceImages.length > 0) await validateReferenceImages(args.referenceImages);
const provider = detectProvider(args);
const providerModule = await loadProviderModule(provider);
const model = getModelForProvider(provider, args.model, extendConfig, providerModule);
return {
id: "single",
prompt,
args,
provider,
model,
outputPath: normalizeOutputImagePath(args.imagePath),
providerModule,
};
}
async function loadBatchTasks(batchFilePath: string): Promise<LoadedBatchTasks> {
const resolvedBatchFilePath = path.resolve(batchFilePath);
const content = await readFile(resolvedBatchFilePath, "utf8");
const parsed = JSON.parse(content.replace(/^\uFEFF/, "")) as BatchFile;
const batchDir = path.dirname(resolvedBatchFilePath);
if (Array.isArray(parsed)) {
return {
tasks: parsed,
jobs: null,
batchDir,
};
}
if (parsed && typeof parsed === "object" && Array.isArray(parsed.tasks)) {
const jobs = parsePositiveBatchInt(parsed.jobs);
if (parsed.jobs !== undefined && parsed.jobs !== null && jobs === null) {
throw new Error("Invalid batch file. jobs must be a positive integer when provided.");
}
return {
tasks: parsed.tasks,
jobs,
batchDir,
};
}
throw new Error("Invalid batch file. Expected an array of tasks or an object with a tasks array.");
}
function resolveBatchPath(batchDir: string, filePath: string): string {
return path.isAbsolute(filePath) ? filePath : path.resolve(batchDir, filePath);
}
function createTaskArgs(baseArgs: CliArgs, task: BatchTaskInput, batchDir: string): CliArgs {
return {
...baseArgs,
prompt: task.prompt ?? null,
promptFiles: task.promptFiles ? task.promptFiles.map((filePath) => resolveBatchPath(batchDir, filePath)) : [],
imagePath: task.image ? resolveBatchPath(batchDir, task.image) : null,
provider: task.provider ?? baseArgs.provider ?? null,
model: task.model ?? baseArgs.model ?? null,
aspectRatio: task.ar ?? baseArgs.aspectRatio ?? null,
size: task.size ?? baseArgs.size ?? null,
quality: task.quality ?? baseArgs.quality ?? null,
imageSize: task.imageSize ?? baseArgs.imageSize ?? null,
referenceImages: task.ref ? task.ref.map((filePath) => resolveBatchPath(batchDir, filePath)) : [],
n: task.n ?? baseArgs.n,
batchFile: null,
jobs: baseArgs.jobs,
json: baseArgs.json,
help: false,
};
}
async function prepareBatchTasks(
args: CliArgs,
extendConfig: Partial<ExtendConfig>
): Promise<{ tasks: PreparedTask[]; jobs: number | null }> {
if (!args.batchFile) throw new Error("--batchfile is required in batch mode");
const { tasks: taskInputs, jobs: batchJobs, batchDir } = await loadBatchTasks(args.batchFile);
if (taskInputs.length === 0) throw new Error("Batch file does not contain any tasks.");
const prepared: PreparedTask[] = [];
for (let i = 0; i < taskInputs.length; i++) {
const task = taskInputs[i]!;
const taskArgs = createTaskArgs(args, task, batchDir);
const prompt = await loadPromptForArgs(taskArgs);
if (!prompt) throw new Error(`Task ${i + 1} is missing prompt or promptFiles.`);
if (!taskArgs.imagePath) throw new Error(`Task ${i + 1} is missing image output path.`);
if (taskArgs.referenceImages.length > 0) await validateReferenceImages(taskArgs.referenceImages);
const provider = detectProvider(taskArgs);
const providerModule = await loadProviderModule(provider);
const model = getModelForProvider(provider, taskArgs.model, extendConfig, providerModule);
prepared.push({
id: task.id || `task-${String(i + 1).padStart(2, "0")}`,
prompt,
args: taskArgs,
provider,
model,
outputPath: normalizeOutputImagePath(taskArgs.imagePath),
providerModule,
});
}
return {
tasks: prepared,
jobs: args.jobs ?? batchJobs,
};
}
async function writeImage(outputPath: string, imageData: Uint8Array): Promise<void> {
await mkdir(path.dirname(outputPath), { recursive: true });
await writeFile(outputPath, imageData);
}
async function generatePreparedTask(task: PreparedTask): Promise<TaskResult> {
console.error(`Using ${task.provider} / ${task.model} for ${task.id}`);
console.error(
`Switch model: --model <id> | EXTEND.md default_model.${task.provider} | env ${task.provider.toUpperCase()}_IMAGE_MODEL`
);
let attempts = 0;
while (attempts < MAX_ATTEMPTS) {
attempts += 1;
try {
const imageData = await task.providerModule.generateImage(task.prompt, task.model, task.args);
await writeImage(task.outputPath, imageData);
return {
id: task.id,
provider: task.provider,
model: task.model,
outputPath: task.outputPath,
success: true,
attempts,
error: null,
};
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
const canRetry = attempts < MAX_ATTEMPTS && isRetryableGenerationError(error);
if (canRetry) {
console.error(`[${task.id}] Attempt ${attempts}/${MAX_ATTEMPTS} failed, retrying...`);
continue;
}
return {
id: task.id,
provider: task.provider,
model: task.model,
outputPath: task.outputPath,
success: false,
attempts,
error: message,
};
}
}
return {
id: task.id,
provider: task.provider,
model: task.model,
outputPath: task.outputPath,
success: false,
attempts: MAX_ATTEMPTS,
error: "Unknown failure",
};
}
function createProviderGate(providerRateLimits: Record<Provider, ProviderRateLimit>) {
const state = new Map<Provider, { active: number; lastStartedAt: number }>();
return async function acquire(provider: Provider): Promise<() => void> {
const limit = providerRateLimits[provider];
while (true) {
const current = state.get(provider) ?? { active: 0, lastStartedAt: 0 };
const now = Date.now();
const enoughCapacity = current.active < limit.concurrency;
const enoughGap = now - current.lastStartedAt >= limit.startIntervalMs;
if (enoughCapacity && enoughGap) {
state.set(provider, { active: current.active + 1, lastStartedAt: now });
return () => {
const latest = state.get(provider) ?? { active: 1, lastStartedAt: now };
state.set(provider, {
active: Math.max(0, latest.active - 1),
lastStartedAt: latest.lastStartedAt,
});
};
}
await new Promise((resolve) => setTimeout(resolve, POLL_WAIT_MS));
}
};
}
function getWorkerCount(taskCount: number, jobs: number | null, maxWorkers: number): number {
const requested = jobs ?? Math.min(taskCount, maxWorkers);
return Math.max(1, Math.min(requested, taskCount, maxWorkers));
}
async function runBatchTasks(
tasks: PreparedTask[],
jobs: number | null,
extendConfig: Partial<ExtendConfig>
): Promise<TaskResult[]> {
if (tasks.length === 1) {
return [await generatePreparedTask(tasks[0]!)];
}
const maxWorkers = getConfiguredMaxWorkers(extendConfig);
const providerRateLimits = getConfiguredProviderRateLimits(extendConfig);
const acquireProvider = createProviderGate(providerRateLimits);
const workerCount = getWorkerCount(tasks.length, jobs, maxWorkers);
console.error(`Batch mode: ${tasks.length} tasks, ${workerCount} workers, parallel mode enabled.`);
for (const provider of ["replicate", "google", "openai", "dashscope"] as Provider[]) {
const limit = providerRateLimits[provider];
console.error(`- ${provider}: concurrency=${limit.concurrency}, startIntervalMs=${limit.startIntervalMs}`);
}
let nextIndex = 0;
const results: TaskResult[] = new Array(tasks.length);
const worker = async (): Promise<void> => {
while (true) {
const currentIndex = nextIndex;
nextIndex += 1;
if (currentIndex >= tasks.length) return;
const task = tasks[currentIndex]!;
const release = await acquireProvider(task.provider);
try {
results[currentIndex] = await generatePreparedTask(task);
} finally {
release();
}
}
};
await Promise.all(Array.from({ length: workerCount }, () => worker()));
return results;
}
function printBatchSummary(results: TaskResult[]): void {
const successCount = results.filter((result) => result.success).length;
const failureCount = results.length - successCount;
console.error("");
console.error("Batch generation summary:");
console.error(`- Total: ${results.length}`);
console.error(`- Succeeded: ${successCount}`);
console.error(`- Failed: ${failureCount}`);
if (failureCount > 0) {
console.error("Failure reasons:");
for (const result of results.filter((item) => !item.success)) {
console.error(`- ${result.id}: ${result.error}`);
}
}
}
function emitJson(payload: unknown): void {
console.log(JSON.stringify(payload, null, 2));
}
async function runSingleMode(args: CliArgs, extendConfig: Partial<ExtendConfig>): Promise<void> {
const task = await prepareSingleTask(args, extendConfig);
const result = await generatePreparedTask(task);
if (!result.success) {
throw new Error(result.error || "Generation failed");
}
if (args.json) {
emitJson({
savedImage: result.outputPath,
provider: result.provider,
model: result.model,
attempts: result.attempts,
prompt: task.prompt.slice(0, 200),
});
return;
}
console.log(result.outputPath);
}
async function runBatchMode(args: CliArgs, extendConfig: Partial<ExtendConfig>): Promise<void> {
const { tasks, jobs } = await prepareBatchTasks(args, extendConfig);
const results = await runBatchTasks(tasks, jobs, extendConfig);
printBatchSummary(results);
if (args.json) {
emitJson({
mode: "batch",
total: results.length,
succeeded: results.filter((item) => item.success).length,
failed: results.filter((item) => !item.success).length,
results,
});
}
if (results.some((item) => !item.success)) {
process.exitCode = 1;
}
}
async function main(): Promise<void> { async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2)); const args = parseArgs(process.argv.slice(2));
if (args.help) { if (args.help) {
printUsage(); printUsage();
return; return;
@@ -412,86 +935,18 @@ async function main(): Promise<void> {
await loadEnv(); await loadEnv();
const extendConfig = await loadExtendConfig(); const extendConfig = await loadExtendConfig();
const mergedArgs = mergeConfig(args, extendConfig); const mergedArgs = mergeConfig(args, extendConfig);
if (!mergedArgs.quality) mergedArgs.quality = "2k"; if (!mergedArgs.quality) mergedArgs.quality = "2k";
let prompt: string | null = mergedArgs.prompt; if (mergedArgs.batchFile) {
if (!prompt && mergedArgs.promptFiles.length > 0) prompt = await readPromptFromFiles(mergedArgs.promptFiles); await runBatchMode(mergedArgs, extendConfig);
if (!prompt) prompt = await readPromptFromStdin();
if (!prompt) {
console.error("Error: Prompt is required");
printUsage();
process.exitCode = 1;
return; return;
} }
if (!mergedArgs.imagePath) { await runSingleMode(mergedArgs, extendConfig);
console.error("Error: --image is required");
printUsage();
process.exitCode = 1;
return;
} }
if (mergedArgs.referenceImages.length > 0) { main().catch((error) => {
await validateReferenceImages(mergedArgs.referenceImages); const message = error instanceof Error ? error.message : String(error);
} console.error(message);
const provider = detectProvider(mergedArgs);
const providerModule = await loadProviderModule(provider);
let model = mergedArgs.model;
if (!model && extendConfig.default_model) {
if (provider === "google") model = extendConfig.default_model.google ?? null;
if (provider === "openai") model = extendConfig.default_model.openai ?? null;
if (provider === "dashscope") model = extendConfig.default_model.dashscope ?? null;
if (provider === "replicate") model = extendConfig.default_model.replicate ?? null;
}
model = model || providerModule.getDefaultModel();
const outputPath = normalizeOutputImagePath(mergedArgs.imagePath);
let imageData: Uint8Array;
let retried = false;
while (true) {
try {
imageData = await providerModule.generateImage(prompt, model, mergedArgs);
break;
} catch (e) {
if (!retried && isRetryableGenerationError(e)) {
retried = true;
console.error("Generation failed, retrying...");
continue;
}
throw e;
}
}
const dir = path.dirname(outputPath);
await mkdir(dir, { recursive: true });
await writeFile(outputPath, imageData);
if (mergedArgs.json) {
console.log(
JSON.stringify(
{
savedImage: outputPath,
provider,
model,
prompt: prompt.slice(0, 200),
},
null,
2
)
);
} else {
console.log(outputPath);
}
}
main().catch((e) => {
const msg = e instanceof Error ? e.message : String(e);
console.error(msg);
process.exit(1); process.exit(1);
}); });
@@ -1,6 +1,6 @@
import path from "node:path"; import path from "node:path";
import { readFile } from "node:fs/promises"; import { readFile } from "node:fs/promises";
import { execSync } from "node:child_process"; import { execFileSync } from "node:child_process";
import type { CliArgs } from "../types"; import type { CliArgs } from "../types";
const GOOGLE_MULTIMODAL_MODELS = [ const GOOGLE_MULTIMODAL_MODELS = [
@@ -76,14 +76,43 @@ async function postGoogleJsonViaCurl<T>(
): Promise<T> { ): Promise<T> {
const proxy = getHttpProxy(); const proxy = getHttpProxy();
const bodyStr = JSON.stringify(body); const bodyStr = JSON.stringify(body);
const proxyArgs = proxy ? `-x "${proxy}"` : ""; const args = [
"-s",
"--connect-timeout",
"30",
"--max-time",
"300",
...(proxy ? ["-x", proxy] : []),
url,
"-H",
"Content-Type: application/json",
"-H",
`x-goog-api-key: ${apiKey}`,
"-d",
"@-",
];
const result = execSync( let result = "";
`curl -s --connect-timeout 30 --max-time 300 ${proxyArgs} "${url}" -H "Content-Type: application/json" -H "x-goog-api-key: ${apiKey}" -d @-`, try {
{ input: bodyStr, maxBuffer: 100 * 1024 * 1024, timeout: 310000 }, result = execFileSync("curl", args, {
); input: bodyStr,
encoding: "utf8",
maxBuffer: 100 * 1024 * 1024,
timeout: 310000,
});
} catch (error) {
const e = error as { message?: string; stderr?: string | Buffer };
const stderrText =
typeof e.stderr === "string"
? e.stderr
: e.stderr
? e.stderr.toString("utf8")
: "";
const details = stderrText.trim() || e.message || "curl request failed";
throw new Error(`Google API request failed via curl: ${details}`);
}
const parsed = JSON.parse(result.toString()) as any; const parsed = JSON.parse(result) as any;
if (parsed.error) { if (parsed.error) {
throw new Error( throw new Error(
`Google API error (${parsed.error.code}): ${parsed.error.message}`, `Google API error (${parsed.error.code}): ${parsed.error.message}`,
@@ -68,7 +68,11 @@ export async function generateImage(
const baseURL = process.env.OPENAI_BASE_URL || "https://api.openai.com/v1"; const baseURL = process.env.OPENAI_BASE_URL || "https://api.openai.com/v1";
const apiKey = process.env.OPENAI_API_KEY; const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) throw new Error("OPENAI_API_KEY is required"); if (!apiKey) {
throw new Error(
"OPENAI_API_KEY is required. Codex/ChatGPT desktop login does not automatically grant OpenAI Images API access to this script."
);
}
if (process.env.OPENAI_IMAGE_USE_CHAT === "true") { if (process.env.OPENAI_IMAGE_USE_CHAT === "true") {
return generateWithChatCompletions(baseURL, apiKey, prompt, model); return generateWithChatCompletions(baseURL, apiKey, prompt, model);
@@ -36,22 +36,24 @@ function buildInput(prompt: string, args: CliArgs, referenceImages: string[]): R
if (args.aspectRatio) { if (args.aspectRatio) {
input.aspect_ratio = args.aspectRatio; input.aspect_ratio = args.aspectRatio;
} else if (referenceImages.length > 0) {
input.aspect_ratio = "match_input_image";
} }
if (args.n > 1) { if (args.n > 1) {
input.number_of_images = args.n; input.number_of_images = args.n;
} }
if (args.quality === "normal") {
input.resolution = "1K";
} else if (args.quality === "2k") {
input.resolution = "2K";
}
input.output_format = "png"; input.output_format = "png";
if (referenceImages.length > 0) { if (referenceImages.length > 0) {
if (referenceImages.length === 1) { input.image_input = referenceImages;
input.image = referenceImages[0];
} else {
for (let i = 0; i < referenceImages.length; i++) {
input[`image${i > 0 ? i + 1 : ""}`] = referenceImages[i];
}
}
} }
return input; return input;
+36
View File
@@ -13,10 +13,34 @@ export type CliArgs = {
imageSize: string | null; imageSize: string | null;
referenceImages: string[]; referenceImages: string[];
n: number; n: number;
batchFile: string | null;
jobs: number | null;
json: boolean; json: boolean;
help: boolean; help: boolean;
}; };
export type BatchTaskInput = {
id?: string;
prompt?: string | null;
promptFiles?: string[];
image?: string;
provider?: Provider | null;
model?: string | null;
ar?: string | null;
size?: string | null;
quality?: Quality | null;
imageSize?: "1K" | "2K" | "4K" | null;
ref?: string[];
n?: number;
};
export type BatchFile =
| BatchTaskInput[]
| {
tasks: BatchTaskInput[];
jobs?: number | null;
};
export type ExtendConfig = { export type ExtendConfig = {
version: number; version: number;
default_provider: Provider | null; default_provider: Provider | null;
@@ -29,4 +53,16 @@ export type ExtendConfig = {
dashscope: string | null; dashscope: string | null;
replicate: string | null; replicate: string | null;
}; };
batch?: {
max_workers?: number | null;
provider_limits?: Partial<
Record<
Provider,
{
concurrency?: number | null;
start_interval_ms?: number | null;
}
>
>;
};
}; };
+10 -2
View File
@@ -17,6 +17,7 @@ Two dimensions: **layout** (information structure) × **style** (visual aestheti
/baoyu-infographic path/to/content.md /baoyu-infographic path/to/content.md
/baoyu-infographic path/to/content.md --layout hierarchical-layers --style technical-schematic /baoyu-infographic path/to/content.md --layout hierarchical-layers --style technical-schematic
/baoyu-infographic path/to/content.md --aspect portrait --lang zh /baoyu-infographic path/to/content.md --aspect portrait --lang zh
/baoyu-infographic path/to/content.md --aspect 3:4
/baoyu-infographic # then paste content /baoyu-infographic # then paste content
``` ```
@@ -26,7 +27,7 @@ Two dimensions: **layout** (information structure) × **style** (visual aestheti
|--------|--------| |--------|--------|
| `--layout` | 21 options (see Layout Gallery), default: bento-grid | | `--layout` | 21 options (see Layout Gallery), default: bento-grid |
| `--style` | 20 options (see Style Gallery), default: craft-handmade | | `--style` | 20 options (see Style Gallery), default: craft-handmade |
| `--aspect` | landscape (16:9), portrait (9:16), square (1:1) | | `--aspect` | Named: landscape (16:9), portrait (9:16), square (1:1). Custom: any W:H ratio (e.g., 3:4, 4:3, 2.35:1) |
| `--lang` | en, zh, ja, etc. | | `--lang` | en, zh, ja, etc. |
## Layout Gallery ## Layout Gallery
@@ -147,12 +148,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-infographic/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-infographic/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-infographic/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-infographic/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-infographic/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-infographic/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-infographic/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-infographic/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-infographic/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-infographic/EXTEND.md") { "user" }
``` ```
@@ -217,7 +221,7 @@ Use **single AskUserQuestion call** with multiple questions to confirm all optio
| Question | When | Options | | Question | When | Options |
|----------|------|---------| |----------|------|---------|
| **Combination** | Always | 3+ layout×style combos with rationale | | **Combination** | Always | 3+ layout×style combos with rationale |
| **Aspect** | Always | landscape (16:9), portrait (9:16), square (1:1) | | **Aspect** | Always | Named presets (landscape/portrait/square) or custom W:H ratio (e.g., 3:4, 4:3, 2.35:1) |
| **Language** | Only if source ≠ user language | Language for text content | | **Language** | Only if source ≠ user language | Language for text content |
**Important**: Do NOT split into separate AskUserQuestion calls. Combine all applicable questions into one call. **Important**: Do NOT split into separate AskUserQuestion calls. Combine all applicable questions into one call.
@@ -233,6 +237,10 @@ Combine:
4. Structured content from Step 2 4. Structured content from Step 2
5. All text in confirmed language 5. All text in confirmed language
**Aspect ratio resolution** for `{{ASPECT_RATIO}}`:
- Named presets → ratio string: landscape→`16:9`, portrait→`9:16`, square→`1:1`
- Custom W:H ratios → use as-is (e.g., `3:4`, `4:3`, `2.35:1`)
### Step 6: Generate Image ### Step 6: Generate Image
1. Select available image generation skill (ask user if multiple) 1. Select available image generation skill (ask user if multiple)
+3
View File
@@ -30,12 +30,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-markdown-to-html/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-markdown-to-html/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-markdown-to-html/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-markdown-to-html/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-markdown-to-html/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-markdown-to-html/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-markdown-to-html/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-markdown-to-html/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-markdown-to-html/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-markdown-to-html/EXTEND.md") { "user" }
``` ```
@@ -37,6 +37,10 @@ function parseExtendYaml(yaml: string): Partial<ExtendConfig> {
export function loadExtendConfig(): Partial<ExtendConfig> { export function loadExtendConfig(): Partial<ExtendConfig> {
const paths = [ const paths = [
path.join(process.cwd(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"), path.join(process.cwd(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"),
path.join(
process.env.XDG_CONFIG_HOME || path.join(homedir(), ".config"),
"baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"
),
path.join(homedir(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"), path.join(homedir(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"),
]; ];
for (const p of paths) { for (const p of paths) {
+111 -1
View File
@@ -36,12 +36,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-post-to-wechat/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-post-to-wechat/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-post-to-wechat/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-post-to-wechat/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-post-to-wechat/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-post-to-wechat/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-post-to-wechat/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-post-to-wechat/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-wechat/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-wechat/EXTEND.md") { "user" }
``` ```
@@ -92,9 +95,115 @@ chrome_profile_path: /path/to/chrome/profile
**Value priority**: **Value priority**:
1. CLI arguments 1. CLI arguments
2. Frontmatter 2. Frontmatter
3. EXTEND.md 3. EXTEND.md (account-level → global-level)
4. Skill defaults 4. Skill defaults
## Multi-Account Support
EXTEND.md supports managing multiple WeChat Official Accounts. When `accounts:` block is present, each account can have its own credentials, Chrome profile, and default settings.
**Compatibility rules**:
| Condition | Mode | Behavior |
|-----------|------|----------|
| No `accounts` block | Single-account | Current behavior, unchanged |
| `accounts` with 1 entry | Single-account | Auto-select, no prompt |
| `accounts` with 2+ entries | Multi-account | Prompt to select before publishing |
| `accounts` with `default: true` | Multi-account | Pre-select default, user can switch |
**Multi-account EXTEND.md example**:
```md
default_theme: default
default_color: blue
accounts:
- name: 宝玉的技术分享
alias: baoyu
default: true
default_publish_method: api
default_author: 宝玉
need_open_comment: 1
only_fans_can_comment: 0
app_id: your_wechat_app_id
app_secret: your_wechat_app_secret
- name: AI工具集
alias: ai-tools
default_publish_method: browser
default_author: AI工具集
need_open_comment: 1
only_fans_can_comment: 0
```
**Per-account keys** (can be set per-account or globally as fallback):
`default_publish_method`, `default_author`, `need_open_comment`, `only_fans_can_comment`, `app_id`, `app_secret`, `chrome_profile_path`
**Global-only keys** (always shared across accounts):
`default_theme`, `default_color`
### Account Selection (Step 0.5)
Insert between Step 0 and Step 1 in the Article Posting Workflow:
```
if no accounts block:
→ single-account mode (current behavior)
elif accounts.length == 1:
→ auto-select the only account
elif --account <alias> CLI arg:
→ select matching account
elif one account has default: true:
→ pre-select, show: "Using account: <name> (--account to switch)"
else:
→ prompt user:
"Multiple WeChat accounts configured:
1) <name1> (<alias1>)
2) <name2> (<alias2>)
Select account [1-N]:"
```
### Credential Resolution (API Method)
For a selected account with alias `{alias}`:
1. `app_id` / `app_secret` inline in EXTEND.md account block
2. Env var `WECHAT_{ALIAS}_APP_ID` / `WECHAT_{ALIAS}_APP_SECRET` (alias uppercased, hyphens → underscores)
3. `.baoyu-skills/.env` with prefixed key `WECHAT_{ALIAS}_APP_ID`
4. `~/.baoyu-skills/.env` with prefixed key
5. Fallback to unprefixed `WECHAT_APP_ID` / `WECHAT_APP_SECRET`
**.env multi-account example**:
```bash
# Account: baoyu
WECHAT_BAOYU_APP_ID=your_wechat_app_id
WECHAT_BAOYU_APP_SECRET=your_wechat_app_secret
# Account: ai-tools
WECHAT_AI_TOOLS_APP_ID=your_ai_tools_wechat_app_id
WECHAT_AI_TOOLS_APP_SECRET=your_ai_tools_wechat_app_secret
```
### Chrome Profile (Browser Method)
Each account uses an isolated Chrome profile for independent login sessions:
| Source | Path |
|--------|------|
| Account `chrome_profile_path` in EXTEND.md | Use as-is |
| Auto-generated from alias | `{shared_profile_parent}/wechat-{alias}/` |
| Single-account fallback | Shared default profile (current behavior) |
### CLI `--account` Argument
All publishing scripts accept `--account <alias>`:
```bash
${BUN_X} {baseDir}/scripts/wechat-api.ts <file> --theme default --account ai-tools
${BUN_X} {baseDir}/scripts/wechat-article.ts --markdown <file> --theme default --account baoyu
${BUN_X} {baseDir}/scripts/wechat-browser.ts --markdown <file> --images ./photos/ --account baoyu
```
## Pre-flight Check (Optional) ## Pre-flight Check (Optional)
Before first use, suggest running the environment check. User can skip if they prefer. Before first use, suggest running the environment check. User can skip if they prefer.
@@ -136,6 +245,7 @@ Copy this checklist and check off items as you complete them:
``` ```
Publishing Progress: Publishing Progress:
- [ ] Step 0: Load preferences (EXTEND.md) - [ ] Step 0: Load preferences (EXTEND.md)
- [ ] Step 0.5: Resolve account (multi-account only)
- [ ] Step 1: Determine input type - [ ] Step 1: Determine input type
- [ ] Step 2: Select method and configure credentials - [ ] Step 2: Select method and configure credentials
- [ ] Step 3: Resolve theme/color and validate metadata - [ ] Step 3: Resolve theme/color and validate metadata
@@ -152,6 +152,8 @@ options:
## EXTEND.md Template ## EXTEND.md Template
### Single Account (Default)
```md ```md
default_theme: [default/grace/simple/modern] default_theme: [default/grace/simple/modern]
default_color: [preset name, hex, or empty for theme default] default_color: [preset name, hex, or empty for theme default]
@@ -162,6 +164,40 @@ only_fans_can_comment: [1/0]
chrome_profile_path: chrome_profile_path:
``` ```
### Multi-Account
```md
default_theme: [default/grace/simple/modern]
default_color: [preset name, hex, or empty for theme default]
accounts:
- name: [display name]
alias: [short key, e.g. "baoyu"]
default: true
default_publish_method: [api/browser]
default_author: [author name]
need_open_comment: [1/0]
only_fans_can_comment: [1/0]
app_id: [WeChat App ID, optional]
app_secret: [WeChat App Secret, optional]
- name: [second account name]
alias: [short key, e.g. "ai-tools"]
default_publish_method: [api/browser]
default_author: [author name]
need_open_comment: [1/0]
only_fans_can_comment: [1/0]
```
## Adding More Accounts Later
After initial setup, users can add accounts by editing EXTEND.md:
1. Add an `accounts:` block with list items
2. Move per-account settings (author, publish method, comments) into each account entry
3. Keep global settings (theme, color) at the top level
4. Each account needs a unique `alias` (used for CLI `--account` arg and Chrome profile naming)
5. Set `default: true` on the primary account
## Modifying Preferences Later ## Modifying Preferences Later
Users can edit EXTEND.md directly or delete it to trigger setup again. Users can edit EXTEND.md directly or delete it to trigger setup again.
@@ -0,0 +1,14 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-post-to-wechat-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
},
},
},
"packages": {
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
}
}
+156 -210
View File
@@ -1,172 +1,89 @@
import { spawn } from 'node:child_process'; import { execSync, type ChildProcess } from 'node:child_process';
import fs from 'node:fs';
import { mkdir } from 'node:fs/promises';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
export function sleep(ms: number): Promise<void> { import {
return new Promise((resolve) => setTimeout(resolve, ms)); CdpConnection,
findChromeExecutable as findChromeExecutableBase,
findExistingChromeDebugPort as findExistingChromeDebugPortBase,
getFreePort as getFreePortBase,
launchChrome as launchChromeBase,
resolveSharedChromeProfileDir,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
export { CdpConnection, sleep, waitForChromeDebugPort };
const CHROME_CANDIDATES_FULL: PlatformCandidates = {
darwin: [
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
],
win32: [
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
],
default: [
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
],
};
let wslHome: string | null | undefined;
function getWslWindowsHome(): string | null {
if (wslHome !== undefined) return wslHome;
if (!process.env.WSL_DISTRO_NAME) {
wslHome = null;
return null;
}
try {
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', {
encoding: 'utf-8',
timeout: 5_000,
}).trim().replace(/\r/g, '');
wslHome = execSync(`wslpath -u "${raw}"`, {
encoding: 'utf-8',
timeout: 5_000,
}).trim() || null;
} catch {
wslHome = null;
}
return wslHome;
} }
export async function getFreePort(): Promise<number> { export async function getFreePort(): Promise<number> {
return new Promise((resolve, reject) => { return await getFreePortBase('WECHAT_BROWSER_DEBUG_PORT');
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
} }
export function findChromeExecutable(): string | undefined { export function findChromeExecutable(chromePathOverride?: string): string | undefined {
const override = process.env.WECHAT_BROWSER_CHROME_PATH?.trim(); if (chromePathOverride?.trim()) return chromePathOverride.trim();
if (override && fs.existsSync(override)) return override; return findChromeExecutableBase({
candidates: CHROME_CANDIDATES_FULL,
const candidates: string[] = []; envNames: ['WECHAT_BROWSER_CHROME_PATH'],
switch (process.platform) { });
case 'darwin':
candidates.push(
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
);
break;
case 'win32':
candidates.push(
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
);
break;
default:
candidates.push('/usr/bin/google-chrome', '/usr/bin/chromium');
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return undefined;
} }
export function getDefaultProfileDir(): string { export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim(); return resolveSharedChromeProfileDir({
if (override) return path.resolve(override); envNames: ['BAOYU_CHROME_PROFILE_DIR', 'WECHAT_BROWSER_PROFILE_DIR'],
const base = process.platform === 'darwin' wslWindowsHome: getWslWindowsHome(),
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
async function fetchJson<T = unknown>(url: string): Promise<T> {
const res = await fetch(url, { redirect: 'follow' });
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
}
async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error('Missing webSocketDebuggerUrl');
} catch (error) {
lastError = error;
}
await sleep(200);
}
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, { resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) handlers.forEach((h) => h(msg.params));
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
}); });
} }
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> { export function getAccountProfileDir(alias: string): string {
const ws = new WebSocket(url); const base = getDefaultProfileDir();
await new Promise<void>((resolve, reject) => { return path.join(path.dirname(base), `wechat-${alias}`);
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) this.eventHandlers.set(method, new Set());
this.eventHandlers.get(method)!.add(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? 15_000;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0 ? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs) : null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
} }
export interface ChromeSession { export interface ChromeSession {
@@ -177,56 +94,38 @@ export interface ChromeSession {
export async function tryConnectExisting(port: number): Promise<CdpConnection | null> { export async function tryConnectExisting(port: number): Promise<CdpConnection | null> {
try { try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`); const wsUrl = await waitForChromeDebugPort(port, 5_000, { includeLastError: true });
if (version.webSocketDebuggerUrl) { return await CdpConnection.connect(wsUrl, 5_000);
const cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, 5_000); } catch {
return cdp;
}
} catch {}
return null; return null;
} }
export async function findExistingChromeDebugPort(): Promise<number | null> {
if (process.platform !== 'darwin' && process.platform !== 'linux') return null;
try {
const { execSync } = await import('node:child_process');
const cmd = process.platform === 'darwin'
? `lsof -nP -iTCP -sTCP:LISTEN 2>/dev/null | grep -i 'google\\|chrome' | awk '{print $9}' | sed 's/.*://'`
: `ss -tlnp 2>/dev/null | grep -i chrome | awk '{print $4}' | sed 's/.*://'`;
const output = execSync(cmd, { encoding: 'utf-8', timeout: 5_000 }).trim();
if (!output) return null;
const ports = output.split('\n').map(p => parseInt(p, 10)).filter(p => !isNaN(p) && p > 0);
for (const port of ports) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return port;
} catch {}
}
} catch {}
return null;
} }
export async function launchChrome(url: string, profileDir?: string): Promise<{ cdp: CdpConnection; chrome: ReturnType<typeof spawn> }> { export async function findExistingChromeDebugPort(profileDir = getDefaultProfileDir()): Promise<number | null> {
const chromePath = findChromeExecutable(); return await findExistingChromeDebugPortBase({ profileDir });
}
export async function launchChrome(
url: string,
profileDir?: string,
chromePathOverride?: string,
): Promise<{ cdp: CdpConnection; chrome: ChildProcess }> {
const chromePath = findChromeExecutable(chromePathOverride);
if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.');
const profile = profileDir ?? getDefaultProfileDir(); const profile = profileDir ?? getDefaultProfileDir();
await mkdir(profile, { recursive: true });
const port = await getFreePort(); const port = await getFreePort();
console.log(`[cdp] Launching Chrome (profile: ${profile})`); console.log(`[cdp] Launching Chrome (profile: ${profile})`);
const chrome = spawn(chromePath, [ const chrome = await launchChromeBase({
`--remote-debugging-port=${port}`, chromePath,
`--user-data-dir=${profile}`, profileDir: profile,
'--no-first-run', port,
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
url, url,
], { stdio: 'ignore' }); extraArgs: ['--disable-blink-features=AutomationControlled', '--start-maximized'],
});
const wsUrl = await waitForChromeDebugPort(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
const cdp = await CdpConnection.connect(wsUrl, 30_000); const cdp = await CdpConnection.connect(wsUrl, 30_000);
return { cdp, chrome }; return { cdp, chrome };
@@ -234,11 +133,14 @@ export async function launchChrome(url: string, profileDir?: string): Promise<{
export async function getPageSession(cdp: CdpConnection, urlPattern: string): Promise<ChromeSession> { export async function getPageSession(cdp: CdpConnection, urlPattern: string): Promise<ChromeSession> {
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes(urlPattern)); const pageTarget = targets.targetInfos.find((target) => target.type === 'page' && target.url.includes(urlPattern));
if (!pageTarget) throw new Error(`Page not found: ${urlPattern}`); if (!pageTarget) throw new Error(`Page not found: ${urlPattern}`);
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', {
targetId: pageTarget.targetId,
flatten: true,
});
await cdp.send('Page.enable', {}, { sessionId }); await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId }); await cdp.send('Runtime.enable', {}, { sessionId });
@@ -247,11 +149,20 @@ export async function getPageSession(cdp: CdpConnection, urlPattern: string): Pr
return { cdp, sessionId, targetId: pageTarget.targetId }; return { cdp, sessionId, targetId: pageTarget.targetId };
} }
export async function waitForNewTab(cdp: CdpConnection, initialIds: Set<string>, urlPattern: string, timeoutMs = 30_000): Promise<string> { export async function waitForNewTab(
cdp: CdpConnection,
initialIds: Set<string>,
urlPattern: string,
timeoutMs = 30_000,
): Promise<string> {
const start = Date.now(); const start = Date.now();
while (Date.now() - start < timeoutMs) { while (Date.now() - start < timeoutMs) {
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
const newTab = targets.targetInfos.find(t => t.type === 'page' && !initialIds.has(t.targetId) && t.url.includes(urlPattern)); const newTab = targets.targetInfos.find((target) => (
target.type === 'page' &&
!initialIds.has(target.targetId) &&
target.url.includes(urlPattern)
));
if (newTab) return newTab.targetId; if (newTab) return newTab.targetId;
await sleep(500); await sleep(500);
} }
@@ -259,7 +170,7 @@ export async function waitForNewTab(cdp: CdpConnection, initialIds: Set<string>,
} }
export async function clickElement(session: ChromeSession, selector: string): Promise<void> { export async function clickElement(session: ChromeSession, selector: string): Promise<void> {
const posResult = await session.cdp.send<{ result: { value: string } }>('Runtime.evaluate', { const position = await session.cdp.send<{ result: { value: string } }>('Runtime.evaluate', {
expression: ` expression: `
(function() { (function() {
const el = document.querySelector('${selector}'); const el = document.querySelector('${selector}');
@@ -272,23 +183,46 @@ export async function clickElement(session: ChromeSession, selector: string): Pr
returnByValue: true, returnByValue: true,
}, { sessionId: session.sessionId }); }, { sessionId: session.sessionId });
if (posResult.result.value === 'null') throw new Error(`Element not found: ${selector}`); if (position.result.value === 'null') throw new Error(`Element not found: ${selector}`);
const pos = JSON.parse(posResult.result.value); const pos = JSON.parse(position.result.value);
await session.cdp.send('Input.dispatchMouseEvent', { type: 'mousePressed', x: pos.x, y: pos.y, button: 'left', clickCount: 1 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchMouseEvent', {
type: 'mousePressed',
x: pos.x,
y: pos.y,
button: 'left',
clickCount: 1,
}, { sessionId: session.sessionId });
await sleep(50); await sleep(50);
await session.cdp.send('Input.dispatchMouseEvent', { type: 'mouseReleased', x: pos.x, y: pos.y, button: 'left', clickCount: 1 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchMouseEvent', {
type: 'mouseReleased',
x: pos.x,
y: pos.y,
button: 'left',
clickCount: 1,
}, { sessionId: session.sessionId });
} }
export async function typeText(session: ChromeSession, text: string): Promise<void> { export async function typeText(session: ChromeSession, text: string): Promise<void> {
const lines = text.split('\n'); const lines = text.split('\n');
for (let i = 0; i < lines.length; i++) { for (let index = 0; index < lines.length; index += 1) {
if (lines[i].length > 0) { const line = lines[index];
await session.cdp.send('Input.insertText', { text: lines[i] }, { sessionId: session.sessionId }); if (line.length > 0) {
await session.cdp.send('Input.insertText', { text: line }, { sessionId: session.sessionId });
} }
if (i < lines.length - 1) { if (index < lines.length - 1) {
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyDown', key: 'Enter', code: 'Enter', windowsVirtualKeyCode: 13 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchKeyEvent', {
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyUp', key: 'Enter', code: 'Enter', windowsVirtualKeyCode: 13 }, { sessionId: session.sessionId }); type: 'keyDown',
key: 'Enter',
code: 'Enter',
windowsVirtualKeyCode: 13,
}, { sessionId: session.sessionId });
await session.cdp.send('Input.dispatchKeyEvent', {
type: 'keyUp',
key: 'Enter',
code: 'Enter',
windowsVirtualKeyCode: 13,
}, { sessionId: session.sessionId });
} }
await sleep(30); await sleep(30);
} }
@@ -296,8 +230,20 @@ export async function typeText(session: ChromeSession, text: string): Promise<vo
export async function pasteFromClipboard(session: ChromeSession): Promise<void> { export async function pasteFromClipboard(session: ChromeSession): Promise<void> {
const modifiers = process.platform === 'darwin' ? 4 : 2; const modifiers = process.platform === 'darwin' ? 4 : 2;
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyDown', key: 'v', code: 'KeyV', modifiers, windowsVirtualKeyCode: 86 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchKeyEvent', {
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyUp', key: 'v', code: 'KeyV', modifiers, windowsVirtualKeyCode: 86 }, { sessionId: session.sessionId }); type: 'keyDown',
key: 'v',
code: 'KeyV',
modifiers,
windowsVirtualKeyCode: 86,
}, { sessionId: session.sessionId });
await session.cdp.send('Input.dispatchKeyEvent', {
type: 'keyUp',
key: 'v',
code: 'KeyV',
modifiers,
windowsVirtualKeyCode: 86,
}, { sessionId: session.sessionId });
} }
export async function evaluate<T = unknown>(session: ChromeSession, expression: string): Promise<T> { export async function evaluate<T = unknown>(session: ChromeSession, expression: string): Promise<T> {
@@ -37,6 +37,10 @@ function parseExtendYaml(yaml: string): Partial<ExtendConfig> {
export function loadExtendConfig(): Partial<ExtendConfig> { export function loadExtendConfig(): Partial<ExtendConfig> {
const paths = [ const paths = [
path.join(process.cwd(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"), path.join(process.cwd(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"),
path.join(
process.env.XDG_CONFIG_HOME || path.join(homedir(), ".config"),
"baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"
),
path.join(homedir(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"), path.join(homedir(), ".baoyu-skills", "baoyu-markdown-to-html", "EXTEND.md"),
]; ];
for (const p of paths) { for (const p of paths) {
@@ -0,0 +1,8 @@
{
"name": "baoyu-post-to-wechat-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
@@ -1,4 +1,4 @@
import { execSync, spawnSync } from 'node:child_process'; import { spawnSync } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
@@ -10,30 +10,51 @@ function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms)); return new Promise((resolve) => setTimeout(resolve, ms));
} }
function ab(cmd: string, json = false): string { function quoteForLog(arg: string): string {
const fullCmd = `agent-browser --session ${SESSION} ${cmd}${json ? ' --json' : ''}`; return /[\s"'\\]/.test(arg) ? JSON.stringify(arg) : arg;
console.log(`[ab] ${fullCmd}`);
try {
const result = execSync(fullCmd, { encoding: 'utf8', stdio: ['pipe', 'pipe', 'pipe'] });
return result.trim();
} catch (e: unknown) {
const err = e as { stdout?: string; stderr?: string; message?: string };
console.error(`[ab] Error: ${err.stderr || err.message}`);
return err.stdout || '';
}
} }
function abRaw(args: string[]): { success: boolean; output: string } { function toSafeJsStringLiteral(value: string): string {
return JSON.stringify(value)
.replace(/\u2028/g, '\\u2028')
.replace(/\u2029/g, '\\u2029');
}
function runAgentBrowser(args: string[]): {
success: boolean;
output: string;
spawnError?: string;
} {
const result = spawnSync('agent-browser', ['--session', SESSION, ...args], { const result = spawnSync('agent-browser', ['--session', SESSION, ...args], {
encoding: 'utf8', encoding: 'utf8',
stdio: ['pipe', 'pipe', 'pipe'] stdio: ['pipe', 'pipe', 'pipe']
}); });
const spawnError = result.error?.message?.trim();
const output = result.stdout || result.stderr || '';
return { return {
success: result.status === 0, success: result.status === 0,
output: result.stdout || result.stderr || '' output: output || spawnError || '',
spawnError
}; };
} }
function ab(args: string[], json = false): string {
const fullArgs = json ? [...args, '--json'] : args;
console.log(`[ab] agent-browser --session ${SESSION} ${fullArgs.map(quoteForLog).join(' ')}`);
const result = runAgentBrowser(fullArgs);
if (result.spawnError) {
throw new Error(`agent-browser failed to start: ${result.spawnError}`);
}
if (!result.success) {
console.error(`[ab] Error: ${result.output.trim()}`);
}
return result.output.trim();
}
function abRaw(args: string[]): { success: boolean; output: string } {
return runAgentBrowser(args);
}
interface SnapshotElement { interface SnapshotElement {
ref: string; ref: string;
role: string; role: string;
@@ -99,17 +120,17 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
} }
console.log('[wechat] Opening WeChat Official Account...'); console.log('[wechat] Opening WeChat Official Account...');
ab(`open ${WECHAT_URL} --headed`); ab(['open', WECHAT_URL, '--headed']);
await sleep(5000); await sleep(5000);
console.log('[wechat] Checking login status...'); console.log('[wechat] Checking login status...');
let url = ab('get url'); let url = ab(['get', 'url']);
console.log(`[wechat] Current URL: ${url}`); console.log(`[wechat] Current URL: ${url}`);
const waitForLogin = async (timeoutMs = 120_000): Promise<boolean> => { const waitForLogin = async (timeoutMs = 120_000): Promise<boolean> => {
const start = Date.now(); const start = Date.now();
while (Date.now() - start < timeoutMs) { while (Date.now() - start < timeoutMs) {
url = ab('get url'); url = ab(['get', 'url']);
if (url.includes('/cgi-bin/home')) return true; if (url.includes('/cgi-bin/home')) return true;
console.log('[wechat] Waiting for login...'); console.log('[wechat] Waiting for login...');
await sleep(3000); await sleep(3000);
@@ -126,7 +147,7 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
await sleep(2000); await sleep(2000);
console.log('[wechat] Getting page snapshot...'); console.log('[wechat] Getting page snapshot...');
let snapshot = ab('snapshot'); let snapshot = ab(['snapshot']);
console.log(snapshot); console.log(snapshot);
console.log('[wechat] Looking for "图文" menu...'); console.log('[wechat] Looking for "图文" menu...');
@@ -134,16 +155,16 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
if (!tuWenRef) { if (!tuWenRef) {
console.log('[wechat] Using eval to find and click menu...'); console.log('[wechat] Using eval to find and click menu...');
ab(`eval "document.querySelectorAll('.new-creation__menu .new-creation__menu-item')[2].click()"`); ab(['eval', "document.querySelectorAll('.new-creation__menu .new-creation__menu-item')[2].click()"]);
} else { } else {
console.log(`[wechat] Clicking menu ref: ${tuWenRef}`); console.log(`[wechat] Clicking menu ref: ${tuWenRef}`);
ab(`click ${tuWenRef}`); ab(['click', tuWenRef]);
} }
await sleep(4000); await sleep(4000);
console.log('[wechat] Checking for new tab...'); console.log('[wechat] Checking for new tab...');
const tabsOutput = ab('tab'); const tabsOutput = ab(['tab']);
console.log(`[wechat] Tabs: ${tabsOutput}`); console.log(`[wechat] Tabs: ${tabsOutput}`);
const tabLines = tabsOutput.split('\n'); const tabLines = tabsOutput.split('\n');
@@ -153,14 +174,14 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
const tabMatch = tabsOutput.match(/\[(\d+)\].*(?:appmsg|edit)/i); const tabMatch = tabsOutput.match(/\[(\d+)\].*(?:appmsg|edit)/i);
if (tabMatch) { if (tabMatch) {
console.log(`[wechat] Switching to editor tab ${tabMatch[1]}...`); console.log(`[wechat] Switching to editor tab ${tabMatch[1]}...`);
ab(`tab ${tabMatch[1]}`); ab(['tab', tabMatch[1]]);
} else { } else {
const lastTabMatch = tabsOutput.match(/\[(\d+)\]/g); const lastTabMatch = tabsOutput.match(/\[(\d+)\]/g);
if (lastTabMatch && lastTabMatch.length > 1) { if (lastTabMatch && lastTabMatch.length > 1) {
const lastTab = lastTabMatch[lastTabMatch.length - 1].match(/\d+/)?.[0]; const lastTab = lastTabMatch[lastTabMatch.length - 1].match(/\d+/)?.[0];
if (lastTab) { if (lastTab) {
console.log(`[wechat] Switching to last tab ${lastTab}...`); console.log(`[wechat] Switching to last tab ${lastTab}...`);
ab(`tab ${lastTab}`); ab(['tab', lastTab]);
} }
} }
} }
@@ -168,38 +189,44 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
await sleep(3000); await sleep(3000);
url = ab('get url'); url = ab(['get', 'url']);
console.log(`[wechat] Editor URL: ${url}`); console.log(`[wechat] Editor URL: ${url}`);
console.log('[wechat] Getting editor snapshot...'); console.log('[wechat] Getting editor snapshot...');
snapshot = ab('snapshot'); snapshot = ab(['snapshot']);
console.log(snapshot.substring(0, 2000)); console.log(snapshot.substring(0, 2000));
console.log('[wechat] Uploading images...'); console.log('[wechat] Uploading images...');
const fileInputSelector = '.js_upload_btn_container input[type=file]'; const fileInputSelector = '.js_upload_btn_container input[type=file]';
const fileInputSelectorJs = toSafeJsStringLiteral(fileInputSelector);
ab(`eval "document.querySelector('${fileInputSelector}').style.display = 'block'"`); ab(['eval', `{
const input = document.querySelector(${fileInputSelectorJs});
if (input) input.style.display = 'block';
}`]);
await sleep(500); await sleep(500);
const uploadResult = abRaw(['upload', `"${fileInputSelector}"`, ...absoluteImages]); const uploadResult = abRaw(['upload', fileInputSelector, ...absoluteImages]);
console.log(`[wechat] Upload result: ${uploadResult.output}`); console.log(`[wechat] Upload result: ${uploadResult.output}`);
if (!uploadResult.success) { if (!uploadResult.success) {
console.log('[wechat] Using alternative upload method...'); console.log('[wechat] Using alternative upload method...');
for (const img of absoluteImages) { for (const img of absoluteImages) {
console.log(`[wechat] Uploading: ${img}`); console.log(`[wechat] Uploading: ${img}`);
ab(`eval " const imgUrlJs = toSafeJsStringLiteral(`file://${img}`);
const input = document.querySelector('${fileInputSelector}'); const imgFileNameJs = toSafeJsStringLiteral(path.basename(img));
ab(['eval', `
const input = document.querySelector(${fileInputSelectorJs});
if (input) { if (input) {
const dt = new DataTransfer(); const dt = new DataTransfer();
fetch('file://${img}').then(r => r.blob()).then(b => { fetch(${imgUrlJs}).then(r => r.blob()).then(b => {
const file = new File([b], '${path.basename(img)}', { type: 'image/png' }); const file = new File([b], ${imgFileNameJs}, { type: 'image/png' });
dt.items.add(file); dt.items.add(file);
input.files = dt.files; input.files = dt.files;
input.dispatchEvent(new Event('change', { bubbles: true })); input.dispatchEvent(new Event('change', { bubbles: true }));
}); });
} }
"`); `]);
await sleep(2000); await sleep(2000);
} }
} }
@@ -208,13 +235,14 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
await sleep(10000); await sleep(10000);
console.log('[wechat] Filling title...'); console.log('[wechat] Filling title...');
snapshot = ab('snapshot -i'); snapshot = ab(['snapshot', '-i']);
const titleRef = findElementByText(snapshot, 'title') || findElementByText(snapshot, '标题'); const titleRef = findElementByText(snapshot, 'title') || findElementByText(snapshot, '标题');
if (titleRef) { if (titleRef) {
ab(`fill ${titleRef} "${title.replace(/"/g, '\\"')}"`); ab(['fill', titleRef, title]);
} else { } else {
ab(`eval "const t = document.querySelector('#title'); if(t) { t.value = '${title.replace(/'/g, "\\'")}'; t.dispatchEvent(new Event('input', {bubbles: true})); }"`); const titleJs = toSafeJsStringLiteral(title);
ab(['eval', `const t = document.querySelector('#title'); if(t) { t.value = ${titleJs}; t.dispatchEvent(new Event('input', {bubbles: true})); }`]);
} }
await sleep(500); await sleep(500);
@@ -222,9 +250,9 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
const editorRef = findElementByText(snapshot, 'js_pmEditorArea') || findElementByText(snapshot, 'textbox'); const editorRef = findElementByText(snapshot, 'js_pmEditorArea') || findElementByText(snapshot, 'textbox');
if (editorRef) { if (editorRef) {
ab(`click ${editorRef}`); ab(['click', editorRef]);
} else { } else {
ab(`eval "document.querySelector('.js_pmEditorArea')?.click()"`); ab(['eval', "document.querySelector('.js_pmEditorArea')?.click()"]);
} }
await sleep(500); await sleep(500);
@@ -233,11 +261,11 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
for (let i = 0; i < lines.length; i++) { for (let i = 0; i < lines.length; i++) {
const line = lines[i]; const line = lines[i];
if (line.length > 0) { if (line.length > 0) {
const escapedLine = line.replace(/"/g, '\\"').replace(/'/g, "\\'"); const lineJs = toSafeJsStringLiteral(line);
ab(`eval "document.execCommand('insertText', false, '${escapedLine}')"`); ab(['eval', `document.execCommand('insertText', false, ${lineJs})`]);
} }
if (i < lines.length - 1) { if (i < lines.length - 1) {
ab('press Enter'); ab(['press', 'Enter']);
} }
await sleep(100); await sleep(100);
} }
@@ -249,9 +277,9 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
console.log('[wechat] Saving as draft...'); console.log('[wechat] Saving as draft...');
const submitRef = findElementByText(snapshot, 'js_submit') || findElementByText(snapshot, '保存'); const submitRef = findElementByText(snapshot, 'js_submit') || findElementByText(snapshot, '保存');
if (submitRef) { if (submitRef) {
ab(`click ${submitRef}`); ab(['click', submitRef]);
} else { } else {
ab(`eval "document.querySelector('#js_submit')?.click()"`); ab(['eval', "document.querySelector('#js_submit')?.click()"]);
} }
await sleep(3000); await sleep(3000);
console.log('[wechat] Draft saved!'); console.log('[wechat] Draft saved!');
@@ -261,7 +289,7 @@ async function postToWeChat(options: WeChatOptions): Promise<void> {
if (!keepOpen) { if (!keepOpen) {
console.log('[wechat] Closing browser...'); console.log('[wechat] Closing browser...');
ab('close'); ab(['close']);
} else { } else {
console.log('[wechat] Done. Browser window left open.'); console.log('[wechat] Done. Browser window left open.');
} }
@@ -1,13 +1,8 @@
import fs from "node:fs"; import fs from "node:fs";
import path from "node:path"; import path from "node:path";
import os from "node:os";
import { spawnSync } from "node:child_process"; import { spawnSync } from "node:child_process";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { loadWechatExtendConfig, resolveAccount, loadCredentials } from "./wechat-extend-config.ts";
interface WechatConfig {
appId: string;
appSecret: string;
}
interface AccessTokenResponse { interface AccessTokenResponse {
access_token?: string; access_token?: string;
@@ -38,53 +33,14 @@ interface ArticleOptions {
thumbMediaId: string; thumbMediaId: string;
articleType: ArticleType; articleType: ArticleType;
imageMediaIds?: string[]; imageMediaIds?: string[];
needOpenComment?: number;
onlyFansCanComment?: number;
} }
const TOKEN_URL = "https://api.weixin.qq.com/cgi-bin/token"; const TOKEN_URL = "https://api.weixin.qq.com/cgi-bin/token";
const UPLOAD_URL = "https://api.weixin.qq.com/cgi-bin/material/add_material"; const UPLOAD_URL = "https://api.weixin.qq.com/cgi-bin/material/add_material";
const DRAFT_URL = "https://api.weixin.qq.com/cgi-bin/draft/add"; const DRAFT_URL = "https://api.weixin.qq.com/cgi-bin/draft/add";
function loadEnvFile(envPath: string): Record<string, string> {
const env: Record<string, string> = {};
if (!fs.existsSync(envPath)) return env;
const content = fs.readFileSync(envPath, "utf-8");
for (const line of content.split("\n")) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const eqIdx = trimmed.indexOf("=");
if (eqIdx > 0) {
const key = trimmed.slice(0, eqIdx).trim();
let value = trimmed.slice(eqIdx + 1).trim();
if ((value.startsWith('"') && value.endsWith('"')) ||
(value.startsWith("'") && value.endsWith("'"))) {
value = value.slice(1, -1);
}
env[key] = value;
}
}
return env;
}
function loadConfig(): WechatConfig {
const cwdEnvPath = path.join(process.cwd(), ".baoyu-skills", ".env");
const homeEnvPath = path.join(os.homedir(), ".baoyu-skills", ".env");
const cwdEnv = loadEnvFile(cwdEnvPath);
const homeEnv = loadEnvFile(homeEnvPath);
const appId = process.env.WECHAT_APP_ID || cwdEnv.WECHAT_APP_ID || homeEnv.WECHAT_APP_ID;
const appSecret = process.env.WECHAT_APP_SECRET || cwdEnv.WECHAT_APP_SECRET || homeEnv.WECHAT_APP_SECRET;
if (!appId || !appSecret) {
throw new Error(
"Missing WECHAT_APP_ID or WECHAT_APP_SECRET.\n" +
"Set via environment variables or in .baoyu-skills/.env file."
);
}
return { appId, appSecret };
}
async function fetchAccessToken(appId: string, appSecret: string): Promise<string> { async function fetchAccessToken(appId: string, appSecret: string): Promise<string> {
const url = `${TOKEN_URL}?grant_type=client_credential&appid=${appId}&secret=${appSecret}`; const url = `${TOKEN_URL}?grant_type=client_credential&appid=${appId}&secret=${appSecret}`;
@@ -241,6 +197,9 @@ async function publishToDraft(
let article: Record<string, unknown>; let article: Record<string, unknown>;
const noc = options.needOpenComment ?? 1;
const ofcc = options.onlyFansCanComment ?? 0;
if (options.articleType === "newspic") { if (options.articleType === "newspic") {
if (!options.imageMediaIds || options.imageMediaIds.length === 0) { if (!options.imageMediaIds || options.imageMediaIds.length === 0) {
throw new Error("newspic requires at least one image"); throw new Error("newspic requires at least one image");
@@ -249,8 +208,8 @@ async function publishToDraft(
article_type: "newspic", article_type: "newspic",
title: options.title, title: options.title,
content: options.content, content: options.content,
need_open_comment: 1, need_open_comment: noc,
only_fans_can_comment: 0, only_fans_can_comment: ofcc,
image_info: { image_info: {
image_list: options.imageMediaIds.map(id => ({ image_media_id: id })), image_list: options.imageMediaIds.map(id => ({ image_media_id: id })),
}, },
@@ -262,8 +221,8 @@ async function publishToDraft(
title: options.title, title: options.title,
content: options.content, content: options.content,
thumb_media_id: options.thumbMediaId, thumb_media_id: options.thumbMediaId,
need_open_comment: 1, need_open_comment: noc,
only_fans_can_comment: 0, only_fans_can_comment: ofcc,
}; };
if (options.author) article.author = options.author; if (options.author) article.author = options.author;
if (options.digest) article.digest = options.digest; if (options.digest) article.digest = options.digest;
@@ -368,6 +327,7 @@ Options:
--theme <name> Theme name for markdown (default, grace, simple, modern). Default: default --theme <name> Theme name for markdown (default, grace, simple, modern). Default: default
--color <name|hex> Primary color (blue, green, vermilion, etc. or hex) --color <name|hex> Primary color (blue, green, vermilion, etc. or hex)
--cover <path> Cover image path (local or URL) --cover <path> Cover image path (local or URL)
--account <alias> Select account by alias (for multi-account setups)
--no-cite Disable bottom citations for ordinary external links in markdown mode --no-cite Disable bottom citations for ordinary external links in markdown mode
--dry-run Parse and render only, don't publish --dry-run Parse and render only, don't publish
--help Show this help --help Show this help
@@ -412,6 +372,7 @@ interface CliArgs {
theme: string; theme: string;
color?: string; color?: string;
cover?: string; cover?: string;
account?: string;
citeStatus: boolean; citeStatus: boolean;
dryRun: boolean; dryRun: boolean;
} }
@@ -449,6 +410,8 @@ function parseArgs(argv: string[]): CliArgs {
args.color = argv[++i]; args.color = argv[++i];
} else if (arg === "--cover" && argv[i + 1]) { } else if (arg === "--cover" && argv[i + 1]) {
args.cover = argv[++i]; args.cover = argv[++i];
} else if (arg === "--account" && argv[i + 1]) {
args.account = argv[++i];
} else if (arg === "--cite") { } else if (arg === "--cite") {
args.citeStatus = true; args.citeStatus = true;
} else if (arg === "--no-cite") { } else if (arg === "--no-cite") {
@@ -550,6 +513,12 @@ async function main(): Promise<void> {
if (digest) console.error(`[wechat-api] Digest: ${digest.slice(0, 50)}...`); if (digest) console.error(`[wechat-api] Digest: ${digest.slice(0, 50)}...`);
console.error(`[wechat-api] Type: ${args.articleType}`); console.error(`[wechat-api] Type: ${args.articleType}`);
const extConfig = loadWechatExtendConfig();
const resolved = resolveAccount(extConfig, args.account);
if (resolved.name) console.error(`[wechat-api] Account: ${resolved.name} (${resolved.alias})`);
if (!author && resolved.default_author) author = resolved.default_author;
if (args.dryRun) { if (args.dryRun) {
console.log(JSON.stringify({ console.log(JSON.stringify({
articleType: args.articleType, articleType: args.articleType,
@@ -558,13 +527,14 @@ async function main(): Promise<void> {
digest: digest || undefined, digest: digest || undefined,
htmlPath, htmlPath,
contentLength: htmlContent.length, contentLength: htmlContent.length,
account: resolved.alias || undefined,
}, null, 2)); }, null, 2));
return; return;
} }
const config = loadConfig(); const creds = loadCredentials(resolved);
console.error("[wechat-api] Fetching access token..."); console.error("[wechat-api] Fetching access token...");
const accessToken = await fetchAccessToken(config.appId, config.appSecret); const accessToken = await fetchAccessToken(creds.appId, creds.appSecret);
console.error("[wechat-api] Uploading images..."); console.error("[wechat-api] Uploading images...");
const { html: processedHtml, firstMediaId, allMediaIds } = await uploadImagesInHtml( const { html: processedHtml, firstMediaId, allMediaIds } = await uploadImagesInHtml(
@@ -617,6 +587,8 @@ async function main(): Promise<void> {
thumbMediaId, thumbMediaId,
articleType: args.articleType, articleType: args.articleType,
imageMediaIds: args.articleType === "newspic" ? allMediaIds : undefined, imageMediaIds: args.articleType === "newspic" ? allMediaIds : undefined,
needOpenComment: resolved.need_open_comment,
onlyFansCanComment: resolved.only_fans_can_comment,
}, accessToken); }, accessToken);
console.log(JSON.stringify({ console.log(JSON.stringify({
@@ -3,7 +3,8 @@ import path from 'node:path';
import { spawnSync } from 'node:child_process'; import { spawnSync } from 'node:child_process';
import process from 'node:process'; import process from 'node:process';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
import { launchChrome, tryConnectExisting, findExistingChromeDebugPort, getPageSession, waitForNewTab, clickElement, typeText, evaluate, sleep, type ChromeSession, type CdpConnection } from './cdp.ts'; import { launchChrome, tryConnectExisting, findExistingChromeDebugPort, getPageSession, waitForNewTab, clickElement, typeText, evaluate, sleep, getAccountProfileDir, type ChromeSession, type CdpConnection } from './cdp.ts';
import { loadWechatExtendConfig, resolveAccount } from './wechat-extend-config.ts';
const WECHAT_URL = 'https://mp.weixin.qq.com/'; const WECHAT_URL = 'https://mp.weixin.qq.com/';
@@ -690,6 +691,7 @@ Options:
--image <path> Content image, can repeat (only with --content) --image <path> Content image, can repeat (only with --content)
--submit Save as draft --submit Save as draft
--profile <dir> Chrome profile directory --profile <dir> Chrome profile directory
--account <alias> Select account by alias (for multi-account setups)
--cdp-port <port> Connect to existing Chrome debug port instead of launching new instance --cdp-port <port> Connect to existing Chrome debug port instead of launching new instance
Examples: Examples:
@@ -725,6 +727,7 @@ async function main(): Promise<void> {
let submit = false; let submit = false;
let profileDir: string | undefined; let profileDir: string | undefined;
let cdpPort: number | undefined; let cdpPort: number | undefined;
let accountAlias: string | undefined;
for (let i = 0; i < args.length; i++) { for (let i = 0; i < args.length; i++) {
const arg = args[i]!; const arg = args[i]!;
@@ -741,9 +744,20 @@ async function main(): Promise<void> {
else if (arg === '--image' && args[i + 1]) images.push(args[++i]!); else if (arg === '--image' && args[i + 1]) images.push(args[++i]!);
else if (arg === '--submit') submit = true; else if (arg === '--submit') submit = true;
else if (arg === '--profile' && args[i + 1]) profileDir = args[++i]; else if (arg === '--profile' && args[i + 1]) profileDir = args[++i];
else if (arg === '--account' && args[i + 1]) accountAlias = args[++i];
else if (arg === '--cdp-port' && args[i + 1]) cdpPort = parseInt(args[++i]!, 10); else if (arg === '--cdp-port' && args[i + 1]) cdpPort = parseInt(args[++i]!, 10);
} }
const extConfig = loadWechatExtendConfig();
const resolved = resolveAccount(extConfig, accountAlias);
if (resolved.name) console.log(`[wechat] Account: ${resolved.name} (${resolved.alias})`);
if (!author && resolved.default_author) author = resolved.default_author;
if (!profileDir && resolved.alias) {
profileDir = resolved.chrome_profile_path || getAccountProfileDir(resolved.alias);
}
if (!markdownFile && !htmlFile && !title) { console.error('Error: --title is required (or use --markdown/--html)'); process.exit(1); } if (!markdownFile && !htmlFile && !title) { console.error('Error: --title is required (or use --markdown/--html)'); process.exit(1); }
if (!markdownFile && !htmlFile && !content) { console.error('Error: --content, --html, or --markdown is required'); process.exit(1); } if (!markdownFile && !htmlFile && !content) { console.error('Error: --content, --html, or --markdown is required'); process.exit(1); }
@@ -1,11 +1,18 @@
import { execSync, spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir, readdir } from 'node:fs/promises'; import { readdir } from 'node:fs/promises';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
import {
CdpConnection,
findChromeExecutable,
getDefaultProfileDir,
getAccountProfileDir,
launchChrome,
sleep,
} from './cdp.ts';
import { loadWechatExtendConfig, resolveAccount } from './wechat-extend-config.ts';
const WECHAT_URL = 'https://mp.weixin.qq.com/'; const WECHAT_URL = 'https://mp.weixin.qq.com/';
interface MarkdownMeta { interface MarkdownMeta {
@@ -104,195 +111,6 @@ async function loadImagesFromDir(dir: string): Promise<string[]> {
return images; return images;
} }
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.WECHAT_BROWSER_DEBUG_PORT || '', 10);
if (fixed > 0) return fixed;
return new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
function findChromeExecutable(): string | undefined {
const override = process.env.WECHAT_BROWSER_CHROME_PATH?.trim();
if (override && fs.existsSync(override)) return override;
const candidates: string[] = [];
switch (process.platform) {
case 'darwin':
candidates.push(
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
);
break;
case 'win32':
candidates.push(
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
);
break;
default:
candidates.push(
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
);
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return undefined;
}
let _wslHome: string | null | undefined;
function getWslWindowsHome(): string | null {
if (_wslHome !== undefined) return _wslHome;
if (!process.env.WSL_DISTRO_NAME) { _wslHome = null; return null; }
try {
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', { encoding: 'utf-8', timeout: 5000 }).trim().replace(/\r/g, '');
_wslHome = execSync(`wslpath -u "${raw}"`, { encoding: 'utf-8', timeout: 5000 }).trim() || null;
} catch { _wslHome = null; }
return _wslHome;
}
function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.WECHAT_BROWSER_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
const wslHome = getWslWindowsHome();
if (wslHome) return path.join(wslHome, '.local', 'share', 'baoyu-skills', 'chrome-profile');
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
async function fetchJson<T = unknown>(url: string): Promise<T> {
const res = await fetch(url, { redirect: 'follow' });
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
}
async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error('Missing webSocketDebuggerUrl');
} catch (error) {
lastError = error;
}
await sleep(200);
}
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, { resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) handlers.forEach((h) => h(msg.params));
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) this.eventHandlers.set(method, new Set());
this.eventHandlers.get(method)!.add(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? 15_000;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0 ? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs) : null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
}
interface WeChatBrowserOptions { interface WeChatBrowserOptions {
title?: string; title?: string;
content?: string; content?: string;
@@ -348,29 +166,18 @@ export async function postToWeChat(options: WeChatBrowserOptions): Promise<void>
if (!fs.existsSync(img)) throw new Error(`Image not found: ${img}`); if (!fs.existsSync(img)) throw new Error(`Image not found: ${img}`);
} }
const chromePath = options.chromePath ?? findChromeExecutable(); const chromePath = findChromeExecutable(options.chromePath);
if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.');
await mkdir(profileDir, { recursive: true });
const port = await getFreePort();
console.log(`[wechat-browser] Launching Chrome (profile: ${profileDir})`); console.log(`[wechat-browser] Launching Chrome (profile: ${profileDir})`);
const chrome = spawn(chromePath, [ const launched = await launchChrome(WECHAT_URL, profileDir, chromePath);
`--remote-debugging-port=${port}`, const chrome = launched.chrome;
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
WECHAT_URL,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000); cdp = launched.cdp;
cdp = await CdpConnection.connect(wsUrl, 30_000);
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('mp.weixin.qq.com')); let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('mp.weixin.qq.com'));
@@ -859,6 +666,7 @@ Options:
--image <path> Add image (can be repeated) --image <path> Add image (can be repeated)
--submit Save as draft (default: preview only) --submit Save as draft (default: preview only)
--profile <dir> Chrome profile directory --profile <dir> Chrome profile directory
--account <alias> Select account by alias (for multi-account setups)
--help Show this help --help Show this help
Examples: Examples:
@@ -880,6 +688,7 @@ async function main(): Promise<void> {
let content: string | undefined; let content: string | undefined;
let markdownFile: string | undefined; let markdownFile: string | undefined;
let imagesDir: string | undefined; let imagesDir: string | undefined;
let accountAlias: string | undefined;
for (let i = 0; i < args.length; i++) { for (let i = 0; i < args.length; i++) {
const arg = args[i]!; const arg = args[i]!;
@@ -897,9 +706,19 @@ async function main(): Promise<void> {
submit = true; submit = true;
} else if (arg === '--profile' && args[i + 1]) { } else if (arg === '--profile' && args[i + 1]) {
profileDir = args[++i]; profileDir = args[++i];
} else if (arg === '--account' && args[i + 1]) {
accountAlias = args[++i];
} }
} }
const extConfig = loadWechatExtendConfig();
const resolved = resolveAccount(extConfig, accountAlias);
if (resolved.name) console.log(`[wechat-browser] Account: ${resolved.name} (${resolved.alias})`);
if (!profileDir && resolved.alias) {
profileDir = resolved.chrome_profile_path || getAccountProfileDir(resolved.alias);
}
if (!markdownFile && !title) { if (!markdownFile && !title) {
console.error('Error: --title or --markdown is required'); console.error('Error: --title or --markdown is required');
process.exit(1); process.exit(1);
@@ -0,0 +1,245 @@
import fs from "node:fs";
import path from "node:path";
import os from "node:os";
export interface WechatAccount {
name: string;
alias: string;
default?: boolean;
default_publish_method?: string;
default_author?: string;
need_open_comment?: number;
only_fans_can_comment?: number;
app_id?: string;
app_secret?: string;
chrome_profile_path?: string;
}
export interface WechatExtendConfig {
default_theme?: string;
default_color?: string;
default_publish_method?: string;
default_author?: string;
need_open_comment?: number;
only_fans_can_comment?: number;
chrome_profile_path?: string;
accounts?: WechatAccount[];
}
export interface ResolvedAccount {
name?: string;
alias?: string;
default_publish_method?: string;
default_author?: string;
need_open_comment: number;
only_fans_can_comment: number;
app_id?: string;
app_secret?: string;
chrome_profile_path?: string;
}
function stripQuotes(s: string): string {
return s.replace(/^['"]|['"]$/g, "");
}
function toBool01(v: string): number {
return v === "1" || v === "true" ? 1 : 0;
}
function parseWechatExtend(content: string): WechatExtendConfig {
const config: WechatExtendConfig = {};
const lines = content.split("\n");
let inAccounts = false;
let current: Record<string, string> | null = null;
const rawAccounts: Record<string, string>[] = [];
for (const raw of lines) {
const trimmed = raw.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
if (trimmed === "accounts:") {
inAccounts = true;
continue;
}
if (inAccounts) {
const listMatch = raw.match(/^\s+-\s+(.+)$/);
if (listMatch) {
if (current) rawAccounts.push(current);
current = {};
const kv = listMatch[1]!;
const ci = kv.indexOf(":");
if (ci > 0) {
current[kv.slice(0, ci).trim()] = stripQuotes(kv.slice(ci + 1).trim());
}
continue;
}
if (current && /^\s{2,}/.test(raw) && !trimmed.startsWith("-")) {
const ci = trimmed.indexOf(":");
if (ci > 0) {
current[trimmed.slice(0, ci).trim()] = stripQuotes(trimmed.slice(ci + 1).trim());
}
continue;
}
if (!/^\s/.test(raw)) {
if (current) rawAccounts.push(current);
current = null;
inAccounts = false;
} else {
continue;
}
}
const ci = trimmed.indexOf(":");
if (ci < 0) continue;
const key = trimmed.slice(0, ci).trim();
const val = stripQuotes(trimmed.slice(ci + 1).trim());
if (val === "null" || val === "") continue;
switch (key) {
case "default_theme": config.default_theme = val; break;
case "default_color": config.default_color = val; break;
case "default_publish_method": config.default_publish_method = val; break;
case "default_author": config.default_author = val; break;
case "need_open_comment": config.need_open_comment = toBool01(val); break;
case "only_fans_can_comment": config.only_fans_can_comment = toBool01(val); break;
case "chrome_profile_path": config.chrome_profile_path = val; break;
}
}
if (current) rawAccounts.push(current);
if (rawAccounts.length > 0) {
config.accounts = rawAccounts.map(a => ({
name: a.name || "",
alias: a.alias || "",
default: a.default === "true" || a.default === "1",
default_publish_method: a.default_publish_method || undefined,
default_author: a.default_author || undefined,
need_open_comment: a.need_open_comment ? toBool01(a.need_open_comment) : undefined,
only_fans_can_comment: a.only_fans_can_comment ? toBool01(a.only_fans_can_comment) : undefined,
app_id: a.app_id || undefined,
app_secret: a.app_secret || undefined,
chrome_profile_path: a.chrome_profile_path || undefined,
}));
}
return config;
}
export function loadWechatExtendConfig(): WechatExtendConfig {
const paths = [
path.join(process.cwd(), ".baoyu-skills", "baoyu-post-to-wechat", "EXTEND.md"),
path.join(
process.env.XDG_CONFIG_HOME || path.join(os.homedir(), ".config"),
"baoyu-skills", "baoyu-post-to-wechat", "EXTEND.md"
),
path.join(os.homedir(), ".baoyu-skills", "baoyu-post-to-wechat", "EXTEND.md"),
];
for (const p of paths) {
try {
const content = fs.readFileSync(p, "utf-8");
return parseWechatExtend(content);
} catch {
continue;
}
}
return {};
}
function selectAccount(config: WechatExtendConfig, alias?: string): WechatAccount | undefined {
if (!config.accounts || config.accounts.length === 0) return undefined;
if (alias) return config.accounts.find(a => a.alias === alias);
if (config.accounts.length === 1) return config.accounts[0];
return config.accounts.find(a => a.default);
}
export function resolveAccount(config: WechatExtendConfig, alias?: string): ResolvedAccount {
const acct = selectAccount(config, alias);
return {
name: acct?.name,
alias: acct?.alias,
default_publish_method: acct?.default_publish_method ?? config.default_publish_method,
default_author: acct?.default_author ?? config.default_author,
need_open_comment: acct?.need_open_comment ?? config.need_open_comment ?? 1,
only_fans_can_comment: acct?.only_fans_can_comment ?? config.only_fans_can_comment ?? 0,
app_id: acct?.app_id,
app_secret: acct?.app_secret,
chrome_profile_path: acct?.chrome_profile_path ?? config.chrome_profile_path,
};
}
function loadEnvFile(envPath: string): Record<string, string> {
const env: Record<string, string> = {};
if (!fs.existsSync(envPath)) return env;
const content = fs.readFileSync(envPath, "utf-8");
for (const line of content.split("\n")) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const eqIdx = trimmed.indexOf("=");
if (eqIdx > 0) {
const key = trimmed.slice(0, eqIdx).trim();
let value = trimmed.slice(eqIdx + 1).trim();
if ((value.startsWith('"') && value.endsWith('"')) ||
(value.startsWith("'") && value.endsWith("'"))) {
value = value.slice(1, -1);
}
env[key] = value;
}
}
return env;
}
function aliasToEnvKey(alias: string): string {
return alias.toUpperCase().replace(/-/g, "_");
}
export function loadCredentials(account?: ResolvedAccount): { appId: string; appSecret: string } {
if (account?.app_id && account?.app_secret) {
return { appId: account.app_id, appSecret: account.app_secret };
}
const cwdEnvPath = path.join(process.cwd(), ".baoyu-skills", ".env");
const homeEnvPath = path.join(os.homedir(), ".baoyu-skills", ".env");
const cwdEnv = loadEnvFile(cwdEnvPath);
const homeEnv = loadEnvFile(homeEnvPath);
const prefix = account?.alias ? `WECHAT_${aliasToEnvKey(account.alias)}_` : "";
let appId = "";
let appSecret = "";
if (prefix) {
appId = process.env[`${prefix}APP_ID`]
|| cwdEnv[`${prefix}APP_ID`]
|| homeEnv[`${prefix}APP_ID`]
|| "";
appSecret = process.env[`${prefix}APP_SECRET`]
|| cwdEnv[`${prefix}APP_SECRET`]
|| homeEnv[`${prefix}APP_SECRET`]
|| "";
}
if (!appId) {
appId = process.env.WECHAT_APP_ID || cwdEnv.WECHAT_APP_ID || homeEnv.WECHAT_APP_ID || "";
}
if (!appSecret) {
appSecret = process.env.WECHAT_APP_SECRET || cwdEnv.WECHAT_APP_SECRET || homeEnv.WECHAT_APP_SECRET || "";
}
if (!appId || !appSecret) {
const hint = account?.alias ? ` (account: ${account.alias})` : "";
throw new Error(
`Missing WECHAT_APP_ID or WECHAT_APP_SECRET${hint}.\n` +
"Set via EXTEND.md account config, environment variables, or .baoyu-skills/.env file."
);
}
return { appId, appSecret };
}
export function listAccounts(config: WechatExtendConfig): string[] {
return (config.accounts || []).map(a => a.alias);
}
+3
View File
@@ -40,12 +40,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-post-to-weibo/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-post-to-weibo/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-post-to-weibo/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-post-to-weibo/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-post-to-weibo/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-post-to-weibo/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-post-to-weibo/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-post-to-weibo/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-weibo/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-weibo/EXTEND.md") { "user" }
``` ```
@@ -0,0 +1,31 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-post-to-weibo-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2",
"highlight.js": "^11.11.1",
"marked": "^15.0.6",
},
},
},
"packages": {
"argparse": ["argparse@1.0.10", "", { "dependencies": { "sprintf-js": "~1.0.2" } }, "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg=="],
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
"esprima": ["esprima@4.0.1", "", { "bin": { "esparse": "./bin/esparse.js", "esvalidate": "./bin/esvalidate.js" } }, "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A=="],
"front-matter": ["front-matter@4.0.2", "", { "dependencies": { "js-yaml": "^3.13.1" } }, "sha512-I8ZuJ/qG92NWX8i5x1Y8qyj3vizhXS31OxjKDu3LKP+7/qBgfIKValiZIEwoVoJKUHlhWtYrktkxV1XsX+pPlg=="],
"highlight.js": ["highlight.js@11.11.1", "", {}, "sha512-Xwwo44whKBVCYoliBQwaPvtd/2tYFkRQtXDWj1nackaV2JPXx3L0+Jvd8/qCJ2p+ML0/XVkJ2q+Mr+UVdpJK5w=="],
"js-yaml": ["js-yaml@3.14.2", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg=="],
"marked": ["marked@15.0.12", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-8dD6FusOQSrpv9Z1rdNMdlSgQOIP880DHqnohobOmYLElGEqAL/JvxvuxZO16r4HtjTlfPRDC1hbvxC9dPN2nA=="],
"sprintf-js": ["sprintf-js@1.0.3", "", {}, "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g=="],
}
}
@@ -446,7 +446,9 @@ Options:
} }
} }
if (import.meta.main ?? (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename ?? ''))) {
await main().catch((err) => { await main().catch((err) => {
console.error(`Error: ${err instanceof Error ? err.message : String(err)}`); console.error(`Error: ${err instanceof Error ? err.message : String(err)}`);
process.exit(1); process.exit(1);
}); });
}
@@ -0,0 +1,11 @@
{
"name": "baoyu-post-to-weibo-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2",
"highlight.js": "^11.11.1",
"marked": "^15.0.6"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
@@ -1,9 +1,7 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir, writeFile } from 'node:fs/promises'; import { mkdir, writeFile } from 'node:fs/promises';
import os from 'node:os'; import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process';
import { import {
CdpConnection, CdpConnection,
copyHtmlToClipboard, copyHtmlToClipboard,
@@ -11,7 +9,7 @@ import {
findChromeExecutable, findChromeExecutable,
findExistingChromeDebugPort, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, launchChrome,
pasteFromClipboard, pasteFromClipboard,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
@@ -74,36 +72,17 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
// Try reusing an existing Chrome instance with the same profile // Try reusing an existing Chrome instance with the same profile
const existingPort = findExistingChromeDebugPort(profileDir); const existingPort = await findExistingChromeDebugPort(profileDir);
let port: number; let port: number;
let launched = false;
if (existingPort) { if (existingPort) {
console.log(`[weibo-article] Found existing Chrome on port ${existingPort}, reusing...`); console.log(`[weibo-article] Found existing Chrome on port ${existingPort}, reusing...`);
port = existingPort; port = existingPort;
} else { } else {
const chromePath = options.chromePath ?? findChromeExecutable(); const chromePath = findChromeExecutable(options.chromePath);
if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
port = await getFreePort(); port = await launchChrome(WEIBO_ARTICLE_URL, profileDir, chromePath);
console.log(`[weibo-article] Launching Chrome...`);
const chromeArgs = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
WEIBO_ARTICLE_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
launched = true;
} }
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
@@ -1,4 +1,3 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import path from 'node:path'; import path from 'node:path';
@@ -8,8 +7,8 @@ import {
findChromeExecutable, findChromeExecutable,
findExistingChromeDebugPort, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort,
killChromeByProfile, killChromeByProfile,
launchChrome as launchWeiboChrome,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
} from './weibo-utils.js'; } from './weibo-utils.js';
@@ -37,32 +36,11 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const chromePath = options.chromePath ?? findChromeExecutable(); const chromePath = findChromeExecutable(options.chromePath);
if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
const launchChrome = async (): Promise<number> => {
const port = await getFreePort();
console.log(`[weibo-post] Launching Chrome (profile: ${profileDir})`);
const chromeArgs = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
WEIBO_HOME_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
return port;
};
let port: number; let port: number;
const existingPort = findExistingChromeDebugPort(profileDir); const existingPort = await findExistingChromeDebugPort(profileDir);
if (existingPort) { if (existingPort) {
console.log(`[weibo-post] Found existing Chrome on port ${existingPort}, checking health...`); console.log(`[weibo-post] Found existing Chrome on port ${existingPort}, checking health...`);
@@ -77,10 +55,10 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
console.log('[weibo-post] Existing Chrome unresponsive, restarting...'); console.log('[weibo-post] Existing Chrome unresponsive, restarting...');
killChromeByProfile(profileDir); killChromeByProfile(profileDir);
await sleep(2000); await sleep(2000);
port = await launchChrome(); port = await launchWeiboChrome(WEIBO_HOME_URL, profileDir, chromePath);
} }
} else { } else {
port = await launchChrome(); port = await launchWeiboChrome(WEIBO_HOME_URL, profileDir, chromePath);
} }
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
+72 -163
View File
@@ -1,12 +1,23 @@
import { spawnSync } from 'node:child_process'; import { execSync, spawnSync } from 'node:child_process';
import fs from 'node:fs';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
export const CHROME_CANDIDATES = { import {
CdpConnection,
findChromeExecutable as findChromeExecutableBase,
findExistingChromeDebugPort as findExistingChromeDebugPortBase,
getFreePort as getFreePortBase,
launchChrome as launchChromeBase,
resolveSharedChromeProfileDir,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
export { CdpConnection, sleep, waitForChromeDebugPort };
export const CHROME_CANDIDATES: PlatformCandidates = {
darwin: [ darwin: [
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary', '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
@@ -23,183 +34,81 @@ export const CHROME_CANDIDATES = {
], ],
}; };
export function findChromeExecutable(): string | undefined { let wslHome: string | null | undefined;
const override = process.env.WEIBO_BROWSER_CHROME_PATH?.trim(); function getWslWindowsHome(): string | null {
if (override && fs.existsSync(override)) return override; if (wslHome !== undefined) return wslHome;
if (!process.env.WSL_DISTRO_NAME) {
const candidates = process.platform === 'darwin' wslHome = null;
? CHROME_CANDIDATES.darwin
: process.platform === 'win32'
? CHROME_CANDIDATES.win32
: CHROME_CANDIDATES.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function findExistingChromeDebugPort(profileDir: string): number | null {
try {
const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout.split('\n');
for (const line of lines) {
if (!line.includes('--remote-debugging-port=') || !line.includes(profileDir)) continue;
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
if (portMatch) return Number(portMatch[1]);
}
} catch {}
return null; return null;
} }
try {
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', {
encoding: 'utf-8',
timeout: 5_000,
}).trim().replace(/\r/g, '');
wslHome = execSync(`wslpath -u "${raw}"`, {
encoding: 'utf-8',
timeout: 5_000,
}).trim() || null;
} catch {
wslHome = null;
}
return wslHome;
}
export function findChromeExecutable(chromePathOverride?: string): string | undefined {
if (chromePathOverride?.trim()) return chromePathOverride.trim();
return findChromeExecutableBase({
candidates: CHROME_CANDIDATES,
envNames: ['WEIBO_BROWSER_CHROME_PATH'],
});
}
export async function findExistingChromeDebugPort(profileDir: string): Promise<number | null> {
return await findExistingChromeDebugPortBase({ profileDir });
}
export function killChromeByProfile(profileDir: string): void { export function killChromeByProfile(profileDir: string): void {
try { try {
const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5000 }); const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return; if (result.status !== 0 || !result.stdout) return;
for (const line of result.stdout.split('\n')) { for (const line of result.stdout.split('\n')) {
if (!line.includes(profileDir) || !line.includes('--remote-debugging-port=')) continue; if (!line.includes(profileDir) || !line.includes('--remote-debugging-port=')) continue;
const pidMatch = line.trim().split(/\s+/)[1]; const pid = line.trim().split(/\s+/)[1];
if (pidMatch) { if (pid) {
try { process.kill(Number(pidMatch), 'SIGTERM'); } catch {} try {
process.kill(Number(pid), 'SIGTERM');
} catch {}
} }
} }
} catch {} } catch {}
} }
export function getDefaultProfileDir(): string { export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.WEIBO_BROWSER_PROFILE_DIR?.trim(); return resolveSharedChromeProfileDir({
if (override) return path.resolve(override); envNames: ['BAOYU_CHROME_PROFILE_DIR', 'WEIBO_BROWSER_PROFILE_DIR'],
const base = process.platform === 'darwin' wslWindowsHome: getWslWindowsHome(),
? path.join(os.homedir(), 'Library', 'Application Support') });
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
} }
export async function getFreePort(): Promise<number> { export async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.WEIBO_BROWSER_DEBUG_PORT || '', 10); return await getFreePortBase('WEIBO_BROWSER_DEBUG_PORT');
if (fixed > 0) return fixed;
return new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
} }
const port = address.port;
server.close((err) => { export async function launchChrome(url: string, profileDir: string, chromePathOverride?: string): Promise<number> {
if (err) reject(err); const chromePath = findChromeExecutable(chromePathOverride);
else resolve(port); if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
const port = await getFreePort();
console.log(`[weibo-cdp] Launching Chrome (profile: ${profileDir})`);
await launchChromeBase({
chromePath,
profileDir,
port,
url,
extraArgs: ['--disable-blink-features=AutomationControlled', '--start-maximized'],
}); });
}); return port;
});
}
async function fetchJson<T = unknown>(url: string): Promise<T> {
const res = await fetch(url, { redirect: 'follow' });
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
}
export async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error('Missing webSocketDebuggerUrl');
} catch (error) {
lastError = error;
}
await sleep(200);
}
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, options?: { defaultTimeoutMs?: number }) {
this.ws = ws;
this.defaultTimeoutMs = options?.defaultTimeoutMs ?? 15_000;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number, options?: { defaultTimeoutMs?: number }): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws, options);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
} }
export function getScriptDir(): string { export function getScriptDir(): string {
+3
View File
@@ -44,12 +44,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-post-to-x/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-post-to-x/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-post-to-x/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-post-to-x/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-post-to-x/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-post-to-x/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-post-to-x/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-post-to-x/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-x/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-x/EXTEND.md") { "user" }
``` ```
+3
View File
@@ -4,6 +4,7 @@
"": { "": {
"name": "baoyu-post-to-x-scripts", "name": "baoyu-post-to-x-scripts",
"dependencies": { "dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2", "front-matter": "^4.0.2",
"highlight.js": "^11.11.1", "highlight.js": "^11.11.1",
"marked": "^15.0.6", "marked": "^15.0.6",
@@ -27,6 +28,8 @@
"bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="], "bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="],
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
"character-entities": ["character-entities@2.0.2", "", {}, "sha512-shx7oQ0Awen/BRIdkjkvz54PnEEI/EjwXDSIZp86/KKdbafHh1Df/RYGBhn4hbe2+uKC9FnT5UCEdyPz3ai9hQ=="], "character-entities": ["character-entities@2.0.2", "", {}, "sha512-shx7oQ0Awen/BRIdkjkvz54PnEEI/EjwXDSIZp86/KKdbafHh1Df/RYGBhn4hbe2+uKC9FnT5UCEdyPz3ai9hQ=="],
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="], "debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
@@ -3,6 +3,7 @@
"private": true, "private": true,
"type": "module", "type": "module",
"dependencies": { "dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2", "front-matter": "^4.0.2",
"highlight.js": "^11.11.1", "highlight.js": "^11.11.1",
"marked": "^15.0.6", "marked": "^15.0.6",
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
+19 -58
View File
@@ -1,9 +1,7 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir, writeFile } from 'node:fs/promises'; import { mkdir, writeFile } from 'node:fs/promises';
import os from 'node:os'; import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process';
import { parseMarkdown } from './md-to-html.js'; import { parseMarkdown } from './md-to-html.js';
import { import {
@@ -11,9 +9,10 @@ import {
CdpConnection, CdpConnection,
copyHtmlToClipboard, copyHtmlToClipboard,
copyImageToClipboard, copyImageToClipboard,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, launchChrome,
openPageSession,
pasteFromClipboard, pasteFromClipboard,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
@@ -61,25 +60,6 @@ interface ArticleOptions {
chromePath?: string; chromePath?: string;
} }
async function findExistingDebugPort(profileDir: string): Promise<number | null> {
const portFile = path.join(profileDir, 'DevToolsActivePort');
if (!fs.existsSync(portFile)) return null;
try {
const content = fs.readFileSync(portFile, 'utf-8').trim();
if (!content) return null;
const [portLine] = content.split(/\r?\n/);
const port = Number(portLine);
if (!Number.isFinite(port) || port <= 0) return null;
// Verify the port is actually active.
await waitForChromeDebugPort(port, 1500, { includeLastError: true });
return port;
} catch {
return null;
}
}
export async function publishArticle(options: ArticleOptions): Promise<void> { export async function publishArticle(options: ArticleOptions): Promise<void> {
const { markdownPath, submit = false, profileDir = getDefaultProfileDir() } = options; const { markdownPath, submit = false, profileDir = getDefaultProfileDir() } = options;
@@ -98,32 +78,17 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
await writeFile(htmlPath, parsed.html, 'utf-8'); await writeFile(htmlPath, parsed.html, 'utf-8');
console.log(`[x-article] HTML saved to: ${htmlPath}`); console.log(`[x-article] HTML saved to: ${htmlPath}`);
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_BASIC);
if (!chromePath) throw new Error('Chrome not found');
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const existingPort = await findExistingDebugPort(profileDir); const existingPort = await findExistingChromeDebugPort(profileDir);
const port = existingPort ?? await getFreePort(); const reusing = existingPort !== null;
let port = existingPort ?? 0;
if (existingPort) { if (reusing) {
console.log(`[x-article] Reusing existing Chrome instance on port ${port}`); console.log(`[x-article] Reusing existing Chrome instance on port ${port}`);
} else { } else {
console.log(`[x-article] Launching Chrome...`); console.log(`[x-article] Launching Chrome...`);
const chromeArgs = [ const launched = await launchChrome(X_ARTICLES_URL, profileDir, CHROME_CANDIDATES_BASIC, options.chromePath);
`--remote-debugging-port=${port}`, port = launched.port;
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
X_ARTICLES_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
} }
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
@@ -132,20 +97,16 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 60_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 60_000 });
// Get page target const page = await openPageSession({
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); cdp,
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.startsWith(X_ARTICLES_URL)); reusing,
url: X_ARTICLES_URL,
if (!pageTarget) { matchTarget: (target) => target.type === 'page' && target.url.startsWith(X_ARTICLES_URL),
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: X_ARTICLES_URL }); enablePage: true,
pageTarget = { targetId, url: X_ARTICLES_URL, type: 'page' }; enableRuntime: true,
} enableDom: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = page;
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('DOM.enable', {}, { sessionId });
console.log('[x-article] Waiting for articles page...'); console.log('[x-article] Waiting for articles page...');
await sleep(1000); await sleep(1000);
+25 -29
View File
@@ -1,4 +1,3 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import process from 'node:process'; import process from 'node:process';
@@ -6,9 +5,10 @@ import {
CHROME_CANDIDATES_FULL, CHROME_CANDIDATES_FULL,
CdpConnection, CdpConnection,
copyImageToClipboard, copyImageToClipboard,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, launchChrome,
openPageSession,
pasteFromClipboard, pasteFromClipboard,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
@@ -28,23 +28,20 @@ interface XBrowserOptions {
export async function postToX(options: XBrowserOptions): Promise<void> { export async function postToX(options: XBrowserOptions): Promise<void> {
const { text, images = [], submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options; const { text, images = [], submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_FULL);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const port = await getFreePort(); const existingPort = await findExistingChromeDebugPort(profileDir);
console.log(`[x-browser] Launching Chrome (profile: ${profileDir})`); const reusing = existingPort !== null;
let port = existingPort ?? 0;
let chrome: Awaited<ReturnType<typeof launchChrome>>['chrome'] | null = null;
if (!reusing) {
const launched = await launchChrome(X_COMPOSE_URL, profileDir, CHROME_CANDIDATES_FULL, options.chromePath);
port = launched.port;
chrome = launched.chrome;
}
const chrome = spawn(chromePath, [ if (reusing) console.log(`[x-browser] Reusing existing Chrome on port ${port}`);
`--remote-debugging-port=${port}`, else console.log(`[x-browser] Launching Chrome (profile: ${profileDir})`);
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
X_COMPOSE_URL,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
@@ -52,18 +49,15 @@ export async function postToX(options: XBrowserOptions): Promise<void> {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 });
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const page = await openPageSession({
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('x.com')); cdp,
reusing,
if (!pageTarget) { url: X_COMPOSE_URL,
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: X_COMPOSE_URL }); matchTarget: (target) => target.type === 'page' && target.url.includes('x.com'),
pageTarget = { targetId, url: X_COMPOSE_URL, type: 'page' }; enablePage: true,
} enableRuntime: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = page;
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId }); await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId });
console.log('[x-browser] Waiting for X editor...'); console.log('[x-browser] Waiting for X editor...');
@@ -193,9 +187,11 @@ export async function postToX(options: XBrowserOptions): Promise<void> {
if (cdp) { if (cdp) {
cdp.close(); cdp.close();
} }
if (chrome) {
chrome.unref(); chrome.unref();
} }
} }
}
function printUsage(): never { function printUsage(): never {
console.log(`Post to X (Twitter) using real Chrome browser console.log(`Post to X (Twitter) using real Chrome browser
+31 -34
View File
@@ -1,12 +1,13 @@
import { spawn } from 'node:child_process';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import process from 'node:process'; import process from 'node:process';
import { import {
CHROME_CANDIDATES_FULL, CHROME_CANDIDATES_FULL,
CdpConnection, CdpConnection,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, killChrome,
launchChrome,
openPageSession,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
} from './x-utils.js'; } from './x-utils.js';
@@ -31,43 +32,39 @@ interface QuoteOptions {
export async function quotePost(options: QuoteOptions): Promise<void> { export async function quotePost(options: QuoteOptions): Promise<void> {
const { tweetUrl, comment, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options; const { tweetUrl, comment, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_FULL);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const port = await getFreePort(); const existingPort = await findExistingChromeDebugPort(profileDir);
console.log(`[x-quote] Launching Chrome (profile: ${profileDir})`); const reusing = existingPort !== null;
let port = existingPort ?? 0;
console.log(`[x-quote] Opening tweet: ${tweetUrl}`); console.log(`[x-quote] Opening tweet: ${tweetUrl}`);
let chrome: Awaited<ReturnType<typeof launchChrome>>['chrome'] | null = null;
if (!reusing) {
const launched = await launchChrome(tweetUrl, profileDir, CHROME_CANDIDATES_FULL, options.chromePath);
port = launched.port;
chrome = launched.chrome;
}
const chrome = spawn(chromePath, [ if (reusing) console.log(`[x-quote] Reusing existing Chrome on port ${port}`);
`--remote-debugging-port=${port}`, else console.log(`[x-quote] Launching Chrome (profile: ${profileDir})`);
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
tweetUrl,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 });
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const page = await openPageSession({
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('x.com')); cdp,
reusing,
if (!pageTarget) { url: tweetUrl,
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: tweetUrl }); matchTarget: (target) => target.type === 'page' && target.url.includes('x.com'),
pageTarget = { targetId, url: tweetUrl, type: 'page' }; enablePage: true,
} enableRuntime: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = page;
targetId = page.targetId;
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
console.log('[x-quote] Waiting for tweet to load...'); console.log('[x-quote] Waiting for tweet to load...');
await sleep(3000); await sleep(3000);
@@ -176,14 +173,14 @@ export async function quotePost(options: QuoteOptions): Promise<void> {
} }
} finally { } finally {
if (cdp) { if (cdp) {
if (reusing && targetId) {
try { await cdp.send('Target.closeTarget', { targetId }, { timeoutMs: 5_000 }); } catch {}
} else if (!reusing) {
try { await cdp.send('Browser.close', {}, { timeoutMs: 5_000 }); } catch {} try { await cdp.send('Browser.close', {}, { timeoutMs: 5_000 }); } catch {}
}
cdp.close(); cdp.close();
} }
if (chrome) killChrome(chrome);
setTimeout(() => {
if (!chrome.killed) try { chrome.kill('SIGKILL'); } catch {}
}, 2_000).unref?.();
try { chrome.kill('SIGTERM'); } catch {}
} }
} }
+43 -163
View File
@@ -1,16 +1,24 @@
import { execSync, spawnSync } from 'node:child_process'; import { execSync, spawnSync } from 'node:child_process';
import fs from 'node:fs';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
export type PlatformCandidates = { import {
darwin?: string[]; CdpConnection,
win32?: string[]; findChromeExecutable as findChromeExecutableBase,
default: string[]; findExistingChromeDebugPort as findExistingChromeDebugPortBase,
}; getFreePort as getFreePortBase,
killChrome,
launchChrome as launchChromeBase,
openPageSession,
resolveSharedChromeProfileDir,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
export { CdpConnection, killChrome, openPageSession, sleep, waitForChromeDebugPort };
export type { PlatformCandidates } from 'baoyu-chrome-cdp';
export const CHROME_CANDIDATES_BASIC: PlatformCandidates = { export const CHROME_CANDIDATES_BASIC: PlatformCandidates = {
darwin: [ darwin: [
@@ -53,20 +61,11 @@ export const CHROME_CANDIDATES_FULL: PlatformCandidates = {
], ],
}; };
function getCandidatesForPlatform(candidates: PlatformCandidates): string[] {
if (process.platform === 'darwin' && candidates.darwin?.length) return candidates.darwin;
if (process.platform === 'win32' && candidates.win32?.length) return candidates.win32;
return candidates.default;
}
export function findChromeExecutable(candidates: PlatformCandidates): string | undefined { export function findChromeExecutable(candidates: PlatformCandidates): string | undefined {
const override = process.env.X_BROWSER_CHROME_PATH?.trim(); return findChromeExecutableBase({
if (override && fs.existsSync(override)) return override; candidates,
envNames: ['X_BROWSER_CHROME_PATH'],
for (const candidate of getCandidatesForPlatform(candidates)) { });
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
} }
let _wslHome: string | null | undefined; let _wslHome: string | null | undefined;
@@ -81,158 +80,39 @@ function getWslWindowsHome(): string | null {
} }
export function getDefaultProfileDir(): string { export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.X_BROWSER_PROFILE_DIR?.trim(); return resolveSharedChromeProfileDir({
if (override) return path.resolve(override); envNames: ['BAOYU_CHROME_PROFILE_DIR', 'X_BROWSER_PROFILE_DIR'],
const wslHome = getWslWindowsHome(); wslWindowsHome: getWslWindowsHome(),
if (wslHome) return path.join(wslHome, '.local', 'share', 'baoyu-skills', 'chrome-profile'); });
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
} }
export async function getFreePort(): Promise<number> { export async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.X_BROWSER_DEBUG_PORT || '', 10); return await getFreePortBase('X_BROWSER_DEBUG_PORT');
if (fixed > 0) return fixed;
return new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
} }
async function fetchJson<T = unknown>(url: string): Promise<T> { export async function findExistingChromeDebugPort(profileDir: string): Promise<number | null> {
const res = await fetch(url, { redirect: 'follow' }); return await findExistingChromeDebugPortBase({ profileDir });
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
} }
export async function waitForChromeDebugPort( export async function launchChrome(
port: number, url: string,
timeoutMs: number, profileDir: string,
options?: { includeLastError?: boolean }, candidates: PlatformCandidates,
): Promise<string> { chromePathOverride?: string,
const start = Date.now(); ): Promise<{ chrome: Awaited<ReturnType<typeof launchChromeBase>>; port: number }> {
let lastError: unknown = null; const chromePath = chromePathOverride?.trim() || findChromeExecutable(candidates);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
while (Date.now() - start < timeoutMs) { const port = await getFreePort();
try { const chrome = await launchChromeBase({
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`); chromePath,
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl; profileDir,
lastError = new Error('Missing webSocketDebuggerUrl'); port,
} catch (error) { url,
lastError = error; extraArgs: ['--start-maximized'],
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
throw new Error('Chrome debug port not ready');
}
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, options?: { defaultTimeoutMs?: number }) {
this.ws = ws;
this.defaultTimeoutMs = options?.defaultTimeoutMs ?? 15_000;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) handlers.forEach((h) => h(msg.params));
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
}); });
this.ws.addEventListener('close', () => { return { chrome, port };
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number, options?: { defaultTimeoutMs?: number }): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws, options);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) this.eventHandlers.set(method, new Set());
this.eventHandlers.get(method)!.add(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
} }
export function getScriptDir(): string { export function getScriptDir(): string {
+31 -37
View File
@@ -1,4 +1,3 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import path from 'node:path'; import path from 'node:path';
@@ -6,9 +5,11 @@ import process from 'node:process';
import { import {
CHROME_CANDIDATES_FULL, CHROME_CANDIDATES_FULL,
CdpConnection, CdpConnection,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, killChrome,
launchChrome,
openPageSession,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
} from './x-utils.js'; } from './x-utils.js';
@@ -27,9 +28,6 @@ interface XVideoOptions {
export async function postVideoToX(options: XVideoOptions): Promise<void> { export async function postVideoToX(options: XVideoOptions): Promise<void> {
const { text, videoPath, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options; const { text, videoPath, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_FULL);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
if (!fs.existsSync(videoPath)) throw new Error(`Video not found: ${videoPath}`); if (!fs.existsSync(videoPath)) throw new Error(`Video not found: ${videoPath}`);
const absVideoPath = path.resolve(videoPath); const absVideoPath = path.resolve(videoPath);
@@ -37,38 +35,37 @@ export async function postVideoToX(options: XVideoOptions): Promise<void> {
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const port = await getFreePort(); const existingPort = await findExistingChromeDebugPort(profileDir);
console.log(`[x-video] Launching Chrome (profile: ${profileDir})`); const reusing = existingPort !== null;
let port = existingPort ?? 0;
let chrome: Awaited<ReturnType<typeof launchChrome>>['chrome'] | null = null;
if (!reusing) {
const launched = await launchChrome(X_COMPOSE_URL, profileDir, CHROME_CANDIDATES_FULL, options.chromePath);
port = launched.port;
chrome = launched.chrome;
}
const chrome = spawn(chromePath, [ if (reusing) console.log(`[x-video] Reusing existing Chrome on port ${port}`);
`--remote-debugging-port=${port}`, else console.log(`[x-video] Launching Chrome (profile: ${profileDir})`);
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
X_COMPOSE_URL,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 30_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 30_000 });
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const page = await openPageSession({
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('x.com')); cdp,
reusing,
if (!pageTarget) { url: X_COMPOSE_URL,
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: X_COMPOSE_URL }); matchTarget: (target) => target.type === 'page' && target.url.includes('x.com'),
pageTarget = { targetId, url: X_COMPOSE_URL, type: 'page' }; enablePage: true,
} enableRuntime: true,
enableDom: true,
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); });
const { sessionId } = page;
await cdp.send('Page.enable', {}, { sessionId }); targetId = page.targetId;
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('DOM.enable', {}, { sessionId });
await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId }); await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId });
console.log('[x-video] Waiting for X editor...'); console.log('[x-video] Waiting for X editor...');
@@ -183,15 +180,12 @@ export async function postVideoToX(options: XVideoOptions): Promise<void> {
} }
} finally { } finally {
if (cdp) { if (cdp) {
if (reusing && submit && targetId) {
try { await cdp.send('Target.closeTarget', { targetId }, { timeoutMs: 5_000 }); } catch {}
}
cdp.close(); cdp.close();
} }
// Don't kill Chrome in preview mode, let user review if (chrome && submit) killChrome(chrome);
if (submit) {
setTimeout(() => {
if (!chrome.killed) try { chrome.kill('SIGKILL'); } catch {}
}, 2_000).unref?.();
try { chrome.kill('SIGTERM'); } catch {}
}
} }
} }
+3
View File
@@ -203,12 +203,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-slide-deck/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-slide-deck/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-slide-deck/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-slide-deck/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-slide-deck/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-slide-deck/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-slide-deck/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-slide-deck/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-slide-deck/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-slide-deck/EXTEND.md") { "user" }
``` ```
+23 -3
View File
@@ -30,12 +30,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-translate/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-translate/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-translate/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-translate/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-translate/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-translate/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-translate/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-translate/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-translate/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-translate/EXTEND.md") { "user" }
``` ```
@@ -185,7 +188,7 @@ Before translating chunks:
- Splits at markdown block boundaries to preserve structure - Splits at markdown block boundaries to preserve structure
- If a single block exceeds the threshold, falls back to line splitting, then word splitting - If a single block exceeds the threshold, falls back to line splitting, then word splitting
4. **Assemble translation prompt**: 4. **Assemble translation prompt**:
- Main agent reads `01-analysis.md` (if exists) and assembles shared context using Part 1 of [references/subagent-prompt-template.md](references/subagent-prompt-template.md) — inlining content background, merged glossary, and comprehension challenges - Main agent reads `01-analysis.md` (if exists) and assembles shared context using Part 1 of [references/subagent-prompt-template.md](references/subagent-prompt-template.md) — inlining the resolved style preset (from `--style` flag, EXTEND.md `style` setting, or default `storytelling`), content background, merged glossary, and comprehension challenges
- Save as `02-prompt.md` in the output directory (shared context only, no task instructions) - Save as `02-prompt.md` in the output directory (shared context only, no task instructions)
5. **Draft translation via subagents** (if Agent tool available): 5. **Draft translation via subagents** (if Agent tool available):
- Spawn one subagent **per chunk**, all in parallel (Part 2 of the template) - Spawn one subagent **per chunk**, all in parallel (Part 2 of the template)
@@ -209,9 +212,10 @@ Before translating chunks:
- **Natural flow**: Use idiomatic target language word order and sentence patterns; break or restructure sentences freely when the source structure doesn't work naturally in the target language - **Natural flow**: Use idiomatic target language word order and sentence patterns; break or restructure sentences freely when the source structure doesn't work naturally in the target language
- **Terminology**: Use standard translations; annotate with original term in parentheses on first occurrence - **Terminology**: Use standard translations; annotate with original term in parentheses on first occurrence
- **Preserve format**: Keep all markdown formatting (headings, bold, italic, images, links, code blocks) - **Preserve format**: Keep all markdown formatting (headings, bold, italic, images, links, code blocks)
- **Image-language awareness**: Preserve image references exactly during translation, but after the translation is complete, review referenced images and check whether their likely main text language still matches the translated article language
- **Frontmatter transformation**: If the source has YAML frontmatter, preserve it in the translation with these changes: (1) Rename metadata fields that describe the *source* article — `url``sourceUrl`, `title``sourceTitle`, `description``sourceDescription`, `author``sourceAuthor`, `date``sourceDate`, and any similar origin-metadata fields — by adding a `source` prefix (camelCase). (2) Translate the values of text fields (title, description, etc.) and add them as new top-level fields. (3) Keep other fields (tags, categories, custom fields) as-is, translating their values where appropriate - **Frontmatter transformation**: If the source has YAML frontmatter, preserve it in the translation with these changes: (1) Rename metadata fields that describe the *source* article — `url``sourceUrl`, `title``sourceTitle`, `description``sourceDescription`, `author``sourceAuthor`, `date``sourceDate`, and any similar origin-metadata fields — by adding a `source` prefix (camelCase). (2) Translate the values of text fields (title, description, etc.) and add them as new top-level fields. (3) Keep other fields (tags, categories, custom fields) as-is, translating their values where appropriate
- **Respect original**: Maintain original meaning and intent; do not add, remove, or editorialize — but sentence structure and imagery may be adapted freely to serve the meaning - **Respect original**: Maintain original meaning and intent; do not add, remove, or editorialize — but sentence structure and imagery may be adapted freely to serve the meaning
- **Translator's notes**: For terms, concepts, or cultural references that target readers may not understand — due to jargon, cultural gaps, or domain-specific knowledge — add a concise explanatory note in parentheses immediately after the term. The note should explain *what it means* in plain language, not just provide the English original. Format: `译文(English original,通俗解释)`. Calibrate annotation depth to the target audience: general readers need more notes than technical readers. Only add notes where genuinely needed; do not over-annotate obvious terms. - **Translator's notes**: For terms, concepts, or cultural references that target readers may not understand — due to jargon, cultural gaps, or domain-specific knowledge — add a concise explanatory note in parentheses immediately after the term. The note should explain *what it means* in plain language, not just provide the English original. Format: `译文(English original,通俗解释)`. Calibrate annotation depth to the target audience: general readers need more notes than technical readers. For short texts (< 5 sentences), further reduce annotations — only annotate non-common terms that the target audience is unlikely to know; skip terms that are widely recognized or self-explanatory in context. Only add notes where genuinely needed; do not over-annotate obvious terms.
#### Quick Mode #### Quick Mode
@@ -220,7 +224,7 @@ Translate directly → save to `translation.md`. No analysis file, but still app
#### Normal Mode #### Normal Mode
1. **Analyze**`01-analysis.md` (domain, tone, audience, terminology, reader comprehension challenges, figurative language & metaphor mapping) 1. **Analyze**`01-analysis.md` (domain, tone, audience, terminology, reader comprehension challenges, figurative language & metaphor mapping)
2. **Assemble prompt**`02-prompt.md` (translation instructions with inlined context) 2. **Assemble prompt**`02-prompt.md` (translation instructions with inlined style preset, content background, glossary, and comprehension challenges)
3. **Translate** (following `02-prompt.md`) → `translation.md` 3. **Translate** (following `02-prompt.md`) → `translation.md`
After completion, prompt user: "Translation saved. To further review and polish, reply **继续润色** or **refine**." After completion, prompt user: "Translation saved. To further review and polish, reply **继续润色** or **refine**."
@@ -247,6 +251,20 @@ Each step reads the previous step's file and builds on it.
Final translation is always at `translation.md` in the output directory. Final translation is always at `translation.md` in the output directory.
After the final translation is written, do a lightweight image-language pass:
1. Collect image references from the translated article
2. Identify likely text-heavy images such as covers, screenshots, diagrams, charts, frameworks, and infographics
3. If any image likely contains a main text language that does not match the translated article language, proactively remind the user
4. The reminder must be a list only. Do not automatically localize those images unless the user asks
Reminder format (use whatever image syntax the article already uses — standard markdown or wikilink):
```text
Possible image localization needed:
- ![example cover](attachments/example-cover.png): likely still contains source-language text while the article is now in target language
- ![example diagram](attachments/example-diagram.png): likely text-heavy framework graphic, check whether labels need translation
```
Display summary: Display summary:
``` ```
**Translation complete** ({mode} mode) **Translation complete** ({mode} mode)
@@ -258,6 +276,8 @@ Final: {output-dir}/translation.md
Glossary terms applied: {count} Glossary terms applied: {count}
``` ```
If mismatched image-language candidates were found, append a short note after the summary telling the user that some embedded images may still need image-text localization, followed by the candidate list.
## Extension Support ## Extension Support
Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options. Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.
@@ -121,7 +121,7 @@ Implicit assumptions: [unstated premises]
## Step 2: Assemble Translation Prompt ## Step 2: Assemble Translation Prompt
Main agent reads `01-analysis.md` and assembles a complete translation prompt using [references/subagent-prompt-template.md](subagent-prompt-template.md). Inline content background, merged glossary, and comprehension challenges into the prompt. Save to `02-prompt.md`. Main agent reads `01-analysis.md` and assembles a complete translation prompt using [references/subagent-prompt-template.md](subagent-prompt-template.md). Inline the resolved style preset (from `--style` flag, EXTEND.md `style` setting, or default `storytelling`), content background, merged glossary, and comprehension challenges into the prompt. Save to `02-prompt.md`.
This prompt is used by the subagent (chunked) or by the main agent itself (non-chunked). This prompt is used by the subagent (chunked) or by the main agent itself (non-chunked).
+33 -6
View File
@@ -1,7 +1,7 @@
--- ---
name: baoyu-url-to-markdown name: baoyu-url-to-markdown
description: Fetch any URL and convert to markdown using Chrome CDP. Saves the rendered HTML snapshot alongside the markdown, and automatically falls back to the pre-Defuddle HTML-to-Markdown pipeline when Defuddle fails. Supports two modes - auto-capture on page load, or wait for user signal (for pages requiring login). Use when user wants to save a webpage as markdown. description: Fetch any URL and convert to markdown using Chrome CDP. Saves the rendered HTML snapshot alongside the markdown, uses an upgraded Defuddle pipeline with better web-component handling and YouTube transcript extraction, and automatically falls back to the pre-Defuddle HTML-to-Markdown pipeline when needed. If local browser capture fails entirely, it can fall back to the hosted defuddle.md API. Supports two modes - auto-capture on page load, or wait for user signal (for pages requiring login). Use when user wants to save a webpage as markdown.
version: 1.56.1 version: 1.58.1
metadata: metadata:
openclaw: openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-url-to-markdown homepage: https://github.com/JimLiu/baoyu-skills#baoyu-url-to-markdown
@@ -29,7 +29,10 @@ Fetches any URL via Chrome CDP, saves the rendered HTML snapshot, and converts i
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| `scripts/main.ts` | CLI entry point for URL fetching | | `scripts/main.ts` | CLI entry point for URL fetching |
| `scripts/html-to-markdown.ts` | Defuddle-first conversion with automatic legacy fallback | | `scripts/html-to-markdown.ts` | Markdown conversion entry point and converter selection |
| `scripts/defuddle-converter.ts` | Defuddle-based conversion |
| `scripts/legacy-converter.ts` | Pre-Defuddle legacy extraction and markdown conversion |
| `scripts/markdown-conversion-shared.ts` | Shared metadata parsing and markdown document helpers |
## Preferences (EXTEND.md) ## Preferences (EXTEND.md)
@@ -38,12 +41,15 @@ Check EXTEND.md existence (priority order):
```bash ```bash
# macOS, Linux, WSL, Git Bash # macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/baoyu-url-to-markdown/EXTEND.md && echo "project" test -f .baoyu-skills/baoyu-url-to-markdown/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-url-to-markdown/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md" && echo "user" test -f "$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md" && echo "user"
``` ```
```powershell ```powershell
# PowerShell (Windows) # PowerShell (Windows)
if (Test-Path .baoyu-skills/baoyu-url-to-markdown/EXTEND.md) { "project" } if (Test-Path .baoyu-skills/baoyu-url-to-markdown/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/baoyu-url-to-markdown/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md") { "user" } if (Test-Path "$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md") { "user" }
``` ```
@@ -112,7 +118,10 @@ Full reference: [references/config/first-time-setup.md](references/config/first-
- Two capture modes: auto or wait-for-user - Two capture modes: auto or wait-for-user
- Save rendered HTML as a sibling `-captured.html` file - Save rendered HTML as a sibling `-captured.html` file
- Clean markdown output with metadata - Clean markdown output with metadata
- Defuddle-first markdown conversion with automatic fallback to the pre-Defuddle extractor from git history - Upgraded Defuddle-first markdown conversion with automatic fallback to the pre-Defuddle extractor from git history
- Materializes shadow DOM content before conversion so web-component pages survive serialization better
- YouTube pages can include transcript/caption text in the markdown when YouTube exposes a caption track
- If local browser capture fails completely, can fall back to `defuddle.md/<url>` and still save markdown
- Handles login-required pages via wait mode - Handles login-required pages via wait mode
- Download images and videos to local directories - Download images and videos to local directories
@@ -165,7 +174,10 @@ Each run saves two files side by side:
- Markdown: YAML front matter with `url`, `title`, `description`, `author`, `published`, optional `coverImage`, and `captured_at`, followed by converted markdown content - Markdown: YAML front matter with `url`, `title`, `description`, `author`, `published`, optional `coverImage`, and `captured_at`, followed by converted markdown content
- HTML snapshot: `*-captured.html`, containing the rendered page HTML captured from Chrome - HTML snapshot: `*-captured.html`, containing the rendered page HTML captured from Chrome
When Defuddle or page metadata provides a language hint, the markdown front matter also includes `language`.
The HTML snapshot is saved before any markdown media localization, so it stays a faithful capture of the page DOM used for conversion. The HTML snapshot is saved before any markdown media localization, so it stays a faithful capture of the page DOM used for conversion.
If the hosted `defuddle.md` API fallback is used, markdown is still saved, but there is no local `-captured.html` snapshot for that run.
## Output Directory ## Output Directory
@@ -190,13 +202,16 @@ When `--download-media` is enabled:
Conversion order: Conversion order:
1. Try Defuddle first 1. Try Defuddle first
2. If Defuddle throws, cannot load, returns obviously incomplete markdown, or captures lower-quality content than the legacy pipeline, automatically fall back to the pre-Defuddle extractor 2. For rich pages such as YouTube, prefer Defuddle's extractor-specific output (including transcripts when available) instead of replacing it with the legacy pipeline
3. The fallback path uses the older Readability/selector/Next.js-data based HTML-to-Markdown implementation recovered from git history 3. If Defuddle throws, cannot load, returns obviously incomplete markdown, or captures lower-quality content than the legacy pipeline, automatically fall back to the pre-Defuddle extractor
4. If the entire local browser capture flow fails before markdown can be produced, try the hosted `https://defuddle.md/<url>` API and save its markdown output directly
5. The legacy fallback path uses the older Readability/selector/Next.js-data based HTML-to-Markdown implementation recovered from git history
CLI output will show: CLI output will show:
- `Converter: defuddle` when Defuddle succeeds - `Converter: defuddle` when Defuddle succeeds
- `Converter: legacy:...` plus `Fallback used: ...` when fallback was needed - `Converter: legacy:...` plus `Fallback used: ...` when fallback was needed
- `Converter: defuddle-api` when local browser capture failed and the hosted API was used instead
## Media Download Workflow ## Media Download Workflow
@@ -229,6 +244,18 @@ Based on `download_media` setting in EXTEND.md:
**Troubleshooting**: Chrome not found → set `URL_CHROME_PATH`. Timeout → increase `--timeout`. Complex pages → try `--wait` mode. If markdown quality is poor, inspect the saved `-captured.html` and check whether the run logged a legacy fallback. **Troubleshooting**: Chrome not found → set `URL_CHROME_PATH`. Timeout → increase `--timeout`. Complex pages → try `--wait` mode. If markdown quality is poor, inspect the saved `-captured.html` and check whether the run logged a legacy fallback.
### YouTube Notes
- The upgraded Defuddle path uses async extractors, so YouTube pages can include transcript text directly in the markdown body.
- Transcript availability depends on YouTube exposing a caption track. Videos with captions disabled, restricted playback, or blocked regional access may still produce description-only output.
- If the page needs time to finish loading descriptions, chapters, or player metadata, prefer `--wait` and capture after the watch page is fully hydrated.
### Hosted API Fallback
- The hosted fallback endpoint is `https://defuddle.md/<url>`. In shell form: `curl https://defuddle.md/stephango.com`
- Use it only when the local Chrome/CDP capture path fails outright. The local path still has higher fidelity because it can save the captured HTML and handle authenticated pages.
- The hosted API already returns Markdown with YAML frontmatter, so save that response as-is and then apply the normal media-localization step if requested.
## Extension Support ## Extension Support
Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options. Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.
@@ -5,7 +5,8 @@
"name": "baoyu-url-to-markdown-scripts", "name": "baoyu-url-to-markdown-scripts",
"dependencies": { "dependencies": {
"@mozilla/readability": "^0.6.0", "@mozilla/readability": "^0.6.0",
"defuddle": "^0.10.0", "baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"defuddle": "^0.12.0",
"jsdom": "^24.1.3", "jsdom": "^24.1.3",
"linkedom": "^0.18.12", "linkedom": "^0.18.12",
"turndown": "^7.2.2", "turndown": "^7.2.2",
@@ -36,6 +37,8 @@
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="], "asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
"boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="], "boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="],
"call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="], "call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="],
@@ -58,7 +61,7 @@
"decimal.js": ["decimal.js@10.6.0", "", {}, "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg=="], "decimal.js": ["decimal.js@10.6.0", "", {}, "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg=="],
"defuddle": ["defuddle@0.10.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "peerDependencies": { "jsdom": "^24.0.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-a43juTtHv6Vs4+sxvahVLM5NxoyDsarO1Ag3UxLORI4Fo/nsNFwzDxuQBvosKVGTIRxCwN/mfnWAzNXmQfieqw=="], "defuddle": ["defuddle@0.12.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "peerDependencies": { "jsdom": "^24.0.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-Y/WgyGKBxwxFir+hWNth4nmWDDDb8BzQi3qASS2NWYPXsKU42Ku49/3M5yFYefnRef9prynnmasfnXjk99EWgA=="],
"delayed-stream": ["delayed-stream@1.0.0", "", {}, "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ=="], "delayed-stream": ["delayed-stream@1.0.0", "", {}, "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ=="],
+116 -227
View File
@@ -1,209 +1,84 @@
import { spawn, type ChildProcess } from "node:child_process"; import {
import fs from "node:fs"; CdpConnection,
import { mkdir } from "node:fs/promises"; findChromeExecutable as findChromeExecutableBase,
import net from "node:net"; findExistingChromeDebugPort,
import process from "node:process"; getFreePort,
killChrome,
launchChrome as launchChromeBase,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
import { resolveUrlToMarkdownChromeProfileDir } from "./paths.js"; import { resolveUrlToMarkdownChromeProfileDir } from './paths.js';
import { CDP_CONNECT_TIMEOUT_MS, NETWORK_IDLE_TIMEOUT_MS } from "./constants.js"; import { NETWORK_IDLE_TIMEOUT_MS } from './constants.js';
type CdpSendOptions = { sessionId?: string; timeoutMs?: number }; const CHROME_CANDIDATES_FULL: PlatformCandidates = {
darwin: [
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
],
win32: [
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
],
default: [
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
],
};
function sleep(ms: number): Promise<void> { export { CdpConnection, getFreePort, killChrome, sleep, waitForChromeDebugPort };
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function fetchWithTimeout(url: string, init: RequestInit & { timeoutMs?: number } = {}): Promise<Response> { export async function findExistingChromePort(): Promise<number | null> {
const { timeoutMs, ...rest } = init; return await findExistingChromeDebugPort({
if (!timeoutMs || timeoutMs <= 0) return fetch(url, rest); profileDir: resolveUrlToMarkdownChromeProfileDir(),
const ctl = new AbortController();
const t = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { ...rest, signal: ctl.signal });
} finally {
clearTimeout(t);
}
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, { resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string" ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.id) {
const p = this.pending.get(msg.id);
if (p) {
this.pending.delete(msg.id);
if (p.timer) clearTimeout(p.timer);
if (msg.error?.message) p.reject(new Error(msg.error.message));
else p.resolve(msg.result);
}
} else if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
for (const h of handlers) h(msg.params);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, p] of this.pending.entries()) {
this.pending.delete(id);
if (p.timer) clearTimeout(p.timer);
p.reject(new Error("CDP connection closed."));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const t = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => { clearTimeout(t); resolve(); });
ws.addEventListener("error", () => { clearTimeout(t); reject(new Error("CDP connection failed.")); });
});
return new CdpConnection(ws);
}
on(event: string, handler: (params: unknown) => void): void {
let handlers = this.eventHandlers.get(event);
if (!handlers) {
handlers = new Set();
this.eventHandlers.set(event, handlers);
}
handlers.add(handler);
}
off(event: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(event)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, opts?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const msg: Record<string, unknown> = { id, method };
if (params) msg.params = params;
if (opts?.sessionId) msg.sessionId = opts.sessionId;
const timeoutMs = opts?.timeoutMs ?? 15_000;
const out = await new Promise<unknown>((resolve, reject) => {
const t = timeoutMs > 0 ? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs) : null;
this.pending.set(id, { resolve, reject, timer: t });
this.ws.send(JSON.stringify(msg));
});
return out as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
}
export async function getFreePort(): Promise<number> {
return await new Promise((resolve, reject) => {
const srv = net.createServer();
srv.unref();
srv.on("error", reject);
srv.listen(0, "127.0.0.1", () => {
const addr = srv.address();
if (!addr || typeof addr === "string") {
srv.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = addr.port;
srv.close((err) => (err ? reject(err) : resolve(port)));
});
}); });
} }
export function findChromeExecutable(): string | null { export function findChromeExecutable(): string | null {
const override = process.env.URL_CHROME_PATH?.trim(); return findChromeExecutableBase({
if (override && fs.existsSync(override)) return override; candidates: CHROME_CANDIDATES_FULL,
envNames: ['URL_CHROME_PATH'],
const candidates: string[] = []; }) ?? null;
switch (process.platform) {
case "darwin":
candidates.push(
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
"/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge"
);
break;
case "win32":
candidates.push(
"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
"C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe",
"C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe"
);
break;
default:
candidates.push(
"/usr/bin/google-chrome",
"/usr/bin/google-chrome-stable",
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
"/snap/bin/chromium",
"/usr/bin/microsoft-edge"
);
break;
} }
for (const p of candidates) { export async function launchChrome(url: string, port: number, headless = false) {
if (fs.existsSync(p)) return p; const chromePath = findChromeExecutable();
} if (!chromePath) throw new Error('Chrome executable not found. Install Chrome or set URL_CHROME_PATH env.');
return null;
return await launchChromeBase({
chromePath,
profileDir: resolveUrlToMarkdownChromeProfileDir(),
port,
url,
headless,
extraArgs: ['--disable-popup-blocking'],
});
} }
export async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> { export async function waitForNetworkIdle(
const start = Date.now(); cdp: CdpConnection,
while (Date.now() - start < timeoutMs) { sessionId: string,
try { timeoutMs: number = NETWORK_IDLE_TIMEOUT_MS,
const res = await fetchWithTimeout(`http://127.0.0.1:${port}/json/version`, { timeoutMs: 5_000 }); ): Promise<void> {
if (!res.ok) throw new Error(`status=${res.status}`);
const j = (await res.json()) as { webSocketDebuggerUrl?: string };
if (j.webSocketDebuggerUrl) return j.webSocketDebuggerUrl;
} catch {}
await sleep(200);
}
throw new Error("Chrome debug port not ready");
}
export async function launchChrome(url: string, port: number, headless: boolean = false): Promise<ChildProcess> {
const chrome = findChromeExecutable();
if (!chrome) throw new Error("Chrome executable not found. Install Chrome or set URL_CHROME_PATH env.");
const profileDir = resolveUrlToMarkdownChromeProfileDir();
await mkdir(profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
"--no-first-run",
"--no-default-browser-check",
"--disable-popup-blocking",
];
if (headless) args.push("--headless=new");
args.push(url);
return spawn(chrome, args, { stdio: "ignore" });
}
export async function waitForNetworkIdle(cdp: CdpConnection, sessionId: string, timeoutMs: number = NETWORK_IDLE_TIMEOUT_MS): Promise<void> {
return new Promise((resolve) => { return new Promise((resolve) => {
let timer: ReturnType<typeof setTimeout> | null = null; let timer: ReturnType<typeof setTimeout> | null = null;
let pending = 0; let pending = 0;
const cleanup = () => { const cleanup = () => {
if (timer) clearTimeout(timer); if (timer) clearTimeout(timer);
cdp.off("Network.requestWillBeSent", onRequest); cdp.off('Network.requestWillBeSent', onRequest);
cdp.off("Network.loadingFinished", onFinish); cdp.off('Network.loadingFinished', onFinish);
cdp.off("Network.loadingFailed", onFinish); cdp.off('Network.loadingFailed', onFinish);
}; };
const done = () => { cleanup(); resolve(); }; const done = () => { cleanup(); resolve(); };
const resetTimer = () => { const resetTimer = () => {
@@ -212,79 +87,93 @@ export async function waitForNetworkIdle(cdp: CdpConnection, sessionId: string,
}; };
const onRequest = () => { pending++; resetTimer(); }; const onRequest = () => { pending++; resetTimer(); };
const onFinish = () => { pending = Math.max(0, pending - 1); if (pending <= 2) resetTimer(); }; const onFinish = () => { pending = Math.max(0, pending - 1); if (pending <= 2) resetTimer(); };
cdp.on("Network.requestWillBeSent", onRequest); cdp.on('Network.requestWillBeSent', onRequest);
cdp.on("Network.loadingFinished", onFinish); cdp.on('Network.loadingFinished', onFinish);
cdp.on("Network.loadingFailed", onFinish); cdp.on('Network.loadingFailed', onFinish);
resetTimer(); resetTimer();
}); });
} }
export async function waitForPageLoad(cdp: CdpConnection, sessionId: string, timeoutMs: number = 30_000): Promise<void> { export async function waitForPageLoad(
return new Promise((resolve, reject) => { cdp: CdpConnection,
sessionId: string,
timeoutMs: number = 30_000,
): Promise<void> {
void sessionId;
return new Promise((resolve) => {
const timer = setTimeout(() => { const timer = setTimeout(() => {
cdp.off("Page.loadEventFired", handler); cdp.off('Page.loadEventFired', handler);
resolve(); resolve();
}, timeoutMs); }, timeoutMs);
const handler = () => { const handler = () => {
clearTimeout(timer); clearTimeout(timer);
cdp.off("Page.loadEventFired", handler); cdp.off('Page.loadEventFired', handler);
resolve(); resolve();
}; };
cdp.on("Page.loadEventFired", handler); cdp.on('Page.loadEventFired', handler);
}); });
} }
export async function createTargetAndAttach(cdp: CdpConnection, url: string): Promise<{ targetId: string; sessionId: string }> { export async function createTargetAndAttach(
const { targetId } = await cdp.send<{ targetId: string }>("Target.createTarget", { url }); cdp: CdpConnection,
const { sessionId } = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true }); url: string,
await cdp.send("Network.enable", {}, { sessionId }); ): Promise<{ targetId: string; sessionId: string }> {
await cdp.send("Page.enable", {}, { sessionId }); const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url });
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId, flatten: true });
await cdp.send('Network.enable', {}, { sessionId });
await cdp.send('Page.enable', {}, { sessionId });
return { targetId, sessionId }; return { targetId, sessionId };
} }
export async function navigateAndWait(cdp: CdpConnection, sessionId: string, url: string, timeoutMs: number): Promise<void> { export async function navigateAndWait(
cdp: CdpConnection,
sessionId: string,
url: string,
timeoutMs: number,
): Promise<void> {
const loadPromise = new Promise<void>((resolve, reject) => { const loadPromise = new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("Page load timeout")), timeoutMs); const timer = setTimeout(() => reject(new Error('Page load timeout')), timeoutMs);
const handler = (params: unknown) => { const handler = (params: unknown) => {
const p = params as { name?: string }; const event = params as { name?: string };
if (p.name === "load" || p.name === "DOMContentLoaded") { if (event.name === 'load' || event.name === 'DOMContentLoaded') {
clearTimeout(timer); clearTimeout(timer);
cdp.off("Page.lifecycleEvent", handler); cdp.off('Page.lifecycleEvent', handler);
resolve(); resolve();
} }
}; };
cdp.on("Page.lifecycleEvent", handler); cdp.on('Page.lifecycleEvent', handler);
}); });
await cdp.send("Page.navigate", { url }, { sessionId }); await cdp.send('Page.navigate', { url }, { sessionId });
await loadPromise; await loadPromise;
} }
export async function evaluateScript<T>(cdp: CdpConnection, sessionId: string, expression: string, timeoutMs: number = 30_000): Promise<T> { export async function evaluateScript<T>(
const result = await cdp.send<{ result: { value?: T; type?: string; description?: string } }>( cdp: CdpConnection,
"Runtime.evaluate", sessionId: string,
expression: string,
timeoutMs: number = 30_000,
): Promise<T> {
const result = await cdp.send<{ result: { value?: T } }>(
'Runtime.evaluate',
{ expression, returnByValue: true, awaitPromise: true }, { expression, returnByValue: true, awaitPromise: true },
{ sessionId, timeoutMs } { sessionId, timeoutMs },
); );
return result.result.value as T; return result.result.value as T;
} }
export async function autoScroll(cdp: CdpConnection, sessionId: string, steps: number = 8, waitMs: number = 600): Promise<void> { export async function autoScroll(
let lastHeight = await evaluateScript<number>(cdp, sessionId, "document.body.scrollHeight"); cdp: CdpConnection,
sessionId: string,
steps: number = 8,
waitMs: number = 600,
): Promise<void> {
let lastHeight = await evaluateScript<number>(cdp, sessionId, 'document.body.scrollHeight');
for (let i = 0; i < steps; i++) { for (let i = 0; i < steps; i++) {
await evaluateScript<void>(cdp, sessionId, "window.scrollTo(0, document.body.scrollHeight)"); await evaluateScript<void>(cdp, sessionId, 'window.scrollTo(0, document.body.scrollHeight)');
await sleep(waitMs); await sleep(waitMs);
const newHeight = await evaluateScript<number>(cdp, sessionId, "document.body.scrollHeight"); const newHeight = await evaluateScript<number>(cdp, sessionId, 'document.body.scrollHeight');
if (newHeight === lastHeight) break; if (newHeight === lastHeight) break;
lastHeight = newHeight; lastHeight = newHeight;
} }
await evaluateScript<void>(cdp, sessionId, "window.scrollTo(0, 0)"); await evaluateScript<void>(cdp, sessionId, 'window.scrollTo(0, 0)');
}
export function killChrome(chrome: ChildProcess): void {
try { chrome.kill("SIGTERM"); } catch {}
setTimeout(() => {
if (!chrome.killed) {
try { chrome.kill("SIGKILL"); } catch {}
}
}, 2_000).unref?.();
} }
@@ -0,0 +1,58 @@
import { JSDOM, VirtualConsole } from "jsdom";
import { Defuddle } from "defuddle/node";
import {
type ConversionResult,
type PageMetadata,
isMarkdownUsable,
normalizeMarkdown,
pickString,
} from "./markdown-conversion-shared.js";
export async function tryDefuddleConversion(
html: string,
url: string,
baseMetadata: PageMetadata
): Promise<{ ok: true; result: ConversionResult } | { ok: false; reason: string }> {
try {
const virtualConsole = new VirtualConsole();
virtualConsole.on("jsdomError", (error: Error & { type?: string }) => {
if (error.type === "css parsing" || /Could not parse CSS stylesheet/i.test(error.message)) {
return;
}
console.warn(`[url-to-markdown] jsdom: ${error.message}`);
});
const dom = new JSDOM(html, { url, virtualConsole });
const result = await Defuddle(dom, url, { markdown: true });
const markdown = normalizeMarkdown(result.content || "");
if (!isMarkdownUsable(markdown, html)) {
return { ok: false, reason: "Defuddle returned empty or incomplete markdown" };
}
return {
ok: true,
result: {
metadata: {
...baseMetadata,
title: pickString(result.title, baseMetadata.title) ?? "",
description: pickString(result.description, baseMetadata.description) ?? undefined,
author: pickString(result.author, baseMetadata.author) ?? undefined,
published: pickString(result.published, baseMetadata.published) ?? undefined,
coverImage: pickString(result.image, baseMetadata.coverImage) ?? undefined,
language: pickString(result.language, baseMetadata.language) ?? undefined,
},
markdown,
rawHtml: html,
conversionMethod: "defuddle",
variables: result.variables,
},
};
} catch (error) {
return {
ok: false,
reason: error instanceof Error ? error.message : String(error),
};
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,629 @@
import { Readability } from "@mozilla/readability";
import TurndownService from "turndown";
import { gfm } from "turndown-plugin-gfm";
import {
type AnyRecord,
type ConversionResult,
type PageMetadata,
GOOD_CONTENT_LENGTH,
MIN_CONTENT_LENGTH,
extractPublishedTime,
extractTextFromHtml,
extractTitle,
normalizeMarkdown,
parseDocument,
pickString,
sanitizeHtml,
} from "./markdown-conversion-shared.js";
interface ExtractionCandidate {
title: string | null;
byline: string | null;
excerpt: string | null;
published: string | null;
html: string | null;
textContent: string;
method: string;
}
const CONTENT_SELECTORS = [
"article",
"main article",
"[role='main'] article",
"[itemprop='articleBody']",
".article-content",
".article-body",
".post-content",
".entry-content",
".story-body",
"main",
"[role='main']",
"#content",
".content",
];
const REMOVE_SELECTORS = [
"script",
"style",
"noscript",
"template",
"iframe",
"svg",
"path",
"nav",
"aside",
"footer",
"header",
"form",
".advertisement",
".ads",
".social-share",
".related-articles",
".comments",
".newsletter",
".cookie-banner",
".cookie-consent",
"[role='navigation']",
"[aria-label*='cookie' i]",
];
const NEXT_DATA_CONTENT_PATHS = [
"props.pageProps.content.body",
"props.pageProps.article.body",
"props.pageProps.article.content",
"props.pageProps.post.body",
"props.pageProps.post.content",
"props.pageProps.data.body",
"props.pageProps.story.body.content",
];
const LOW_QUALITY_MARKERS = [
/Join The Conversation/i,
/One Community\. Many Voices/i,
/Read our community guidelines/i,
/Create a free account to share your thoughts/i,
/Become a Forbes Member/i,
/Subscribe to trusted journalism/i,
/\bComments\b/i,
];
function generateExcerpt(excerpt: string | null, textContent: string | null): string | null {
if (excerpt) return excerpt;
if (!textContent) return null;
const trimmed = textContent.trim();
if (!trimmed) return null;
return trimmed.length > 200 ? `${trimmed.slice(0, 200)}...` : trimmed;
}
function parseJsonLdItem(item: AnyRecord): ExtractionCandidate | null {
const type = Array.isArray(item["@type"]) ? item["@type"][0] : item["@type"];
if (typeof type !== "string" || !["Article", "NewsArticle", "BlogPosting", "WebPage", "ReportageNewsArticle"].includes(type)) {
return null;
}
const rawContent =
(typeof item.articleBody === "string" && item.articleBody) ||
(typeof item.text === "string" && item.text) ||
(typeof item.description === "string" && item.description) ||
null;
if (!rawContent) return null;
const content = rawContent.trim();
const htmlLike = /<\/?[a-z][\s\S]*>/i.test(content);
const textContent = htmlLike ? extractTextFromHtml(content) : content;
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(item.headline, item.name),
byline: extractAuthorFromJsonLd(item.author),
excerpt: pickString(item.description),
published: pickString(item.datePublished, item.dateCreated),
html: htmlLike ? content : null,
textContent,
method: "json-ld",
};
}
function extractAuthorFromJsonLd(authorData: unknown): string | null {
if (typeof authorData === "string") return authorData;
if (!authorData || typeof authorData !== "object") return null;
if (Array.isArray(authorData)) {
const names = authorData
.map((author) => extractAuthorFromJsonLd(author))
.filter((name): name is string => Boolean(name));
return names.length > 0 ? names.join(", ") : null;
}
const author = authorData as AnyRecord;
return typeof author.name === "string" ? author.name : null;
}
function flattenJsonLdItems(data: unknown): AnyRecord[] {
if (!data || typeof data !== "object") return [];
if (Array.isArray(data)) return data.flatMap(flattenJsonLdItems);
const item = data as AnyRecord;
if (Array.isArray(item["@graph"])) {
return (item["@graph"] as unknown[]).flatMap(flattenJsonLdItems);
}
return [item];
}
function tryJsonLdExtraction(document: Document): ExtractionCandidate | null {
const scripts = document.querySelectorAll("script[type='application/ld+json']");
for (const script of scripts) {
try {
const data = JSON.parse(script.textContent ?? "");
for (const item of flattenJsonLdItems(data)) {
const extracted = parseJsonLdItem(item);
if (extracted) return extracted;
}
} catch {
// Ignore malformed blocks.
}
}
return null;
}
function getByPath(value: unknown, path: string): unknown {
let current = value;
for (const part of path.split(".")) {
if (!current || typeof current !== "object") return undefined;
current = (current as AnyRecord)[part];
}
return current;
}
function isContentBlockArray(value: unknown): value is AnyRecord[] {
if (!Array.isArray(value) || value.length === 0) return false;
return value.slice(0, 5).some((item) => {
if (!item || typeof item !== "object") return false;
const obj = item as AnyRecord;
return "type" in obj || "text" in obj || "textHtml" in obj || "content" in obj;
});
}
function extractTextFromContentBlocks(blocks: AnyRecord[]): string {
const parts: string[] = [];
function pushParagraph(text: string): void {
const trimmed = text.trim();
if (!trimmed) return;
parts.push(trimmed, "\n\n");
}
function walk(node: unknown): void {
if (!node || typeof node !== "object") return;
const block = node as AnyRecord;
if (typeof block.text === "string") {
pushParagraph(block.text);
return;
}
if (typeof block.textHtml === "string") {
pushParagraph(extractTextFromHtml(block.textHtml));
return;
}
if (Array.isArray(block.items)) {
for (const item of block.items) {
if (item && typeof item === "object") {
const text = pickString((item as AnyRecord).text);
if (text) parts.push(`- ${text}\n`);
}
}
parts.push("\n");
}
if (Array.isArray(block.components)) {
for (const component of block.components) {
walk(component);
}
}
if (Array.isArray(block.content)) {
for (const child of block.content) {
walk(child);
}
}
}
for (const block of blocks) {
walk(block);
}
return parts.join("").replace(/\n{3,}/g, "\n\n").trim();
}
function tryStringBodyExtraction(
content: string,
meta: AnyRecord,
document: Document,
method: string
): ExtractionCandidate | null {
if (!content || content.length < MIN_CONTENT_LENGTH) return null;
const isHtml = /<\/?[a-z][\s\S]*>/i.test(content);
const html = isHtml ? sanitizeHtml(content) : null;
const textContent = isHtml ? extractTextFromHtml(html) : content.trim();
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(meta.headline, meta.title, extractTitle(document)),
byline: pickString(meta.byline, meta.author),
excerpt: pickString(meta.description, meta.excerpt, generateExcerpt(null, textContent)),
published: pickString(meta.datePublished, meta.publishedAt, extractPublishedTime(document)),
html,
textContent,
method,
};
}
function tryNextDataExtraction(document: Document): ExtractionCandidate | null {
try {
const script = document.querySelector("script#__NEXT_DATA__");
if (!script?.textContent) return null;
const data = JSON.parse(script.textContent) as AnyRecord;
const pageProps = (getByPath(data, "props.pageProps") ?? {}) as AnyRecord;
for (const path of NEXT_DATA_CONTENT_PATHS) {
const value = getByPath(data, path);
if (typeof value === "string") {
const parentPath = path.split(".").slice(0, -1).join(".");
const parent = (getByPath(data, parentPath) ?? {}) as AnyRecord;
const meta = {
...pageProps,
...parent,
title: parent.title ?? (pageProps.title as string | undefined),
};
const candidate = tryStringBodyExtraction(value, meta, document, "next-data");
if (candidate) return candidate;
}
if (isContentBlockArray(value)) {
const textContent = extractTextFromContentBlocks(value);
if (textContent.length < MIN_CONTENT_LENGTH) continue;
return {
title: pickString(
getByPath(data, "props.pageProps.content.headline"),
getByPath(data, "props.pageProps.article.headline"),
getByPath(data, "props.pageProps.article.title"),
getByPath(data, "props.pageProps.post.title"),
pageProps.title,
extractTitle(document)
),
byline: pickString(
getByPath(data, "props.pageProps.author.name"),
getByPath(data, "props.pageProps.article.author.name")
),
excerpt: pickString(
getByPath(data, "props.pageProps.content.description"),
getByPath(data, "props.pageProps.article.description"),
pageProps.description,
generateExcerpt(null, textContent)
),
published: pickString(
getByPath(data, "props.pageProps.content.datePublished"),
getByPath(data, "props.pageProps.article.datePublished"),
getByPath(data, "props.pageProps.publishedAt"),
extractPublishedTime(document)
),
html: null,
textContent,
method: "next-data",
};
}
}
} catch {
return null;
}
return null;
}
function buildReadabilityCandidate(
article: ReturnType<Readability["parse"]>,
document: Document,
method: string
): ExtractionCandidate | null {
const textContent = article?.textContent?.trim() ?? "";
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(article?.title, extractTitle(document)),
byline: pickString((article as { byline?: string } | null)?.byline),
excerpt: pickString(article?.excerpt, generateExcerpt(null, textContent)),
published: pickString((article as { publishedTime?: string } | null)?.publishedTime, extractPublishedTime(document)),
html: article?.content ? sanitizeHtml(article.content) : null,
textContent,
method,
};
}
function tryReadability(document: Document): ExtractionCandidate | null {
try {
const strictClone = document.cloneNode(true) as Document;
const strictResult = buildReadabilityCandidate(
new Readability(strictClone).parse(),
document,
"readability"
);
if (strictResult) return strictResult;
const relaxedClone = document.cloneNode(true) as Document;
return buildReadabilityCandidate(
new Readability(relaxedClone, { charThreshold: 120 }).parse(),
document,
"readability-relaxed"
);
} catch {
return null;
}
}
function trySelectorExtraction(document: Document): ExtractionCandidate | null {
for (const selector of CONTENT_SELECTORS) {
const element = document.querySelector(selector);
if (!element) continue;
const clone = element.cloneNode(true) as Element;
for (const removeSelector of REMOVE_SELECTORS) {
for (const node of clone.querySelectorAll(removeSelector)) {
node.remove();
}
}
const html = sanitizeHtml(clone.innerHTML);
const textContent = extractTextFromHtml(html);
if (textContent.length < MIN_CONTENT_LENGTH) continue;
return {
title: extractTitle(document),
byline: null,
excerpt: generateExcerpt(null, textContent),
published: extractPublishedTime(document),
html,
textContent,
method: `selector:${selector}`,
};
}
return null;
}
function tryBodyExtraction(document: Document): ExtractionCandidate | null {
const body = document.body;
if (!body) return null;
const clone = body.cloneNode(true) as Element;
for (const removeSelector of REMOVE_SELECTORS) {
for (const node of clone.querySelectorAll(removeSelector)) {
node.remove();
}
}
const html = sanitizeHtml(clone.innerHTML);
const textContent = extractTextFromHtml(html);
if (!textContent) return null;
return {
title: extractTitle(document),
byline: null,
excerpt: generateExcerpt(null, textContent),
published: extractPublishedTime(document),
html,
textContent,
method: "body-fallback",
};
}
function pickBestCandidate(candidates: ExtractionCandidate[]): ExtractionCandidate | null {
if (candidates.length === 0) return null;
const methodOrder = [
"readability",
"readability-relaxed",
"next-data",
"json-ld",
"selector:",
"body-fallback",
];
function methodRank(method: string): number {
const idx = methodOrder.findIndex((entry) =>
entry.endsWith(":") ? method.startsWith(entry) : method === entry
);
return idx === -1 ? methodOrder.length : idx;
}
const ranked = [...candidates].sort((a, b) => {
const rankA = methodRank(a.method);
const rankB = methodRank(b.method);
if (rankA !== rankB) return rankA - rankB;
return (b.textContent.length ?? 0) - (a.textContent.length ?? 0);
});
for (const candidate of ranked) {
if (candidate.textContent.length >= GOOD_CONTENT_LENGTH) {
return candidate;
}
}
for (const candidate of ranked) {
if (candidate.textContent.length >= MIN_CONTENT_LENGTH) {
return candidate;
}
}
return ranked[0];
}
function extractFromHtml(html: string): ExtractionCandidate | null {
const document = parseDocument(html);
const readabilityCandidate = tryReadability(document);
const nextDataCandidate = tryNextDataExtraction(document);
const jsonLdCandidate = tryJsonLdExtraction(document);
const selectorCandidate = trySelectorExtraction(document);
const bodyCandidate = tryBodyExtraction(document);
const candidates = [
readabilityCandidate,
nextDataCandidate,
jsonLdCandidate,
selectorCandidate,
bodyCandidate,
].filter((candidate): candidate is ExtractionCandidate => Boolean(candidate));
const winner = pickBestCandidate(candidates);
if (!winner) return null;
return {
...winner,
title: winner.title ?? extractTitle(document),
published: winner.published ?? extractPublishedTime(document),
excerpt: winner.excerpt ?? generateExcerpt(null, winner.textContent),
};
}
const turndown = new TurndownService({
headingStyle: "atx",
hr: "---",
bulletListMarker: "-",
codeBlockStyle: "fenced",
emDelimiter: "*",
strongDelimiter: "**",
linkStyle: "inlined",
});
turndown.use(gfm);
turndown.remove(["script", "style", "iframe", "noscript", "template", "svg", "path"]);
turndown.addRule("collapseFigure", {
filter: "figure",
replacement(content) {
return `\n\n${content.trim()}\n\n`;
},
});
turndown.addRule("dropInvisibleAnchors", {
filter(node) {
return node.nodeName === "A" && !(node as Element).textContent?.trim();
},
replacement() {
return "";
},
});
function convertHtmlToMarkdown(html: string): string {
if (!html || !html.trim()) return "";
try {
const sanitized = sanitizeHtml(html);
return turndown.turndown(sanitized);
} catch {
return "";
}
}
function fallbackPlainText(html: string): string {
const document = parseDocument(html);
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
for (const el of document.querySelectorAll(selector)) {
el.remove();
}
}
const text = document.body?.textContent ?? document.documentElement?.textContent ?? "";
return normalizeMarkdown(text.replace(/\s+/g, " "));
}
function countBylines(markdown: string): number {
return (markdown.match(/(^|\n)By\s+/g) || []).length;
}
function countUsefulParagraphs(markdown: string): number {
const paragraphs = normalizeMarkdown(markdown).split(/\n{2,}/);
let count = 0;
for (const paragraph of paragraphs) {
const trimmed = paragraph.trim();
if (!trimmed) continue;
if (/^!?\[[^\]]*\]\([^)]+\)$/.test(trimmed)) continue;
if (/^#{1,6}\s+/.test(trimmed)) continue;
if ((trimmed.match(/\b[\p{L}\p{N}']+\b/gu) || []).length < 8) continue;
count++;
}
return count;
}
function countMarkerHits(markdown: string, markers: RegExp[]): number {
let hits = 0;
for (const marker of markers) {
if (marker.test(markdown)) hits++;
}
return hits;
}
export function scoreMarkdownQuality(markdown: string): number {
const normalized = normalizeMarkdown(markdown);
const wordCount = (normalized.match(/\b[\p{L}\p{N}']+\b/gu) || []).length;
const usefulParagraphs = countUsefulParagraphs(normalized);
const headingCount = (normalized.match(/^#{1,6}\s+/gm) || []).length;
const markerHits = countMarkerHits(normalized, LOW_QUALITY_MARKERS);
const bylineCount = countBylines(normalized);
const staffCount = (normalized.match(/\bForbes Staff\b/gi) || []).length;
return (
Math.min(wordCount, 4000) +
usefulParagraphs * 40 +
headingCount * 10 -
markerHits * 180 -
Math.max(0, bylineCount - 1) * 120 -
Math.max(0, staffCount - 1) * 80
);
}
export function shouldCompareWithLegacy(markdown: string): boolean {
const normalized = normalizeMarkdown(markdown);
return (
countMarkerHits(normalized, LOW_QUALITY_MARKERS) > 0 ||
countBylines(normalized) > 1 ||
countUsefulParagraphs(normalized) < 6
);
}
export function convertWithLegacyExtractor(html: string, baseMetadata: PageMetadata): ConversionResult {
const extracted = extractFromHtml(html);
let markdown = extracted?.html ? convertHtmlToMarkdown(extracted.html) : "";
if (!markdown.trim()) {
markdown = extracted?.textContent?.trim() || fallbackPlainText(html);
}
return {
metadata: {
...baseMetadata,
title: pickString(extracted?.title, baseMetadata.title) ?? "",
description: pickString(extracted?.excerpt, baseMetadata.description) ?? undefined,
author: pickString(extracted?.byline, baseMetadata.author) ?? undefined,
published: pickString(extracted?.published, baseMetadata.published) ?? undefined,
},
markdown: normalizeMarkdown(markdown),
rawHtml: html,
conversionMethod: extracted ? `legacy:${extracted.method}` : "legacy:plain-text",
};
}
+116 -13
View File
@@ -3,7 +3,7 @@ import { writeFile, mkdir, access } from "node:fs/promises";
import path from "node:path"; import path from "node:path";
import process from "node:process"; import process from "node:process";
import { CdpConnection, getFreePort, launchChrome, waitForChromeDebugPort, waitForNetworkIdle, waitForPageLoad, autoScroll, evaluateScript, killChrome } from "./cdp.js"; import { CdpConnection, getFreePort, findExistingChromePort, launchChrome, waitForChromeDebugPort, waitForNetworkIdle, waitForPageLoad, autoScroll, evaluateScript, killChrome } from "./cdp.js";
import { absolutizeUrlsScript, extractContent, createMarkdownDocument, type ConversionResult } from "./html-to-markdown.js"; import { absolutizeUrlsScript, extractContent, createMarkdownDocument, type ConversionResult } from "./html-to-markdown.js";
import { localizeMarkdownMedia, countRemoteMedia } from "./media-localizer.js"; import { localizeMarkdownMedia, countRemoteMedia } from "./media-localizer.js";
import { resolveUrlToMarkdownDataDir } from "./paths.js"; import { resolveUrlToMarkdownDataDir } from "./paths.js";
@@ -75,6 +75,55 @@ function deriveHtmlSnapshotPath(markdownPath: string): string {
return path.join(parsed.dir, `${basename}-captured.html`); return path.join(parsed.dir, `${basename}-captured.html`);
} }
function extractTitleFromMarkdownDocument(document: string): string {
const normalized = document.replace(/\r\n/g, "\n");
const frontmatterMatch = normalized.match(/^---\n([\s\S]*?)\n---\n?/);
if (frontmatterMatch) {
const titleLine = frontmatterMatch[1]
.split("\n")
.find((line) => /^title:\s*/i.test(line));
if (titleLine) {
const rawValue = titleLine.replace(/^title:\s*/i, "").trim();
const unquoted = rawValue
.replace(/^"(.*)"$/, "$1")
.replace(/^'(.*)'$/, "$1")
.replace(/\\"/g, '"');
if (unquoted) return unquoted;
}
}
const headingMatch = normalized.match(/^#\s+(.+)$/m);
return headingMatch?.[1]?.trim() ?? "";
}
function buildDefuddleApiUrl(targetUrl: string): string {
return `https://defuddle.md/${encodeURIComponent(targetUrl)}`;
}
async function fetchDefuddleApiMarkdown(targetUrl: string): Promise<{ markdown: string; title: string }> {
const apiUrl = buildDefuddleApiUrl(targetUrl);
const response = await fetch(apiUrl, {
headers: {
accept: "text/markdown,text/plain;q=0.9,*/*;q=0.1",
},
});
if (!response.ok) {
throw new Error(`defuddle.md returned ${response.status} ${response.statusText}`);
}
const markdown = (await response.text()).replace(/\r\n/g, "\n").trim();
if (!markdown) {
throw new Error("defuddle.md returned empty markdown");
}
return {
markdown,
title: extractTitleFromMarkdownDocument(markdown),
};
}
async function generateOutputPath(url: string, title: string, outputDir?: string): Promise<string> { async function generateOutputPath(url: string, title: string, outputDir?: string): Promise<string> {
const domain = new URL(url).hostname.replace(/^www\./, ""); const domain = new URL(url).hostname.replace(/^www\./, "");
const slug = generateSlug(title, url); const slug = generateSlug(title, url);
@@ -98,21 +147,37 @@ async function waitForUserSignal(): Promise<void> {
} }
async function captureUrl(args: Args): Promise<ConversionResult> { async function captureUrl(args: Args): Promise<ConversionResult> {
const port = await getFreePort(); const existingPort = await findExistingChromePort();
const chrome = await launchChrome(args.url, port, false); const reusing = existingPort !== null;
const port = existingPort ?? await getFreePort();
const chrome = reusing ? null : await launchChrome(args.url, port, false);
if (reusing) console.log(`Reusing existing Chrome on port ${port}`);
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000);
cdp = await CdpConnection.connect(wsUrl, CDP_CONNECT_TIMEOUT_MS); cdp = await CdpConnection.connect(wsUrl, CDP_CONNECT_TIMEOUT_MS);
let sessionId: string;
if (reusing) {
const created = await cdp.send<{ targetId: string }>("Target.createTarget", { url: args.url });
targetId = created.targetId;
const attached = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true });
sessionId = attached.sessionId;
await cdp.send("Network.enable", {}, { sessionId });
await cdp.send("Page.enable", {}, { sessionId });
} else {
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; type: string; url: string }> }>("Target.getTargets"); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; type: string; url: string }> }>("Target.getTargets");
const pageTarget = targets.targetInfos.find(t => t.type === "page" && t.url.startsWith("http")); const pageTarget = targets.targetInfos.find(t => t.type === "page" && t.url.startsWith("http"));
if (!pageTarget) throw new Error("No page target found"); if (!pageTarget) throw new Error("No page target found");
targetId = pageTarget.targetId;
const { sessionId } = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId: pageTarget.targetId, flatten: true }); const attached = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true });
sessionId = attached.sessionId;
await cdp.send("Network.enable", {}, { sessionId }); await cdp.send("Network.enable", {}, { sessionId });
await cdp.send("Page.enable", {}, { sessionId }); await cdp.send("Page.enable", {}, { sessionId });
}
if (args.wait) { if (args.wait) {
await waitForUserSignal(); await waitForUserSignal();
@@ -136,11 +201,18 @@ async function captureUrl(args: Args): Promise<ConversionResult> {
return await extractContent(html, args.url); return await extractContent(html, args.url);
} finally { } finally {
if (reusing) {
if (cdp && targetId) {
try { await cdp.send("Target.closeTarget", { targetId }, { timeoutMs: 5_000 }); } catch {}
}
if (cdp) cdp.close();
} else {
if (cdp) { if (cdp) {
try { await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); } catch {} try { await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); } catch {}
cdp.close(); cdp.close();
} }
killChrome(chrome); if (chrome) killChrome(chrome);
}
} }
} }
@@ -169,14 +241,41 @@ async function main(): Promise<void> {
console.log(`Fetching: ${args.url}`); console.log(`Fetching: ${args.url}`);
console.log(`Mode: ${args.wait ? "wait" : "auto"}`); console.log(`Mode: ${args.wait ? "wait" : "auto"}`);
let outputPath: string;
let htmlSnapshotPath: string | null = null;
let document: string;
let conversionMethod: string;
let fallbackReason: string | undefined;
try {
const result = await captureUrl(args); const result = await captureUrl(args);
const outputPath = args.output || await generateOutputPath(args.url, result.metadata.title, args.outputDir); outputPath = args.output || await generateOutputPath(args.url, result.metadata.title, args.outputDir);
const outputDir = path.dirname(outputPath); const outputDir = path.dirname(outputPath);
const htmlSnapshotPath = deriveHtmlSnapshotPath(outputPath); htmlSnapshotPath = deriveHtmlSnapshotPath(outputPath);
await mkdir(outputDir, { recursive: true }); await mkdir(outputDir, { recursive: true });
await writeFile(htmlSnapshotPath, result.rawHtml, "utf-8"); await writeFile(htmlSnapshotPath, result.rawHtml, "utf-8");
let document = createMarkdownDocument(result); document = createMarkdownDocument(result);
conversionMethod = result.conversionMethod;
fallbackReason = result.fallbackReason;
} catch (error) {
const primaryError = error instanceof Error ? error.message : String(error);
console.warn(`Primary capture failed: ${primaryError}`);
console.warn("Trying defuddle.md API fallback...");
try {
const remoteResult = await fetchDefuddleApiMarkdown(args.url);
outputPath = args.output || await generateOutputPath(args.url, remoteResult.title, args.outputDir);
await mkdir(path.dirname(outputPath), { recursive: true });
document = remoteResult.markdown;
conversionMethod = "defuddle-api";
fallbackReason = `Local browser capture failed: ${primaryError}`;
} catch (remoteError) {
const remoteMessage = remoteError instanceof Error ? remoteError.message : String(remoteError);
throw new Error(`Local browser capture failed (${primaryError}); defuddle.md fallback failed (${remoteMessage})`);
}
}
if (args.downloadMedia) { if (args.downloadMedia) {
const mediaResult = await localizeMarkdownMedia(document, { const mediaResult = await localizeMarkdownMedia(document, {
@@ -197,11 +296,15 @@ async function main(): Promise<void> {
await writeFile(outputPath, document, "utf-8"); await writeFile(outputPath, document, "utf-8");
console.log(`Saved: ${outputPath}`); console.log(`Saved: ${outputPath}`);
if (htmlSnapshotPath) {
console.log(`Saved HTML: ${htmlSnapshotPath}`); console.log(`Saved HTML: ${htmlSnapshotPath}`);
console.log(`Title: ${result.metadata.title || "(no title)"}`); } else {
console.log(`Converter: ${result.conversionMethod}`); console.log("Saved HTML: unavailable (defuddle.md fallback)");
if (result.fallbackReason) { }
console.warn(`Fallback used: ${result.fallbackReason}`); console.log(`Title: ${extractTitleFromMarkdownDocument(document) || "(no title)"}`);
console.log(`Converter: ${conversionMethod}`);
if (fallbackReason) {
console.warn(`Fallback used: ${fallbackReason}`);
} }
} }
@@ -0,0 +1,305 @@
import { parseHTML } from "linkedom";
export interface PageMetadata {
url: string;
title: string;
description?: string;
author?: string;
published?: string;
coverImage?: string;
language?: string;
captured_at: string;
}
export interface ConversionResult {
metadata: PageMetadata;
markdown: string;
rawHtml: string;
conversionMethod: string;
fallbackReason?: string;
variables?: Record<string, string>;
}
export type AnyRecord = Record<string, unknown>;
export const MIN_CONTENT_LENGTH = 120;
export const GOOD_CONTENT_LENGTH = 900;
const PUBLISHED_TIME_SELECTORS = [
"meta[property='article:published_time']",
"meta[name='pubdate']",
"meta[name='publishdate']",
"meta[name='date']",
"time[datetime]",
];
const ARTICLE_TYPES = new Set([
"Article",
"NewsArticle",
"BlogPosting",
"WebPage",
"ReportageNewsArticle",
]);
export function pickString(...values: unknown[]): string | null {
for (const value of values) {
if (typeof value === "string") {
const trimmed = value.trim();
if (trimmed) return trimmed;
}
}
return null;
}
export function normalizeMarkdown(markdown: string): string {
return markdown
.replace(/\r\n/g, "\n")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
}
export function parseDocument(html: string): Document {
const normalized = /<\s*html[\s>]/i.test(html)
? html
: `<!doctype html><html><body>${html}</body></html>`;
return parseHTML(normalized).document as unknown as Document;
}
export function sanitizeHtml(html: string): string {
const { document } = parseHTML(`<div id="__root">${html}</div>`);
const root = document.querySelector("#__root");
if (!root) return html;
for (const selector of ["script", "style", "iframe", "noscript", "template", "svg", "path"]) {
for (const el of root.querySelectorAll(selector)) {
el.remove();
}
}
return root.innerHTML;
}
export function extractTextFromHtml(html: string): string {
const { document } = parseHTML(`<!doctype html><html><body>${html}</body></html>`);
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
for (const el of document.querySelectorAll(selector)) {
el.remove();
}
}
return document.body?.textContent?.replace(/\s+/g, " ").trim() ?? "";
}
export function getMetaContent(document: Document, names: string[]): string | null {
for (const name of names) {
const element =
document.querySelector(`meta[name="${name}"]`) ??
document.querySelector(`meta[property="${name}"]`);
const content = element?.getAttribute("content");
if (content && content.trim()) return content.trim();
}
return null;
}
function normalizeLanguageTag(value: string | null): string | null {
if (!value) return null;
const trimmed = value.trim();
if (!trimmed) return null;
const primary = trimmed.split(/[,\s;]/, 1)[0]?.trim();
if (!primary) return null;
return primary.replace(/_/g, "-");
}
function flattenJsonLdItems(data: unknown): AnyRecord[] {
if (!data || typeof data !== "object") return [];
if (Array.isArray(data)) return data.flatMap(flattenJsonLdItems);
const item = data as AnyRecord;
if (Array.isArray(item["@graph"])) {
return (item["@graph"] as unknown[]).flatMap(flattenJsonLdItems);
}
return [item];
}
function parseJsonLdScripts(document: Document): AnyRecord[] {
const results: AnyRecord[] = [];
const scripts = document.querySelectorAll("script[type='application/ld+json']");
for (const script of scripts) {
try {
const data = JSON.parse(script.textContent ?? "");
results.push(...flattenJsonLdItems(data));
} catch {
// Ignore malformed blocks.
}
}
return results;
}
function isArticleType(item: AnyRecord): boolean {
const value = Array.isArray(item["@type"]) ? item["@type"][0] : item["@type"];
return typeof value === "string" && ARTICLE_TYPES.has(value);
}
function extractAuthorFromJsonLd(authorData: unknown): string | null {
if (typeof authorData === "string") return authorData;
if (!authorData || typeof authorData !== "object") return null;
if (Array.isArray(authorData)) {
const names = authorData
.map((author) => extractAuthorFromJsonLd(author))
.filter((name): name is string => Boolean(name));
return names.length > 0 ? names.join(", ") : null;
}
const author = authorData as AnyRecord;
return typeof author.name === "string" ? author.name : null;
}
function extractPrimaryJsonLdMeta(document: Document): Partial<PageMetadata> {
for (const item of parseJsonLdScripts(document)) {
if (!isArticleType(item)) continue;
return {
title: pickString(item.headline, item.name) ?? undefined,
description: pickString(item.description) ?? undefined,
author: extractAuthorFromJsonLd(item.author) ?? undefined,
published: pickString(item.datePublished, item.dateCreated) ?? undefined,
coverImage:
pickString(
item.image,
(item.image as AnyRecord | undefined)?.url,
(Array.isArray(item.image) ? item.image[0] : undefined) as unknown
) ?? undefined,
};
}
return {};
}
export function extractPublishedTime(document: Document): string | null {
for (const selector of PUBLISHED_TIME_SELECTORS) {
const el = document.querySelector(selector);
if (!el) continue;
const value = el.getAttribute("content") ?? el.getAttribute("datetime");
if (value && value.trim()) return value.trim();
}
return null;
}
export function extractTitle(document: Document): string | null {
const ogTitle = document.querySelector("meta[property='og:title']")?.getAttribute("content");
if (ogTitle && ogTitle.trim()) return ogTitle.trim();
const twitterTitle = document.querySelector("meta[name='twitter:title']")?.getAttribute("content");
if (twitterTitle && twitterTitle.trim()) return twitterTitle.trim();
const title = document.querySelector("title")?.textContent?.trim();
if (title) {
const cleaned = title.split(/\s*[-|–—]\s*/)[0]?.trim();
if (cleaned) return cleaned;
}
const h1 = document.querySelector("h1")?.textContent?.trim();
return h1 || null;
}
export function extractMetadataFromHtml(html: string, url: string, capturedAt: string): PageMetadata {
const document = parseDocument(html);
const jsonLd = extractPrimaryJsonLdMeta(document);
const timeEl = document.querySelector("time[datetime]");
const htmlLang = normalizeLanguageTag(document.documentElement?.getAttribute("lang"));
const metaLanguage = normalizeLanguageTag(
pickString(
getMetaContent(document, ["language", "content-language", "og:locale"]),
document.querySelector("meta[http-equiv='content-language']")?.getAttribute("content")
)
);
return {
url,
title:
pickString(
getMetaContent(document, ["og:title", "twitter:title"]),
jsonLd.title,
document.querySelector("h1")?.textContent,
document.title
) ?? "",
description:
pickString(
getMetaContent(document, ["description", "og:description", "twitter:description"]),
jsonLd.description
) ?? undefined,
author:
pickString(
getMetaContent(document, ["author", "article:author", "twitter:creator"]),
jsonLd.author
) ?? undefined,
published:
pickString(
timeEl?.getAttribute("datetime"),
getMetaContent(document, ["article:published_time", "datePublished", "publishdate", "date"]),
jsonLd.published,
extractPublishedTime(document)
) ?? undefined,
coverImage:
pickString(
getMetaContent(document, ["og:image", "twitter:image", "twitter:image:src"]),
jsonLd.coverImage
) ?? undefined,
language: pickString(htmlLang, metaLanguage) ?? undefined,
captured_at: capturedAt,
};
}
export function isMarkdownUsable(markdown: string, html: string): boolean {
const normalized = normalizeMarkdown(markdown);
if (!normalized) return false;
const htmlTextLength = extractTextFromHtml(html).length;
if (htmlTextLength < MIN_CONTENT_LENGTH) return true;
if (normalized.length >= 80) return true;
return normalized.length >= Math.min(200, Math.floor(htmlTextLength * 0.2));
}
export function isYouTubeUrl(url: string): boolean {
try {
const hostname = new URL(url).hostname.toLowerCase();
return hostname === "youtu.be" || hostname.endsWith(".youtube.com") || hostname === "youtube.com";
} catch {
return false;
}
}
function escapeYamlValue(value: string): string {
return value.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\r?\n/g, "\\n");
}
export function formatMetadataYaml(meta: PageMetadata): string {
const lines = ["---"];
lines.push(`url: ${meta.url}`);
lines.push(`title: "${escapeYamlValue(meta.title)}"`);
if (meta.description) lines.push(`description: "${escapeYamlValue(meta.description)}"`);
if (meta.author) lines.push(`author: "${escapeYamlValue(meta.author)}"`);
if (meta.published) lines.push(`published: "${escapeYamlValue(meta.published)}"`);
if (meta.coverImage) lines.push(`coverImage: "${escapeYamlValue(meta.coverImage)}"`);
if (meta.language) lines.push(`language: "${escapeYamlValue(meta.language)}"`);
lines.push(`captured_at: "${escapeYamlValue(meta.captured_at)}"`);
lines.push("---");
return lines.join("\n");
}
export function createMarkdownDocument(result: ConversionResult): string {
const yaml = formatMetadataYaml(result.metadata);
const escapedTitle = result.metadata.title.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const titleRegex = new RegExp(`^#\\s+${escapedTitle}\\s*(\\n|$)`, "i");
const hasTitle = titleRegex.test(result.markdown.trimStart());
const title = result.metadata.title && !hasTitle ? `\n\n# ${result.metadata.title}\n\n` : "\n\n";
return yaml + title + result.markdown;
}
@@ -4,7 +4,8 @@
"type": "module", "type": "module",
"dependencies": { "dependencies": {
"@mozilla/readability": "^0.6.0", "@mozilla/readability": "^0.6.0",
"defuddle": "^0.10.0", "baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"defuddle": "^0.12.0",
"jsdom": "^24.1.3", "jsdom": "^24.1.3",
"linkedom": "^0.18.12", "linkedom": "^0.18.12",
"turndown": "^7.2.2", "turndown": "^7.2.2",
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}

Some files were not shown because too many files have changed in this diff Show More