Compare commits

...

20 Commits

Author SHA1 Message Date
Jim Liu 宝玉 53c788eb3b chore: release v1.60.0 2026-03-11 22:22:03 -05:00
Jim Liu 宝玉 c79066e96e fix(baoyu-post-to-x): add missing fs import in x-article 2026-03-11 22:18:11 -05:00
Jim Liu 宝玉 05dba5c320 refactor: publish skills directly from synced vendor 2026-03-11 21:36:19 -05:00
Jim Liu 宝玉 270a9af804 refactor: simplify CLAUDE.md, move detailed docs to docs/
CLAUDE.md reduced from 621 to 76 lines (↓88%). Detailed guidelines
moved to on-demand reference files under docs/.
2026-03-11 21:01:59 -05:00
Jim Liu 宝玉 3bba18c1fe build: commit vendored shared skill packages 2026-03-11 20:45:25 -05:00
Jim Liu 宝玉 069c5dc7d7 refactor: unify skill cdp and release artifacts 2026-03-11 19:38:59 -05:00
Jim Liu 宝玉 00bf946403 支持复用已有 Chrome CDP 实例,修复端口检测顺序问题 2026-03-11 17:24:18 -05:00
Jim Liu 宝玉 1cb54420e0 chore: release v1.59.1 2026-03-11 17:08:33 -05:00
Jim Liu 宝玉 6363bd83e2 refactor(scripts): replace clawhub CLI with local sync-clawhub.mjs 2026-03-11 16:45:57 -05:00
Jim Liu 宝玉 7b8247544d refactor(baoyu-post-to-weibo): add entry point guard to md-to-html.ts 2026-03-11 16:45:56 -05:00
Jim Liu 宝玉 a9576ebc67 fix(baoyu-post-to-x): remove disable-blink-features Chrome flag 2026-03-11 16:45:55 -05:00
Jim Liu 宝玉 79ca378229 fix(baoyu-translate): improve annotation density and style preset passing
- Add short text (<5 sentences) annotation reduction rule to translator's notes
- Explicitly pass resolved style preset to 02-prompt.md assembly in all modes
2026-03-11 16:45:51 -05:00
Jim Liu 宝玉 e1a1fe23cb Merge pull request #77 from jackL1020/claude/claude-md-mmlh0po6953zhhbf-D8rFo
docs: Update CLAUDE.md with v1.59.0 features and skill registry
2026-03-11 13:40:27 -05:00
Claude 3d85a7e663 docs: fix marketplace.json path (it's in .claude-plugin/, not root)
https://claude.ai/code/session_01DREj1A7KKhNVE1RXjmg9ZH
2026-03-11 03:28:56 +00:00
Claude 10aabb39f8 docs: update CLAUDE.md to reflect v1.59.0 codebase state
- Add 8 missing skills to architecture tree (baoyu-post-to-weibo,
  baoyu-infographic, baoyu-image-gen, baoyu-url-to-markdown,
  baoyu-format-markdown, baoyu-markdown-to-html, baoyu-translate)
- Fix marketplace.json path (root, not .claude-plugin/)
- Rename SKILL_DIR → baseDir in Script Directory template (v1.57.0 change)
- Add ClawHub/OpenClaw publishing section with sync-clawhub.sh usage
- Add openclaw frontmatter template and requirement for new skills
- Document baoyu-image-gen batch parallel generation capability
- Update Project Overview to mention all AI providers
- Expand Key Dependencies and Choosing a Category tables
- Add image-gen API key requirement note

https://claude.ai/code/session_01DREj1A7KKhNVE1RXjmg9ZH
2026-03-11 03:28:14 +00:00
Jim Liu 宝玉 661c74cce4 docs: add contributor credit to v1.59.0 changelog 2026-03-09 00:58:14 -05:00
Jim Liu 宝玉 82445f2a39 chore: release v1.59.0 2026-03-09 00:55:52 -05:00
Jim Liu 宝玉 df9ce95c46 fix: restore Google as default provider when multiple keys available
EXTEND.md default_provider still takes priority over auto-detect.
Priority: CLI --provider > EXTEND.md default_provider > auto-detect (Google first).
2026-03-09 00:52:53 -05:00
Jim Liu 宝玉 88b433d565 docs: improve skill documentation clarity
- Fix Provider Selection: default to Replicate when multiple keys available (matches code)
- Add Replicate column to Quality Presets table (normal→1K, 2k→2K)
- Add Replicate aspect ratio behavior (match_input_image when --ref without --ar)
- Remove stale Google Imagen reference from Aspect Ratios
- Add batch file format example with JSON schema to SKILL.md
- Note that batch paths resolve relative to batch file directory
- Move batch execution strategy in article-illustrator before numbered steps
- Fix translate image-language reminder to use standard markdown syntax
  with note to match article's own image syntax convention
2026-03-09 00:43:05 -05:00
Jim Liu 宝玉 5acef7151b feat: add batch parallel image generation and provider-level throttling
- Add --batchfile and --jobs flags for multi-image parallel generation
  with per-provider concurrency control and rate limiting
- Refactor main.ts into prepareSingleTask/prepareBatchTasks/runBatchTasks
  with worker pool pattern and up to 3 retries per image
- Fix Replicate provider: use image_input array (nano-banana-pro schema),
  add match_input_image aspect ratio, add quality-to-resolution mapping
- Improve OpenAI error message for missing API key (Codex auth hint)
- Expand non-retryable error detection (4xx codes, disabled models)
- Add batch config to EXTEND.md schema (max_workers, provider_limits)
- Add build-batch.ts for article-illustrator batch workflow integration
- Add image-language awareness pass to baoyu-translate

Co-authored-by: 敖氏 <aoshi@MacBook-Air.local>
2026-03-09 00:07:45 -05:00
72 changed files with 6372 additions and 2268 deletions
+1 -1
View File
@@ -6,7 +6,7 @@
}, },
"metadata": { "metadata": {
"description": "Skills shared by Baoyu for improving daily work efficiency", "description": "Skills shared by Baoyu for improving daily work efficiency",
"version": "1.58.0" "version": "1.60.0"
}, },
"plugins": [ "plugins": [
{ {
+29
View File
@@ -35,6 +35,7 @@ Just run `/release-skills` - auto-detects your project configuration.
### Step 1: Detect Project Configuration ### Step 1: Detect Project Configuration
1. Check for `.releaserc.yml` (optional config override) 1. Check for `.releaserc.yml` (optional config override)
- If present, inspect whether it defines release hooks
2. Auto-detect version file by scanning (priority order): 2. Auto-detect version file by scanning (priority order):
- `package.json` (Node.js) - `package.json` (Node.js)
- `pyproject.toml` (Python) - `pyproject.toml` (Python)
@@ -48,6 +49,34 @@ Just run `/release-skills` - auto-detects your project configuration.
4. Identify language of each changelog by filename suffix 4. Identify language of each changelog by filename suffix
5. Display detected configuration 5. Display detected configuration
**Project Hook Contract**:
If `.releaserc.yml` defines `release.hooks`, keep the release workflow generic and delegate project-specific packaging/publishing to those hooks.
Supported hooks:
| Hook | Purpose | Expected Responsibility |
|------|---------|-------------------------|
| `prepare_artifact` | Make one target releasable | Validate the target is self-contained, sync/embed local dependencies, optionally stage extra files |
| `publish_artifact` | Publish one releasable target | Upload the prepared target (or a staged directory if the project uses one), attach version/changelog/tags |
Supported placeholders:
| Placeholder | Meaning |
|-------------|---------|
| `{project_root}` | Absolute path to repository root |
| `{target}` | Absolute path to the module/skill being released |
| `{artifact_dir}` | Absolute path to a temporary staging directory for this target, when the project uses one |
| `{version}` | Version selected by the release workflow |
| `{dry_run}` | `true` or `false` |
| `{release_notes_file}` | Absolute path to a UTF-8 file containing release notes/changelog text |
Execution rules:
- Keep the skill generic: do not hardcode registry/package-manager/project layout details into this SKILL.
- If `prepare_artifact` exists, run it once per target before publish-related checks that need the final releasable target state.
- Write release notes to a temp file and pass that file path to `publish_artifact`; do not inline multiline changelog text into shell commands.
- If hooks are absent, fall back to the default project-agnostic release workflow.
**Language Detection Rules**: **Language Detection Rules**:
Changelog files follow the pattern `CHANGELOG_{LANG}.md` or `CHANGELOG.{lang}.md`, where `{lang}` / `{LANG}` is a language or region code. Changelog files follow the pattern `CHANGELOG_{LANG}.md` or `CHANGELOG.{lang}.md`, where `{lang}` / `{LANG}` is a language or region code.
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
REPO_ROOT=$(git rev-parse --show-toplevel)
cd "$REPO_ROOT"
node scripts/sync-shared-skill-packages.mjs --repo-root "$REPO_ROOT" --enforce-clean
+2
View File
@@ -164,3 +164,5 @@ posts/
# ClawHub local state (current and legacy directory names from the official CLI) # ClawHub local state (current and legacy directory names from the official CLI)
.clawhub/ .clawhub/
.clawdhub/ .clawdhub/
.release-artifacts/
.worktrees/
+6
View File
@@ -0,0 +1,6 @@
release:
target_globs:
- skills/*
hooks:
prepare_artifact: node scripts/sync-shared-skill-packages.mjs --repo-root "{project_root}" --target "{target}"
publish_artifact: node scripts/publish-skill.mjs --skill-dir "{target}" --version "{version}" --changelog-file "{release_notes_file}" --dry-run "{dry_run}"
+37
View File
@@ -2,6 +2,43 @@
English | [中文](./CHANGELOG.zh.md) English | [中文](./CHANGELOG.zh.md)
## 1.60.0 - 2026-03-11
### Features
- `baoyu-url-to-markdown`: support reusing existing Chrome CDP instances and fix port detection order
### Fixes
- `baoyu-post-to-x`: add missing `fs` import in x-article
### Refactor
- Unify all CDP skills to use shared `baoyu-chrome-cdp` package with vendored copies
- Simplify CLAUDE.md, move detailed documentation to `docs/` directory
- Publish skills directly from synced vendor, removing separate artifact preparation step
## 1.59.1 - 2026-03-11
### Fixes
- `baoyu-translate`: improve short text annotation density rule and add explicit style preset passing to 02-prompt.md
- `baoyu-post-to-x`: remove `--disable-blink-features=AutomationControlled` Chrome flag
### Refactor
- `baoyu-post-to-weibo`: add entry point guard to md-to-html.ts for module import compatibility
- Replace clawhub CLI with local sync-clawhub.mjs script
### Documentation
- Update CLAUDE.md to reflect v1.59.0 codebase state (by @jackL1020)
## 1.59.0 - 2026-03-09
### Features
- `baoyu-image-gen`: add batch parallel image generation and provider-level throttling (by @SeamoonAO)
### Fixes
- `baoyu-image-gen`: restore Google as default provider when multiple keys available
### Documentation
- Improve skill documentation clarity (by @SeamoonAO)
## 1.58.0 - 2026-03-08 ## 1.58.0 - 2026-03-08
### Features ### Features
+37
View File
@@ -2,6 +2,43 @@
[English](./CHANGELOG.md) | 中文 [English](./CHANGELOG.md) | 中文
## 1.60.0 - 2026-03-11
### 新功能
- `baoyu-url-to-markdown`:支持复用已有 Chrome CDP 实例,修复端口检测顺序问题
### 修复
- `baoyu-post-to-x`:补充 x-article 缺失的 `fs` 导入
### 重构
- 统一所有 CDP 技能使用共享 `baoyu-chrome-cdp` 包,各技能内置 vendor 副本
- 精简 CLAUDE.md,将详细文档移至 `docs/` 目录
- 从 synced vendor 直接发布技能,移除单独的 artifact 准备步骤
## 1.59.1 - 2026-03-11
### 修复
- `baoyu-translate`:改进短文本注释密度规则,补充风格预设到 02-prompt.md 的显式传递
- `baoyu-post-to-x`:移除 `--disable-blink-features=AutomationControlled` Chrome 启动参数
### 重构
- `baoyu-post-to-weibo`:为 md-to-html.ts 添加入口守卫,支持模块导入
- 使用本地 sync-clawhub.mjs 脚本替代 clawhub CLI
### 文档
- 更新 CLAUDE.md 以反映 v1.59.0 代码库状态 (by @jackL1020)
## 1.59.0 - 2026-03-09
### 新功能
- `baoyu-image-gen`:新增批量并行图片生成和提供商级别限流 (by @SeamoonAO)
### 修复
- `baoyu-image-gen`:修复多个 API key 可用时恢复 Google 为默认提供商
### 文档
- 改进技能文档清晰度 (by @SeamoonAO)
## 1.58.0 - 2026-03-08 ## 1.58.0 - 2026-03-08
### 新功能 ### 新功能
+37 -494
View File
@@ -1,533 +1,76 @@
# CLAUDE.md # CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Claude Code marketplace plugin providing AI-powered content generation skills. Version: **1.60.0**.
## Project Overview
Claude Code marketplace plugin providing AI-powered content generation skills. Skills use Gemini Web API (reverse-engineered) for text/image generation and Chrome CDP for browser automation.
## Architecture ## Architecture
Skills are organized into three plugin categories in `marketplace.json`: Skills organized into three categories in `.claude-plugin/marketplace.json` (defines plugin metadata, version, and skill paths):
```
skills/
├── [content-skills] # Content generation and publishing
│ ├── baoyu-xhs-images/ # Xiaohongshu infographic series (1-10 images)
│ ├── baoyu-cover-image/ # Article cover images (2.35:1 aspect)
│ ├── baoyu-slide-deck/ # Presentation slides with outlines
│ ├── baoyu-article-illustrator/ # Smart illustration placement
│ ├── baoyu-comic/ # Knowledge comics (Logicomix/Ohmsha style)
│ ├── baoyu-post-to-x/ # X/Twitter posting automation
│ └── baoyu-post-to-wechat/ # WeChat Official Account posting
├── [ai-generation-skills] # AI-powered generation backends
│ └── baoyu-danger-gemini-web/ # Gemini API wrapper (text + image gen)
└── [utility-skills] # Utility tools for content processing
├── baoyu-danger-x-to-markdown/ # X/Twitter content to markdown
└── baoyu-compress-image/ # Image compression
```
**Plugin Categories**:
| Category | Description | | Category | Description |
|----------|-------------| |----------|-------------|
| `content-skills` | Skills that generate or publish content (images, slides, comics, posts) | | `content-skills` | Generate or publish content (images, slides, comics, posts) |
| `ai-generation-skills` | Backend skills providing AI generation capabilities | | `ai-generation-skills` | AI generation backends |
| `utility-skills` | Helper tools for content processing (conversion, compression) | | `utility-skills` | Content processing (conversion, compression, translation) |
Each skill contains: Each skill contains `SKILL.md` (YAML front matter + docs), optional `scripts/`, `references/`, `prompts/`.
- `SKILL.md` - YAML front matter (name, description) + documentation
- `scripts/` - TypeScript implementations Top-level `scripts/` contains repo maintenance utilities (sync, hooks, publish).
- `prompts/system.md` - AI generation guidelines (optional)
## Running Skills ## Running Skills
All scripts are TypeScript, executed via Bun runtime (no build step). TypeScript via Bun (no build step). Detect runtime once per session:
### Runtime Detection (`${BUN_X}`)
Before running any script, the agent MUST detect the runtime **once per session** and set `${BUN_X}`:
```bash ```bash
# Detect runtime (run once, reuse result) if command -v bun &>/dev/null; then BUN_X="bun"
if command -v bun &>/dev/null; then elif command -v npx &>/dev/null; then BUN_X="npx -y bun"
BUN_X="bun" else echo "Error: install bun: brew install oven-sh/bun/bun or npm install -g bun"; exit 1; fi
elif command -v npx &>/dev/null; then
BUN_X="npx -y bun"
else
echo "Error: Neither bun nor npx found. Install bun: brew install oven-sh/bun/bun (macOS) or npm install -g bun"
exit 1
fi
``` ```
| Priority | Condition | `${BUN_X}` value | Notes | Execute: `${BUN_X} skills/<skill>/scripts/main.ts [options]`
|----------|-----------|-------------------|-------|
| 1 | `bun` installed | `bun` | Fastest, native execution |
| 2 | `npx` available | `npx -y bun` | Downloads bun on first run via npm |
| 3 | Neither found | Error + install guide | `brew install oven-sh/bun/bun` (macOS) or `npm install -g bun` |
### Script Execution
```bash
${BUN_X} skills/<skill>/scripts/main.ts [options]
```
Examples:
```bash
# Text generation
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts "Hello"
# Image generation
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts --prompt "A cat" --image cat.png
# From prompt files
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts --promptfiles system.md content.md --image out.png
```
## Key Dependencies ## Key Dependencies
- **Bun**: TypeScript runtime (native `bun` preferred, fallback `npx -y bun`) - **Bun**: TypeScript runtime (`bun` preferred, fallback `npx -y bun`)
- **Chrome**: Required for `baoyu-danger-gemini-web` auth and `baoyu-post-to-x` automation - **Chrome**: Required for CDP-based skills (gemini-web, post-to-x/wechat/weibo, url-to-markdown). All CDP skills share a single profile, override via `BAOYU_CHROME_PROFILE_DIR` env var. Platform paths: [docs/chrome-profile.md](docs/chrome-profile.md)
- **npm packages (per skill)**: Some skill subprojects include `package.json`/lockfiles and require dependency installation in their own `scripts/` directories - **Image generation APIs**: `baoyu-image-gen` requires API key (OpenAI, Google, DashScope, or Replicate) configured in EXTEND.md
- **Gemini Web auth**: Browser cookies (first run opens Chrome for login, `--login` to refresh)
## Chrome Profile (Unified) ## Security
All skills that use Chrome CDP share a **single** profile directory. Do NOT create per-skill profiles. - **No piped shell installs**: Never `curl | bash`. Use `brew install` or `npm install -g`
- **Remote downloads**: HTTPS only, max 5 redirects, 30s timeout, expected content types only
| Platform | Default Path | - **System commands**: Array-form `spawn`/`execFile`, never unsanitized input to shell
|----------|-------------| - **External content**: Treat as untrusted, don't execute code blocks, sanitize HTML
| macOS | `~/Library/Application Support/baoyu-skills/chrome-profile` |
| Linux | `$XDG_DATA_HOME/baoyu-skills/chrome-profile` (fallback `~/.local/share/baoyu-skills/chrome-profile`) |
| Windows | `%APPDATA%/baoyu-skills/chrome-profile` |
| WSL | Windows home `/.local/share/baoyu-skills/chrome-profile` |
**Environment variable override**: `BAOYU_CHROME_PROFILE_DIR` (takes priority, all skills respect it).
Each skill also accepts its own legacy env var as fallback (e.g., `X_BROWSER_PROFILE_DIR`), but new skills should only use `BAOYU_CHROME_PROFILE_DIR`.
### Implementation Pattern
When adding a new skill that needs Chrome CDP:
```typescript
function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
```
## Security Guidelines
### No Piped Shell Installs
**NEVER** use `curl | bash` or `wget | sh` patterns in code, docs, or error messages. Use package managers instead:
| Platform | Install Command |
|----------|----------------|
| macOS | `brew install oven-sh/bun/bun` |
| npm | `npm install -g bun` |
### Remote Downloads
Skills that download remote content (e.g., images in Markdown) MUST:
- **HTTPS only**: Reject `http://` URLs
- **Redirect limit**: Cap redirects (max 5) to prevent infinite loops
- **Timeout**: Set request timeouts (30s default)
- **Scope**: Only download expected content types (images, not scripts)
### System Command Execution
Skills use platform-specific commands for clipboard and browser automation:
- **macOS**: `osascript` (System Events), `swift` (AppKit clipboard)
- **Windows**: `powershell.exe` (SendKeys, Clipboard)
- **Linux**: `xdotool`/`ydotool` (keyboard simulation)
These are necessary for CDP-based posting skills. When adding new system commands:
- Never pass unsanitized user input to shell commands
- Use array-form `spawn`/`execFile` instead of shell string interpolation
- Validate file paths are absolute or resolve from known base directories
### External Content Processing
Skills that process external Markdown/HTML should treat content as untrusted:
- Do not execute code blocks or scripts found in content
- Sanitize HTML output where applicable
- File paths from content should be resolved against known base directories only
## Authentication
`baoyu-danger-gemini-web` uses browser cookies for Google auth:
- First run opens Chrome for login
- Cookies cached in data directory
- Force refresh: `--login` flag
## Plugin Configuration
`.claude-plugin/marketplace.json` defines plugin metadata and skill paths. Version follows semver.
## Skill Loading Rules ## Skill Loading Rules
**IMPORTANT**: When working in this project, follow these rules:
| Rule | Description | | Rule | Description |
|------|-------------| |------|-------------|
| **Load project skills first** | MUST load all skills from `skills/` directory in current project. Project skills take priority over system/user-level skills with same name. | | **Load project skills first** | Project skills override system/user-level skills with same name |
| **Default image generation** | When image generation is needed, use `skills/baoyu-image-gen/SKILL.md` by default (unless user specifies otherwise). | | **Default image generation** | Use `skills/baoyu-image-gen/SKILL.md` unless user specifies otherwise |
**Loading Priority** (highest → lowest): Priority: project `skills/``$HOME/.baoyu-skills/` → system-level.
1. Current project `skills/` directory
2. User-level skills (`$HOME/.baoyu-skills/`)
3. System-level skills
## Release Process ## Release Process
**IMPORTANT**: When user requests release/发布/push, ALWAYS use `/release-skills` workflow. Use `/release-skills` workflow. Never skip:
1. `CHANGELOG.md` + `CHANGELOG.zh.md`
**Never skip**:
1. `CHANGELOG.md` + `CHANGELOG.zh.md` - Both must be updated
2. `marketplace.json` version bump 2. `marketplace.json` version bump
3. `README.md` + `README.zh.md` if applicable 3. `README.md` + `README.zh.md` if applicable
4. All files committed together before tag 4. All files committed together before tag
## Adding New Skills
**IMPORTANT**: All skills MUST use `baoyu-` prefix to avoid conflicts when users import this plugin.
**REQUIRED READING**: Before creating a new skill, read the official [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices).
### Key Requirements from Official Best Practices
| Requirement | Details |
|-------------|---------|
| **Concise is key** | Claude is smart—only add context it doesn't have. Challenge each token. |
| **name field** | Max 64 chars, lowercase letters/numbers/hyphens only, no "anthropic"/"claude" |
| **description field** | Max 1024 chars, non-empty, MUST be third person, include what + when to use |
| **SKILL.md body** | Keep under 500 lines; use separate files for additional content |
| **Naming convention** | Gerund form preferred (e.g., `processing-pdfs`), but `baoyu-` prefix required here |
| **References** | Keep one level deep from SKILL.md; avoid nested references |
| **No time-sensitive info** | Avoid dates/versions that become outdated |
### Steps
1. Create `skills/baoyu-<name>/SKILL.md` with YAML front matter
- Directory name: `baoyu-<name>`
- SKILL.md `name` field: `baoyu-<name>`
2. Add TypeScript in `skills/baoyu-<name>/scripts/`
3. Add prompt templates in `skills/baoyu-<name>/prompts/` if needed
4. **Choose the appropriate category** and register in `marketplace.json`:
- `content-skills`: For content generation/publishing (images, slides, posts)
- `ai-generation-skills`: For AI backend capabilities
- `utility-skills`: For helper tools (conversion, compression)
- If none fit, create a new category with descriptive name
5. **Add Script Directory section** to SKILL.md (see template below)
### Choosing a Category
| If your skill... | Use category |
|------------------|--------------|
| Generates visual content (images, slides, comics) | `content-skills` |
| Publishes to platforms (X, WeChat, etc.) | `content-skills` |
| Provides AI generation backend | `ai-generation-skills` |
| Converts or processes content | `utility-skills` |
| Compresses or optimizes files | `utility-skills` |
**Creating a new category**: If the skill doesn't fit existing categories, add a new plugin object to `marketplace.json` with:
- `name`: Descriptive kebab-case name (e.g., `analytics-skills`)
- `description`: Brief description of the category
- `skills`: Array with the skill path
### Writing Effective Descriptions
**MUST write in third person** (not "I can help you" or "You can use this"):
```yaml
# Good
description: Generates Xiaohongshu infographic series from content. Use when user asks for "小红书图片", "XHS images", or "RedNote infographics".
# Bad
description: I can help you create Xiaohongshu images
description: You can use this to generate XHS content
```
Include both **what** the skill does and **when** to use it (triggers/keywords).
### Script Directory Template
Every SKILL.md with scripts MUST include this section after Usage:
```markdown
## Script Directory
**Important**: All scripts are located in the `scripts/` subdirectory of this skill.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `${SKILL_DIR}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
|--------|---------|
| `scripts/main.ts` | Main entry point |
| `scripts/other.ts` | Other functionality |
```
When referencing scripts in workflow sections, use `${BUN_X} ${SKILL_DIR}/scripts/<name>.ts` so agents can resolve the correct runtime and path.
### Progressive Disclosure
For skills with extensive content, use separate reference files:
```
skills/baoyu-example/
├── SKILL.md # Main instructions (<500 lines)
├── references/
│ ├── styles.md # Style definitions (loaded as needed)
│ └── examples.md # Usage examples (loaded as needed)
└── scripts/
└── main.ts # Executable script
```
In SKILL.md, link to reference files (one level deep only):
```markdown
**Available styles**: See [references/styles.md](references/styles.md)
**Examples**: See [references/examples.md](references/examples.md)
```
## Code Style ## Code Style
- TypeScript throughout, no comments TypeScript, no comments, async/await, short variable names, type-safe interfaces.
- Async/await patterns
- Short variable names
- Type-safe interfaces
## Image Generation Guidelines ## Adding New Skills
Skills that require image generation MUST delegate to available image generation skills rather than implementing their own. All skills MUST use `baoyu-` prefix. Details: [docs/creating-skills.md](docs/creating-skills.md)
### Image Generation Skill Selection ## Reference Docs
**Default**: Use `skills/baoyu-image-gen/SKILL.md` (unless user specifies otherwise). | Topic | File |
|-------|------|
1. Read `skills/baoyu-image-gen/SKILL.md` for parameters and capabilities | Image generation guidelines | [docs/image-generation.md](docs/image-generation.md) |
2. If user explicitly requests a different skill, check `skills/` directory for alternatives | Chrome profile platform paths | [docs/chrome-profile.md](docs/chrome-profile.md) |
3. Only ask user to choose when they haven't specified and multiple viable options exist | Comic style maintenance | [docs/comic-style-maintenance.md](docs/comic-style-maintenance.md) |
| ClawHub/OpenClaw publishing | [docs/publishing.md](docs/publishing.md) |
### Generation Flow Template
Use this template when implementing image generation in skills:
```markdown
### Step N: Generate Images
**Skill Selection**:
1. Check available image generation skills (e.g., `baoyu-danger-gemini-web`)
2. Read selected skill's SKILL.md for parameter reference
3. If multiple skills available, ask user to choose
**Generation Flow**:
1. Call selected image generation skill with:
- Prompt file path (or inline prompt)
- Output image path
- Any skill-specific parameters (refer to skill's SKILL.md)
2. Generate images sequentially (one at a time)
3. After each image, output progress: "Generated X/N"
4. On failure, auto-retry once before reporting error
```
### Output Path Convention
Each session creates an independent directory. Even the same source file generates a new directory per session.
**Output Directory**:
```
<skill-suffix>/<topic-slug>/
```
- `<skill-suffix>`: Skill name suffix (e.g., `xhs-images`, `cover-image`, `slide-deck`, `comic`)
- `<topic-slug>`: Generated from content topic (2-4 words, kebab-case)
- Example: `xhs-images/ai-future-trends/`
**Slug Generation**:
1. Extract main topic from content (2-4 words, kebab-case)
2. Example: "Introduction to Machine Learning" → `intro-machine-learning`
**Conflict Resolution**:
If `<skill-suffix>/<topic-slug>/` already exists:
- Append timestamp: `<topic-slug>-YYYYMMDD-HHMMSS`
- Example: `ai-future` exists → `ai-future-20260118-143052`
**Source Files**:
- Copy all sources to `<skill-suffix>/<topic-slug>/` with naming: `source-{slug}.{ext}`
- Multiple sources supported: text, images, files from conversation
- Examples:
- `source-article.md` (main text content)
- `source-reference.png` (image from conversation)
- `source-data.csv` (additional file)
- Original source files remain unchanged
### Image Naming Convention
Image filenames MUST include meaningful slugs for readability:
**Format**: `NN-{type}-[slug].png`
- `NN`: Two-digit sequence number (01, 02, ...)
- `{type}`: Image type (cover, content, page, slide, illustration, etc.)
- `[slug]`: Descriptive kebab-case slug derived from content
**Examples**:
```
01-cover-ai-future.png
02-content-key-benefits.png
03-page-enigma-machine.png
04-slide-architecture-overview.png
```
**Slug Rules**:
- Derived from image purpose or content (kebab-case)
- Must be unique within the output directory
- 2-5 words, concise but descriptive
- When content changes significantly, update slug accordingly
### Best Practices
- Always read the image generation skill's SKILL.md before calling
- Pass parameters exactly as documented in the skill
- Handle failures gracefully with retry logic
- Provide clear progress feedback to user
## Style Maintenance (baoyu-comic)
When adding, updating, or deleting styles for `baoyu-comic`, follow this workflow:
### Adding a New Style
1. **Create style definition**: `skills/baoyu-comic/references/styles/<style-name>.md`
2. **Update SKILL.md**:
- Add style to `--style` options table
- Add auto-selection entry if applicable
3. **Generate showcase image**:
```bash
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts \
--prompt "A single comic book page in <style-name> style showing [appropriate scene]. Features: [style characteristics from style definition]. 3:4 portrait aspect ratio comic page." \
--image screenshots/comic-styles/<style-name>.png
```
4. **Compress to WebP**:
```bash
${BUN_X} skills/baoyu-compress-image/scripts/main.ts screenshots/comic-styles/<style-name>.png
```
5. **Update both READMEs** (`README.md` and `README.zh.md`):
- Add style to `--style` options
- Add row to style description table
- Add image to style preview grid
### Updating an Existing Style
1. **Update style definition**: `skills/baoyu-comic/references/styles/<style-name>.md`
2. **Regenerate showcase image** (if visual characteristics changed):
- Follow steps 3-4 from "Adding a New Style"
3. **Update READMEs** if description changed
### Deleting a Style
1. **Delete style definition**: `skills/baoyu-comic/references/styles/<style-name>.md`
2. **Delete showcase image**: `screenshots/comic-styles/<style-name>.webp`
3. **Update SKILL.md**:
- Remove from `--style` options
- Remove auto-selection entry
4. **Update both READMEs**:
- Remove from `--style` options
- Remove from style description table
- Remove from style preview grid
### Style Preview Grid Format
READMEs use 3-column tables for style previews:
```markdown
| | | |
|:---:|:---:|:---:|
| ![style1](./screenshots/comic-styles/style1.webp) | ![style2](./screenshots/comic-styles/style2.webp) | ![style3](./screenshots/comic-styles/style3.webp) |
| style1 | style2 | style3 |
```
## Extension Support
Every SKILL.md MUST include two parts for extension support:
### 1. Load Preferences Section (in Step 1 or as "Preferences" section)
For skills with workflows, add as Step 1.1. For utility skills, add as "Preferences (EXTEND.md)" section before Usage:
```markdown
**1.1 Load Preferences (EXTEND.md)**
Check EXTEND.md existence (priority order):
\`\`\`bash
# macOS, Linux, WSL, Git Bash
test -f .baoyu-skills/<skill-name>/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/<skill-name>/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/<skill-name>/EXTEND.md" && echo "user"
\`\`\`
\`\`\`powershell
# PowerShell (Windows)
if (Test-Path .baoyu-skills/<skill-name>/EXTEND.md) { "project" }
$xdg = if ($env:XDG_CONFIG_HOME) { $env:XDG_CONFIG_HOME } else { "$HOME/.config" }
if (Test-Path "$xdg/baoyu-skills/<skill-name>/EXTEND.md") { "xdg" }
if (Test-Path "$HOME/.baoyu-skills/<skill-name>/EXTEND.md") { "user" }
\`\`\`
┌────────────────────────────────────────────────────────┬──────────────────────────┐
│ Path │ Location │
├────────────────────────────────────────────────────────┼──────────────────────────┤
│ .baoyu-skills/<skill-name>/EXTEND.md │ Project directory │
├────────────────────────────────────────────────────────┼──────────────────────────┤
│ $XDG_CONFIG_HOME/baoyu-skills/<skill-name>/EXTEND.md │ XDG config (~/.config) │
├────────────────────────────────────────────────────────┼──────────────────────────┤
│ $HOME/.baoyu-skills/<skill-name>/EXTEND.md │ User home (legacy) │
└────────────────────────────────────────────────────────┴──────────────────────────┘
┌───────────┬───────────────────────────────────────────────────────────────────────────┐
│ Result │ Action │
├───────────┼───────────────────────────────────────────────────────────────────────────┤
│ Found │ Read, parse, display summary │
├───────────┼───────────────────────────────────────────────────────────────────────────┤
│ Not found │ Ask user with AskUserQuestion (see references/config/first-time-setup.md) │
└───────────┴───────────────────────────────────────────────────────────────────────────┘
**EXTEND.md Supports**: [List supported configuration options for this skill]
Schema: `references/config/preferences-schema.md`
```
### 2. Extension Support Section (at the end)
Simplified section that references the preferences section:
```markdown
## Extension Support
Custom configurations via EXTEND.md. See **Step 1.1** for paths and supported options.
```
Or for utility skills without workflow steps:
```markdown
## Extension Support
Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.
```
**Notes**:
- Replace `<skill-name>` with actual skill name (e.g., `baoyu-cover-image`)
- Use `$HOME` instead of `~` for cross-platform compatibility (macOS/Linux/WSL/PowerShell)
- `$XDG_CONFIG_HOME` defaults to `~/.config` when unset
- Use `test -f` (Bash) or `Test-Path` (PowerShell) for explicit file existence check
- ASCII tables for clear visual formatting
+27
View File
@@ -0,0 +1,27 @@
# Chrome Profile
All CDP skills share a single profile directory. Do NOT create per-skill profiles.
Override: `BAOYU_CHROME_PROFILE_DIR` env var (takes priority over all defaults).
| Platform | Default Path |
|----------|-------------|
| macOS | `~/Library/Application Support/baoyu-skills/chrome-profile` |
| Linux | `$XDG_DATA_HOME/baoyu-skills/chrome-profile` (fallback `~/.local/share/`) |
| Windows | `%APPDATA%/baoyu-skills/chrome-profile` |
| WSL | Windows home `/.local/share/baoyu-skills/chrome-profile` |
New skills: use `BAOYU_CHROME_PROFILE_DIR` only (not per-skill env vars like `X_BROWSER_PROFILE_DIR`).
## Implementation Pattern
```typescript
function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
```
+35
View File
@@ -0,0 +1,35 @@
# Style Maintenance (baoyu-comic)
## Adding a New Style
1. Create style definition: `skills/baoyu-comic/references/styles/<style-name>.md`
2. Update SKILL.md: add to `--style` options table + auto-selection entry
3. Generate showcase image:
```bash
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts \
--prompt "A single comic book page in <style-name> style showing [scene]. Features: [characteristics]. 3:4 portrait aspect ratio comic page." \
--image screenshots/comic-styles/<style-name>.png
```
4. Compress: `${BUN_X} skills/baoyu-compress-image/scripts/main.ts screenshots/comic-styles/<style-name>.png`
5. Update both READMEs (`README.md` + `README.zh.md`): add style to options, description table, preview grid
## Updating an Existing Style
1. Update style definition in `references/styles/`
2. Regenerate showcase image if visual characteristics changed (steps 3-4 above)
3. Update READMEs if description changed
## Deleting a Style
1. Delete style definition + showcase image (`.webp`)
2. Remove from SKILL.md `--style` options + auto-selection
3. Remove from both READMEs (options, description table, preview grid)
## Style Preview Grid Format
```markdown
| | | |
|:---:|:---:|:---:|
| ![style1](./screenshots/comic-styles/style1.webp) | ![style2](./screenshots/comic-styles/style2.webp) | ![style3](./screenshots/comic-styles/style3.webp) |
| style1 | style2 | style3 |
```
+135
View File
@@ -0,0 +1,135 @@
# Creating New Skills
**REQUIRED READING**: [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)
## Key Requirements
| Requirement | Details |
|-------------|---------|
| **Prefix** | All skills MUST use `baoyu-` prefix |
| **name field** | Max 64 chars, lowercase letters/numbers/hyphens only, no "anthropic"/"claude" |
| **description** | Max 1024 chars, third person, include what + when to use |
| **SKILL.md body** | Keep under 500 lines; use `references/` for additional content |
| **References** | One level deep from SKILL.md; avoid nested references |
## SKILL.md Frontmatter Template
```yaml
---
name: baoyu-<name>
description: <Third-person description. What it does + when to use it.>
version: <semver matching marketplace.json>
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-<name>
requires: # include only if skill has scripts
anyBins:
- bun
- npx
---
```
## Steps
1. Create `skills/baoyu-<name>/SKILL.md` with YAML front matter
2. Add TypeScript in `skills/baoyu-<name>/scripts/` (if applicable)
3. Add prompt templates in `skills/baoyu-<name>/prompts/` if needed
4. Register in `marketplace.json` under appropriate category
5. Add Script Directory section to SKILL.md if skill has scripts
6. Add openclaw metadata to frontmatter
## Category Selection
| If your skill... | Use category |
|------------------|--------------|
| Generates visual content (images, slides, comics) | `content-skills` |
| Publishes to platforms (X, WeChat, Weibo) | `content-skills` |
| Provides AI generation backend | `ai-generation-skills` |
| Converts or processes content | `utility-skills` |
New category: add plugin object to `marketplace.json` with `name`, `description`, `skills[]`.
## Writing Descriptions
**MUST write in third person**:
```yaml
# Good
description: Generates Xiaohongshu infographic series from content. Use when user asks for "小红书图片", "XHS images".
# Bad
description: I can help you create Xiaohongshu images
```
## Script Directory Template
Every SKILL.md with scripts MUST include:
```markdown
## Script Directory
**Important**: All scripts are located in the `scripts/` subdirectory of this skill.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `{baseDir}`
2. Script path = `{baseDir}/scripts/<script-name>.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `{baseDir}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
|--------|---------|
| `scripts/main.ts` | Main entry point |
```
## Progressive Disclosure
For skills with extensive content:
```
skills/baoyu-example/
├── SKILL.md # Main instructions (<500 lines)
├── references/
│ ├── styles.md # Loaded as needed
│ └── examples.md # Loaded as needed
└── scripts/
└── main.ts
```
Link from SKILL.md (one level deep only):
```markdown
**Available styles**: See [references/styles.md](references/styles.md)
```
## Extension Support (EXTEND.md)
Every SKILL.md MUST include EXTEND.md loading. Add as Step 1.1 (workflow skills) or "Preferences" section (utility skills):
```markdown
**1.1 Load Preferences (EXTEND.md)**
Check EXTEND.md existence (priority order):
\`\`\`bash
test -f .baoyu-skills/<skill-name>/EXTEND.md && echo "project"
test -f "${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/<skill-name>/EXTEND.md" && echo "xdg"
test -f "$HOME/.baoyu-skills/<skill-name>/EXTEND.md" && echo "user"
\`\`\`
| Path | Location |
|------|----------|
| `.baoyu-skills/<skill-name>/EXTEND.md` | Project directory |
| `$XDG_CONFIG_HOME/baoyu-skills/<skill-name>/EXTEND.md` | XDG config (~/.config) |
| `$HOME/.baoyu-skills/<skill-name>/EXTEND.md` | User home (legacy) |
| Result | Action |
|--------|--------|
| Found | Read, parse, display summary |
| Not found | Ask user with AskUserQuestion |
```
End of SKILL.md should include:
```markdown
## Extension Support
Custom configurations via EXTEND.md. See **Step 1.1** for paths and supported options.
```
+53
View File
@@ -0,0 +1,53 @@
# Image Generation Guidelines
Skills that require image generation MUST delegate to available image generation skills.
## Skill Selection
**Default**: `skills/baoyu-image-gen/SKILL.md` (unless user specifies otherwise).
1. Read skill's SKILL.md for parameters and capabilities
2. If user requests different skill, check `skills/` for alternatives
3. Only ask user when multiple viable options exist
## Generation Flow Template
```markdown
### Step N: Generate Images
**Skill Selection**:
1. Check available skills (`baoyu-image-gen` default, or `baoyu-danger-gemini-web`)
2. Read selected skill's SKILL.md for parameters
3. If multiple skills available, ask user to choose
**Generation Flow**:
1. Call skill with prompt, output path, and skill-specific parameters
2. Generate sequentially by default (batch parallel only when user has multiple prompts)
3. Output progress: "Generated X/N"
4. On failure, auto-retry once before reporting error
```
**Batch Parallel** (`baoyu-image-gen` only): concurrent workers with per-provider throttling via `batch.max_workers` in EXTEND.md.
## Output Path Convention
**Output Directory**: `<skill-suffix>/<topic-slug>/`
- `<skill-suffix>`: e.g., `xhs-images`, `cover-image`, `slide-deck`, `comic`
- `<topic-slug>`: 2-4 words, kebab-case from content topic
- Conflict: append timestamp `<topic-slug>-YYYYMMDD-HHMMSS`
**Source Files**: Copy to output dir as `source-{slug}.{ext}`
## Image Naming Convention
**Format**: `NN-{type}-[slug].png`
- `NN`: Two-digit sequence (01, 02, ...)
- `{type}`: cover, content, page, slide, illustration, etc.
- `[slug]`: 2-5 word kebab-case descriptor, unique within directory
Examples:
```
01-cover-ai-future.png
02-content-key-benefits.png
03-slide-architecture-overview.png
```
+39
View File
@@ -0,0 +1,39 @@
# ClawHub / OpenClaw Publishing
## OpenClaw Metadata
Skills include `metadata.openclaw` in YAML front matter:
```yaml
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#<skill-name>
requires: # only for skills with scripts
anyBins:
- bun
- npx
```
## Publishing Commands
```bash
bash scripts/sync-clawhub.sh # sync all skills
bash scripts/sync-clawhub.sh <skill> # sync one skill
```
Release hooks are configured via `.releaserc.yml`. This repo does not stage a separate release directory: release prep only syncs `packages/` into each skill's committed `scripts/vendor/`, and publish reads the skill directory directly.
## Shared Workspace Packages
`packages/` is the **only** source of truth. Never edit `skills/*/scripts/vendor/` directly.
Current package: `baoyu-chrome-cdp` (Chrome CDP utilities), consumed by 6 skills (`baoyu-danger-gemini-web`, `baoyu-danger-x-to-markdown`, `baoyu-post-to-wechat`, `baoyu-post-to-weibo`, `baoyu-post-to-x`, `baoyu-url-to-markdown`).
**How it works**: Sync script copies packages into each consuming skill's `vendor/` directory and rewrites dependency specs to `file:./vendor/<name>`. Vendor copies are committed to git, making skills self-contained.
**Update workflow**:
1. Edit package under `packages/`
2. Run `node scripts/sync-shared-skill-packages.mjs`
3. Commit synced `vendor/`, `package.json`, and `bun.lock` together
**Git hook**: Run `node scripts/install-git-hooks.mjs` once to enable the `pre-push` hook. It re-syncs and blocks push if vendor copies are stale (`--enforce-clean`).
+9
View File
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
+408
View File
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env node
import { spawnSync } from "node:child_process";
import path from "node:path";
async function main() {
const repoRoot = path.resolve(process.cwd());
const hooksPath = path.join(repoRoot, ".githooks");
const result = spawnSync("git", ["config", "core.hooksPath", hooksPath], {
cwd: repoRoot,
stdio: "inherit",
});
if (result.status !== 0) {
throw new Error("Failed to configure core.hooksPath");
}
console.log(`Configured git hooks path: ${hooksPath}`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
+75
View File
@@ -0,0 +1,75 @@
import fs from "node:fs/promises";
import path from "node:path";
const PACKAGE_DEPENDENCY_SECTIONS = [
"dependencies",
"optionalDependencies",
"peerDependencies",
"devDependencies",
];
const SKIPPED_DIRS = new Set([".git", ".clawhub", ".clawdhub", "node_modules"]);
const SKIPPED_FILES = new Set([".DS_Store"]);
export async function listReleaseFiles(root) {
const resolvedRoot = path.resolve(root);
const files = [];
async function walk(folder) {
const entries = await fs.readdir(folder, { withFileTypes: true });
for (const entry of entries) {
if (entry.isDirectory() && SKIPPED_DIRS.has(entry.name)) continue;
if (entry.isFile() && SKIPPED_FILES.has(entry.name)) continue;
const fullPath = path.join(folder, entry.name);
if (entry.isDirectory()) {
await walk(fullPath);
continue;
}
if (!entry.isFile()) continue;
const relPath = path.relative(resolvedRoot, fullPath).split(path.sep).join("/");
const bytes = await fs.readFile(fullPath);
files.push({ relPath, bytes });
}
}
await walk(resolvedRoot);
files.sort((left, right) => left.relPath.localeCompare(right.relPath));
return files;
}
export async function validateSelfContainedRelease(root) {
const files = await listReleaseFiles(root);
for (const file of files.filter((entry) => path.posix.basename(entry.relPath) === "package.json")) {
const packageDir = path.resolve(root, fromPosixRel(path.posix.dirname(file.relPath)));
const packageJson = JSON.parse(file.bytes.toString("utf8"));
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
if (!dependencies || typeof dependencies !== "object") continue;
for (const [name, spec] of Object.entries(dependencies)) {
if (typeof spec !== "string" || !spec.startsWith("file:")) continue;
const targetDir = path.resolve(packageDir, spec.slice(5));
if (!isWithinRoot(root, targetDir)) {
throw new Error(
`Release target is not self-contained: ${file.relPath} depends on ${name} via ${spec}`,
);
}
await fs.access(targetDir).catch(() => {
throw new Error(`Missing local dependency for release: ${file.relPath} -> ${spec}`);
});
}
}
}
}
function fromPosixRel(relPath) {
return relPath === "." ? "." : relPath.split("/").join(path.sep);
}
function isWithinRoot(root, target) {
const resolvedRoot = path.resolve(root);
const relative = path.relative(resolvedRoot, path.resolve(target));
return relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative));
}
+280
View File
@@ -0,0 +1,280 @@
import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
const PACKAGE_DEPENDENCY_SECTIONS = [
"dependencies",
"optionalDependencies",
"peerDependencies",
"devDependencies",
];
const SKIPPED_DIRS = new Set([".git", ".clawhub", ".clawdhub", "node_modules"]);
const SKIPPED_FILES = new Set([".DS_Store"]);
export async function syncSharedSkillPackages(repoRoot, options = {}) {
const root = path.resolve(repoRoot);
const workspacePackages = await discoverWorkspacePackages(root);
const targetConsumerDirs = normalizeTargetConsumerDirs(root, options.targets ?? []);
const consumers = await discoverSkillScriptPackages(root, targetConsumerDirs);
const runtime = options.install === false ? null : resolveBunRuntime();
const managedPaths = new Set();
const packageDirs = [];
for (const consumer of consumers) {
const result = await syncConsumerPackage({
consumer,
root,
workspacePackages,
runtime,
});
if (!result) continue;
packageDirs.push(consumer.dir);
for (const managedPath of result.managedPaths) {
managedPaths.add(managedPath);
}
}
return {
packageDirs,
managedPaths: [...managedPaths].sort(),
};
}
function normalizeTargetConsumerDirs(repoRoot, targets) {
if (!targets || targets.length === 0) return null;
const consumerDirs = new Set();
for (const target of targets) {
if (!target) continue;
const resolvedTarget = path.resolve(repoRoot, target);
if (path.basename(resolvedTarget) === "scripts") {
consumerDirs.add(resolvedTarget);
continue;
}
consumerDirs.add(path.join(resolvedTarget, "scripts"));
}
return consumerDirs;
}
export function ensureManagedPathsClean(repoRoot, managedPaths) {
if (managedPaths.length === 0) return;
const result = spawnSync("git", ["status", "--porcelain", "--", ...managedPaths], {
cwd: repoRoot,
encoding: "utf8",
stdio: ["ignore", "pipe", "pipe"],
});
if (result.status !== 0) {
throw new Error(result.stderr.trim() || "Failed to inspect git status for managed paths");
}
const output = result.stdout.trim();
if (!output) return;
throw new Error(
[
"Shared skill package sync produced uncommitted managed changes.",
"Review and commit these files before pushing:",
output,
].join("\n"),
);
}
async function syncConsumerPackage({ consumer, root, workspacePackages, runtime }) {
const packageJsonPath = path.join(consumer.dir, "package.json");
const packageJson = JSON.parse(await fs.readFile(packageJsonPath, "utf8"));
const localDeps = collectLocalDependencies(packageJson, workspacePackages);
if (localDeps.length === 0) {
return null;
}
const vendorRoot = path.join(consumer.dir, "vendor");
await fs.rm(vendorRoot, { recursive: true, force: true });
for (const name of localDeps) {
const sourceDir = workspacePackages.get(name);
if (!sourceDir) continue;
await syncPackageTree({
sourceDir,
targetDir: path.join(vendorRoot, name),
workspacePackages,
});
}
rewriteLocalDependencySpecs(packageJson, localDeps);
await writeJson(packageJsonPath, packageJson);
if (runtime) {
runInstall(runtime, consumer.dir);
}
const managedPaths = [
path.relative(root, packageJsonPath).split(path.sep).join("/"),
path.relative(root, path.join(consumer.dir, "bun.lock")).split(path.sep).join("/"),
path.relative(root, vendorRoot).split(path.sep).join("/"),
];
return { managedPaths };
}
async function syncPackageTree({ sourceDir, targetDir, workspacePackages }) {
await fs.rm(targetDir, { recursive: true, force: true });
await fs.mkdir(targetDir, { recursive: true });
const sourcePackageJsonPath = path.join(sourceDir, "package.json");
const packageJson = JSON.parse(await fs.readFile(sourcePackageJsonPath, "utf8"));
const localDeps = collectLocalDependencies(packageJson, workspacePackages);
const entries = await fs.readdir(sourceDir, { withFileTypes: true });
for (const entry of entries) {
if (SKIPPED_DIRS.has(entry.name) || SKIPPED_FILES.has(entry.name)) continue;
const sourcePath = path.join(sourceDir, entry.name);
const targetPath = path.join(targetDir, entry.name);
if (entry.isDirectory()) {
await copyDirectory(sourcePath, targetPath);
continue;
}
if (!entry.isFile() || entry.name === "package.json") continue;
await fs.mkdir(path.dirname(targetPath), { recursive: true });
await fs.copyFile(sourcePath, targetPath);
}
for (const name of localDeps) {
const nestedSourceDir = workspacePackages.get(name);
if (!nestedSourceDir) continue;
await syncPackageTree({
sourceDir: nestedSourceDir,
targetDir: path.join(targetDir, "vendor", name),
workspacePackages,
});
}
rewriteLocalDependencySpecs(packageJson, localDeps);
await writeJson(path.join(targetDir, "package.json"), packageJson);
}
async function copyDirectory(sourceDir, targetDir) {
await fs.mkdir(targetDir, { recursive: true });
const entries = await fs.readdir(sourceDir, { withFileTypes: true });
for (const entry of entries) {
if (SKIPPED_DIRS.has(entry.name) || SKIPPED_FILES.has(entry.name)) continue;
const sourcePath = path.join(sourceDir, entry.name);
const targetPath = path.join(targetDir, entry.name);
if (entry.isDirectory()) {
await copyDirectory(sourcePath, targetPath);
continue;
}
if (!entry.isFile()) continue;
await fs.mkdir(path.dirname(targetPath), { recursive: true });
await fs.copyFile(sourcePath, targetPath);
}
}
async function discoverWorkspacePackages(repoRoot) {
const packagesRoot = path.join(repoRoot, "packages");
const map = new Map();
if (!existsSync(packagesRoot)) return map;
const entries = await fs.readdir(packagesRoot, { withFileTypes: true });
for (const entry of entries) {
if (!entry.isDirectory()) continue;
const packageJsonPath = path.join(packagesRoot, entry.name, "package.json");
if (!existsSync(packageJsonPath)) continue;
const packageJson = JSON.parse(await fs.readFile(packageJsonPath, "utf8"));
if (!packageJson.name) continue;
map.set(packageJson.name, path.join(packagesRoot, entry.name));
}
return map;
}
async function discoverSkillScriptPackages(repoRoot, targetConsumerDirs = null) {
const skillsRoot = path.join(repoRoot, "skills");
const consumers = [];
const skillEntries = await fs.readdir(skillsRoot, { withFileTypes: true });
for (const entry of skillEntries) {
if (!entry.isDirectory()) continue;
const scriptsDir = path.join(skillsRoot, entry.name, "scripts");
if (targetConsumerDirs && !targetConsumerDirs.has(path.resolve(scriptsDir))) continue;
const packageJsonPath = path.join(scriptsDir, "package.json");
if (!existsSync(packageJsonPath)) continue;
consumers.push({ dir: scriptsDir, packageJsonPath });
}
return consumers.sort((left, right) => left.dir.localeCompare(right.dir));
}
function collectLocalDependencies(packageJson, workspacePackages) {
const localDeps = [];
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
if (!dependencies || typeof dependencies !== "object") continue;
for (const name of Object.keys(dependencies)) {
if (!workspacePackages.has(name)) continue;
localDeps.push(name);
}
}
return [...new Set(localDeps)].sort();
}
function rewriteLocalDependencySpecs(packageJson, localDeps) {
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
const dependencies = packageJson[section];
if (!dependencies || typeof dependencies !== "object") continue;
for (const name of localDeps) {
if (!(name in dependencies)) continue;
dependencies[name] = `file:./vendor/${name}`;
}
}
}
async function writeJson(filePath, value) {
await fs.mkdir(path.dirname(filePath), { recursive: true });
await fs.writeFile(filePath, `${JSON.stringify(value, null, 2)}\n`);
}
function resolveBunRuntime() {
if (commandExists("bun")) {
return { command: "bun", args: [] };
}
if (commandExists("npx")) {
return { command: "npx", args: ["-y", "bun"] };
}
throw new Error(
"Neither bun nor npx is installed. Install bun with `brew install oven-sh/bun/bun` or `npm install -g bun`.",
);
}
function commandExists(command) {
const result = spawnSync("sh", ["-lc", `command -v ${command}`], {
stdio: "ignore",
});
return result.status === 0;
}
function runInstall(runtime, cwd) {
const result = spawnSync(runtime.command, [...runtime.args, "install"], {
cwd,
stdio: "inherit",
});
if (result.status !== 0) {
throw new Error(`Failed to refresh Bun dependencies in ${cwd}`);
}
}
+291
View File
@@ -0,0 +1,291 @@
#!/usr/bin/env node
import fs from "node:fs/promises";
import { existsSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { listReleaseFiles, validateSelfContainedRelease } from "./lib/release-files.mjs";
const DEFAULT_REGISTRY = "https://clawhub.ai";
async function main() {
const options = parseArgs(process.argv.slice(2));
if (!options.skillDir || !options.version) {
throw new Error("--skill-dir and --version are required");
}
const skillDir = path.resolve(options.skillDir);
const skill = buildSkillEntry(skillDir, options.slug, options.displayName);
const changelog = options.changelogFile
? await fs.readFile(path.resolve(options.changelogFile), "utf8")
: "";
await validateSelfContainedRelease(skillDir);
const files = await listReleaseFiles(skillDir);
if (files.length === 0) {
throw new Error(`Skill directory is empty: ${skillDir}`);
}
if (options.dryRun) {
console.log(`Dry run: would publish ${skill.slug}@${options.version}`);
console.log(`Skill: ${skillDir}`);
console.log(`Files: ${files.length}`);
return;
}
const config = await readClawhubConfig();
const registry = (
options.registry ||
process.env.CLAWHUB_REGISTRY ||
process.env.CLAWDHUB_REGISTRY ||
config.registry ||
DEFAULT_REGISTRY
).replace(/\/+$/, "");
if (!config.token) {
throw new Error("Not logged in. Run: clawhub login");
}
await apiJson(registry, config.token, "/api/v1/whoami");
const tags = options.tags
.split(",")
.map((tag) => tag.trim())
.filter(Boolean);
await publishSkill({
registry,
token: config.token,
skill,
files,
version: options.version,
changelog,
tags,
});
}
function parseArgs(argv) {
const options = {
skillDir: "",
version: "",
changelogFile: "",
registry: "",
tags: "latest",
dryRun: false,
slug: "",
displayName: "",
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--skill-dir") {
options.skillDir = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--version") {
options.version = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--changelog-file") {
options.changelogFile = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--registry") {
options.registry = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--tags") {
options.tags = argv[index + 1] ?? "latest";
index += 1;
continue;
}
if (arg === "--slug") {
options.slug = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--display-name") {
options.displayName = argv[index + 1] ?? "";
index += 1;
continue;
}
if (arg === "--dry-run") {
const next = argv[index + 1];
if (next && !next.startsWith("-")) {
options.dryRun = parseBoolean(next);
index += 1;
} else {
options.dryRun = true;
}
continue;
}
if (arg === "-h" || arg === "--help") {
printUsage();
process.exit(0);
}
throw new Error(`Unknown argument: ${arg}`);
}
return options;
}
function printUsage() {
console.log(`Usage: publish-skill.mjs --skill-dir <dir> --version <semver> [options]
Options:
--skill-dir <dir> Skill directory to publish
--version <semver> Version to publish
--changelog-file <file> Release notes file
--registry <url> Override registry base URL
--tags <tags> Comma-separated tags (default: latest)
--slug <value> Override slug
--display-name <value> Override display name
--dry-run Print publish plan without network calls
-h, --help Show help`);
}
function buildSkillEntry(folder, slugOverride, displayNameOverride) {
const base = path.basename(folder);
return {
folder,
slug: slugOverride || sanitizeSlug(base),
displayName: displayNameOverride || titleCase(base),
};
}
async function readClawhubConfig() {
const configPath = getConfigPath();
try {
return JSON.parse(await fs.readFile(configPath, "utf8"));
} catch {
return {};
}
}
function getConfigPath() {
const override =
process.env.CLAWHUB_CONFIG_PATH?.trim() || process.env.CLAWDHUB_CONFIG_PATH?.trim();
if (override) {
return path.resolve(override);
}
const home = os.homedir();
if (process.platform === "darwin") {
const clawhub = path.join(home, "Library", "Application Support", "clawhub", "config.json");
const clawdhub = path.join(home, "Library", "Application Support", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const xdg = process.env.XDG_CONFIG_HOME;
if (xdg) {
const clawhub = path.join(xdg, "clawhub", "config.json");
const clawdhub = path.join(xdg, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
if (process.platform === "win32" && process.env.APPDATA) {
const clawhub = path.join(process.env.APPDATA, "clawhub", "config.json");
const clawdhub = path.join(process.env.APPDATA, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const clawhub = path.join(home, ".config", "clawhub", "config.json");
const clawdhub = path.join(home, ".config", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
function pathForExistingConfig(primary, legacy) {
if (existsSync(primary)) return path.resolve(primary);
if (existsSync(legacy)) return path.resolve(legacy);
return path.resolve(primary);
}
async function publishSkill({ registry, token, skill, files, version, changelog, tags }) {
const form = new FormData();
form.set(
"payload",
JSON.stringify({
slug: skill.slug,
displayName: skill.displayName,
version,
changelog,
tags,
acceptLicenseTerms: true,
}),
);
for (const file of files) {
form.append("files", new Blob([file.bytes], { type: "application/octet-stream" }), file.relPath);
}
const response = await fetch(`${registry}/api/v1/skills`, {
method: "POST",
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
body: form,
});
const text = await response.text();
if (!response.ok) {
throw new Error(text || `Publish failed for ${skill.slug} (HTTP ${response.status})`);
}
const result = text ? JSON.parse(text) : {};
console.log(`OK. Published ${skill.slug}@${version}${result.versionId ? ` (${result.versionId})` : ""}`);
}
async function apiJson(registry, token, requestPath) {
const response = await fetch(`${registry}${requestPath}`, {
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
});
const text = await response.text();
let body = null;
try {
body = text ? JSON.parse(text) : null;
} catch {
body = { message: text };
}
if (response.status < 200 || response.status >= 300) {
throw new Error(body?.message || `HTTP ${response.status}`);
}
return body;
}
function sanitizeSlug(value) {
return value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+/, "")
.replace(/-+$/, "")
.replace(/--+/g, "-");
}
function titleCase(value) {
return value
.trim()
.replace(/[-_]+/g, " ")
.replace(/\s+/g, " ")
.replace(/\b\w/g, (char) => char.toUpperCase());
}
function parseBoolean(value) {
return ["1", "true", "yes", "on"].includes(String(value).trim().toLowerCase());
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
+528
View File
@@ -0,0 +1,528 @@
#!/usr/bin/env node
import crypto from "node:crypto";
import fs from "node:fs/promises";
import { existsSync } from "node:fs";
import path from "node:path";
import os from "node:os";
const DEFAULT_REGISTRY = "https://clawhub.ai";
const TEXT_EXTENSIONS = new Set([
"md",
"mdx",
"txt",
"json",
"json5",
"yaml",
"yml",
"toml",
"js",
"cjs",
"mjs",
"ts",
"tsx",
"jsx",
"py",
"sh",
"rb",
"go",
"rs",
"swift",
"kt",
"java",
"cs",
"cpp",
"c",
"h",
"hpp",
"sql",
"csv",
"ini",
"cfg",
"env",
"xml",
"html",
"css",
"scss",
"sass",
"svg",
]);
async function main() {
const options = parseArgs(process.argv.slice(2));
const config = await readClawhubConfig();
const registry = (
process.env.CLAWHUB_REGISTRY ||
process.env.CLAWDHUB_REGISTRY ||
config.registry ||
DEFAULT_REGISTRY
).replace(/\/+$/, "");
if (!config.token) {
throw new Error("Not logged in. Run: clawhub login");
}
await apiJson(registry, config.token, "/api/v1/whoami");
const roots = options.roots.length > 0 ? options.roots : [path.resolve("skills")];
const skills = await findSkills(roots);
if (skills.length === 0) {
throw new Error("No skills found.");
}
console.log("ClawHub sync");
console.log(`Roots with skills: ${roots.join(", ")}`);
const locals = await mapWithConcurrency(skills, options.concurrency, async (skill) => {
const files = await listTextFiles(skill.folder);
const fingerprint = buildFingerprint(files);
return {
...skill,
fileCount: files.length,
fingerprint,
};
});
const candidates = await mapWithConcurrency(locals, options.concurrency, async (skill) => {
const query = new URLSearchParams({
slug: skill.slug,
hash: skill.fingerprint,
});
const { status, body } = await apiJsonWithStatus(
registry,
config.token,
`/api/v1/resolve?${query.toString()}`
);
if (status === 404) {
return {
...skill,
status: "new",
latestVersion: null,
matchVersion: null,
};
}
if (status !== 200) {
throw new Error(body?.message || `Resolve failed for ${skill.slug} (HTTP ${status})`);
}
const latestVersion = body?.latestVersion?.version ?? null;
const matchVersion = body?.match?.version ?? null;
if (!latestVersion) {
return {
...skill,
status: "new",
latestVersion: null,
matchVersion: null,
};
}
return {
...skill,
status: matchVersion ? "synced" : "update",
latestVersion,
matchVersion,
};
});
const actionable = candidates.filter((candidate) => candidate.status !== "synced");
if (actionable.length === 0) {
console.log("Nothing to sync.");
return;
}
console.log("");
console.log("To sync");
for (const candidate of actionable) {
console.log(`- ${formatCandidate(candidate, options.bump)}`);
}
if (options.dryRun) {
console.log("");
console.log(`Dry run: would upload ${actionable.length} skill(s).`);
return;
}
const tags = options.tags
.split(",")
.map((tag) => tag.trim())
.filter(Boolean);
for (const candidate of actionable) {
const version =
candidate.status === "new"
? "1.0.0"
: bumpSemver(candidate.latestVersion, options.bump);
console.log(`Publishing ${candidate.slug}@${version}`);
const files = await listTextFiles(candidate.folder);
await publishSkill({
registry,
token: config.token,
skill: candidate,
files,
version,
changelog: options.changelog,
tags,
});
}
console.log("");
console.log(`Uploaded ${actionable.length} skill(s).`);
}
function parseArgs(argv) {
const options = {
roots: [],
dryRun: false,
bump: "patch",
changelog: "",
tags: "latest",
concurrency: 4,
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--dry-run") {
options.dryRun = true;
continue;
}
if (arg === "--all") {
continue;
}
if (arg === "--root") {
const value = argv[index + 1];
if (!value) throw new Error("--root requires a directory");
options.roots.push(path.resolve(value));
index += 1;
continue;
}
if (arg === "--bump") {
const value = argv[index + 1];
if (!["patch", "minor", "major"].includes(value)) {
throw new Error("--bump must be patch, minor, or major");
}
options.bump = value;
index += 1;
continue;
}
if (arg === "--changelog") {
const value = argv[index + 1];
if (value == null) throw new Error("--changelog requires text");
options.changelog = value;
index += 1;
continue;
}
if (arg === "--tags") {
const value = argv[index + 1];
if (value == null) throw new Error("--tags requires a value");
options.tags = value;
index += 1;
continue;
}
if (arg === "--concurrency") {
const value = Number(argv[index + 1]);
if (!Number.isInteger(value) || value < 1 || value > 32) {
throw new Error("--concurrency must be an integer between 1 and 32");
}
options.concurrency = value;
index += 1;
continue;
}
if (arg === "-h" || arg === "--help") {
printUsage();
process.exit(0);
}
throw new Error(`Unknown argument: ${arg}`);
}
return options;
}
function printUsage() {
console.log(`Usage: sync-clawhub.mjs [options]
Options:
--root <dir> Extra skill root (repeatable)
--all Accepted for compatibility
--dry-run Show what would be uploaded
--bump <type> patch | minor | major
--changelog <text> Changelog for updates
--tags <tags> Comma-separated tags
--concurrency <n> Registry check concurrency (1-32)
-h, --help Show help`);
}
async function readClawhubConfig() {
const configPath = getConfigPath();
try {
return JSON.parse(await fs.readFile(configPath, "utf8"));
} catch {
return {};
}
}
function getConfigPath() {
const override =
process.env.CLAWHUB_CONFIG_PATH?.trim() || process.env.CLAWDHUB_CONFIG_PATH?.trim();
if (override) {
return path.resolve(override);
}
const home = os.homedir();
if (process.platform === "darwin") {
const clawhub = path.join(home, "Library", "Application Support", "clawhub", "config.json");
const clawdhub = path.join(home, "Library", "Application Support", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const xdg = process.env.XDG_CONFIG_HOME;
if (xdg) {
const clawhub = path.join(xdg, "clawhub", "config.json");
const clawdhub = path.join(xdg, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
if (process.platform === "win32" && process.env.APPDATA) {
const clawhub = path.join(process.env.APPDATA, "clawhub", "config.json");
const clawdhub = path.join(process.env.APPDATA, "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
const clawhub = path.join(home, ".config", "clawhub", "config.json");
const clawdhub = path.join(home, ".config", "clawdhub", "config.json");
return pathForExistingConfig(clawhub, clawdhub);
}
function pathForExistingConfig(primary, legacy) {
if (existsSync(primary)) return path.resolve(primary);
if (existsSync(legacy)) return path.resolve(legacy);
return path.resolve(primary);
}
async function findSkills(roots) {
const deduped = new Map();
for (const root of roots) {
const folders = await findSkillFolders(root);
for (const folder of folders) {
deduped.set(folder.slug, folder);
}
}
return [...deduped.values()].sort((left, right) => left.slug.localeCompare(right.slug));
}
async function findSkillFolders(root) {
const stat = await safeStat(root);
if (!stat?.isDirectory()) return [];
if (await hasSkillMarker(root)) {
return [buildSkillEntry(root)];
}
const entries = await fs.readdir(root, { withFileTypes: true });
const found = [];
for (const entry of entries) {
if (!entry.isDirectory()) continue;
const folder = path.join(root, entry.name);
if (await hasSkillMarker(folder)) {
found.push(buildSkillEntry(folder));
}
}
return found;
}
function buildSkillEntry(folder) {
const base = path.basename(folder);
return {
folder,
slug: sanitizeSlug(base),
displayName: titleCase(base),
};
}
async function hasSkillMarker(folder) {
return Boolean(
(await safeStat(path.join(folder, "SKILL.md")))?.isFile() ||
(await safeStat(path.join(folder, "skill.md")))?.isFile()
);
}
async function listTextFiles(root) {
const files = [];
async function walk(folder) {
const entries = await fs.readdir(folder, { withFileTypes: true });
for (const entry of entries) {
if (entry.name.startsWith(".")) continue;
if (entry.name === "node_modules") continue;
if (entry.name === ".clawhub" || entry.name === ".clawdhub") continue;
const fullPath = path.join(folder, entry.name);
if (entry.isDirectory()) {
await walk(fullPath);
continue;
}
if (!entry.isFile()) continue;
const relPath = path.relative(root, fullPath).split(path.sep).join("/");
const ext = relPath.split(".").pop()?.toLowerCase() ?? "";
if (!TEXT_EXTENSIONS.has(ext)) continue;
const bytes = await fs.readFile(fullPath);
files.push({ relPath, bytes });
}
}
await walk(root);
files.sort((left, right) => left.relPath.localeCompare(right.relPath));
return files;
}
function buildFingerprint(files) {
const payload = files
.map((file) => `${file.relPath}:${sha256(file.bytes)}`)
.sort((left, right) => left.localeCompare(right))
.join("\n");
return crypto.createHash("sha256").update(payload).digest("hex");
}
function sha256(bytes) {
return crypto.createHash("sha256").update(bytes).digest("hex");
}
async function publishSkill({ registry, token, skill, files, version, changelog, tags }) {
const form = new FormData();
form.set(
"payload",
JSON.stringify({
slug: skill.slug,
displayName: skill.displayName,
version,
changelog,
tags,
acceptLicenseTerms: true,
})
);
for (const file of files) {
form.append("files", new Blob([file.bytes], { type: "text/plain" }), file.relPath);
}
const response = await fetch(`${registry}/api/v1/skills`, {
method: "POST",
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
body: form,
});
const text = await response.text();
if (!response.ok) {
throw new Error(text || `Publish failed for ${skill.slug} (HTTP ${response.status})`);
}
const result = text ? JSON.parse(text) : {};
console.log(`OK. Published ${skill.slug}@${version}${result.versionId ? ` (${result.versionId})` : ""}`);
}
async function apiJson(registry, token, requestPath) {
const { status, body } = await apiJsonWithStatus(registry, token, requestPath);
if (status < 200 || status >= 300) {
throw new Error(body?.message || `HTTP ${status}`);
}
return body;
}
async function apiJsonWithStatus(registry, token, requestPath) {
const response = await fetch(`${registry}${requestPath}`, {
headers: {
Accept: "application/json",
Authorization: `Bearer ${token}`,
},
});
const text = await response.text();
let body = null;
try {
body = text ? JSON.parse(text) : null;
} catch {
body = { message: text };
}
return { status: response.status, body };
}
async function mapWithConcurrency(items, limit, fn) {
const results = new Array(items.length);
let cursor = 0;
async function worker() {
while (cursor < items.length) {
const index = cursor;
cursor += 1;
results[index] = await fn(items[index], index);
}
}
const count = Math.min(Math.max(limit, 1), Math.max(items.length, 1));
await Promise.all(Array.from({ length: count }, () => worker()));
return results;
}
function formatCandidate(candidate, bump) {
if (candidate.status === "new") {
return `${candidate.slug} NEW (${candidate.fileCount} files)`;
}
return `${candidate.slug} UPDATE ${candidate.latestVersion} -> ${bumpSemver(
candidate.latestVersion,
bump
)} (${candidate.fileCount} files)`;
}
function bumpSemver(version, bump) {
const match = /^(\d+)\.(\d+)\.(\d+)$/.exec(version ?? "");
if (!match) {
throw new Error(`Invalid semver: ${version}`);
}
const major = Number(match[1]);
const minor = Number(match[2]);
const patch = Number(match[3]);
if (bump === "major") return `${major + 1}.0.0`;
if (bump === "minor") return `${major}.${minor + 1}.0`;
return `${major}.${minor}.${patch + 1}`;
}
function sanitizeSlug(value) {
return value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+/, "")
.replace(/-+$/, "")
.replace(/--+/g, "-");
}
function titleCase(value) {
return value
.trim()
.replace(/[-_]+/g, " ")
.replace(/\s+/g, " ")
.replace(/\b\w/g, (char) => char.toUpperCase());
}
async function safeStat(filePath) {
try {
return await fs.stat(filePath);
} catch {
return null;
}
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
+3 -8
View File
@@ -4,13 +4,8 @@ set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SKILLS_DIR="${ROOT_DIR}/skills" SKILLS_DIR="${ROOT_DIR}/skills"
if command -v clawhub >/dev/null 2>&1; then if ! command -v node >/dev/null 2>&1; then
CLAWHUB_CMD=(clawhub) echo "Error: node is required."
elif command -v npx >/dev/null 2>&1; then
CLAWHUB_CMD=(npx -y clawhub)
else
echo "Error: neither clawhub nor npx is available."
echo "See https://docs.openclaw.ai/zh-CN/tools/clawhub"
exit 1 exit 1
fi fi
@@ -18,4 +13,4 @@ if [ "$#" -eq 0 ]; then
set -- --all set -- --all
fi fi
exec "${CLAWHUB_CMD[@]}" sync --root "${SKILLS_DIR}" "$@" exec node "${ROOT_DIR}/scripts/sync-clawhub.mjs" --root "${SKILLS_DIR}" "$@"
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env node
import path from "node:path";
import {
ensureManagedPathsClean,
syncSharedSkillPackages,
} from "./lib/shared-skill-packages.mjs";
async function main() {
const options = parseArgs(process.argv.slice(2));
const repoRoot = path.resolve(options.repoRoot);
const result = await syncSharedSkillPackages(repoRoot, {
targets: options.targets,
});
if (options.enforceClean) {
ensureManagedPathsClean(repoRoot, result.managedPaths);
}
console.log(`Synced shared workspace packages into ${result.packageDirs.length} skill script package(s).`);
}
function parseArgs(argv) {
const options = {
repoRoot: process.cwd(),
enforceClean: false,
targets: [],
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--repo-root") {
options.repoRoot = argv[index + 1] ?? options.repoRoot;
index += 1;
continue;
}
if (arg === "--enforce-clean") {
options.enforceClean = true;
continue;
}
if (arg === "--target") {
options.targets.push(argv[index + 1] ?? "");
index += 1;
continue;
}
if (arg === "-h" || arg === "--help") {
printUsage();
process.exit(0);
}
throw new Error(`Unknown argument: ${arg}`);
}
return options;
}
function printUsage() {
console.log(`Usage: sync-shared-skill-packages.mjs [options]
Options:
--repo-root <dir> Repository root (default: current directory)
--target <dir> Sync only one skill directory (can be repeated)
--enforce-clean Fail if managed files change after sync
-h, --help Show help`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
@@ -118,6 +118,8 @@ Full template: [references/workflow.md](references/workflow.md#step-4-generate-o
**BLOCKING: Prompt files MUST be saved before ANY image generation.** **BLOCKING: Prompt files MUST be saved before ANY image generation.**
**Execution strategy**: When multiple illustrations have saved prompt files and the task is now plain generation, prefer `baoyu-image-gen` batch mode (`build-batch.ts``--batchfile`) over spawning subagents. Use subagents only when each image still needs separate prompt iteration or creative exploration.
1. For each illustration, create a prompt file per [references/prompt-construction.md](references/prompt-construction.md) 1. For each illustration, create a prompt file per [references/prompt-construction.md](references/prompt-construction.md)
2. Save to `prompts/NN-{type}-{slug}.md` with YAML frontmatter 2. Save to `prompts/NN-{type}-{slug}.md` with YAML frontmatter
3. Prompts **MUST** use type-specific templates with structured sections (ZONES / LABELS / COLORS / STYLE / ASPECT) 3. Prompts **MUST** use type-specific templates with structured sections (ZONES / LABELS / COLORS / STYLE / ASPECT)
@@ -315,6 +315,10 @@ Prompt Files:
**DO NOT** pass ad-hoc inline text to `--prompt` without first saving prompt files. The generation command should either use `--promptfiles prompts/NN-{type}-{slug}.md` or read the saved file content for `--prompt`. **DO NOT** pass ad-hoc inline text to `--prompt` without first saving prompt files. The generation command should either use `--promptfiles prompts/NN-{type}-{slug}.md` or read the saved file content for `--prompt`.
**Execution choice**:
- If multiple illustrations already have saved prompt files and the task is now plain generation, prefer `baoyu-image-gen` batch mode (`build-batch.ts` -> `main.ts --batchfile`)
- Use subagents only when each illustration still needs separate prompt rewriting, style exploration, or other per-image reasoning before generation
**CRITICAL - References in Frontmatter**: **CRITICAL - References in Frontmatter**:
- Only add `references` field if files ACTUALLY EXIST in `references/` directory - Only add `references` field if files ACTUALLY EXIST in `references/` directory
- If style/palette was extracted verbally (no file), append info to prompt BODY instead - If style/palette was extracted verbally (no file), append info to prompt BODY instead
@@ -0,0 +1,156 @@
import path from "node:path";
import process from "node:process";
import { readdir, readFile, writeFile } from "node:fs/promises";
type CliArgs = {
outlinePath: string | null;
promptsDir: string | null;
outputPath: string | null;
imagesDir: string | null;
provider: string;
model: string;
aspectRatio: string;
quality: string;
jobs: number | null;
help: boolean;
};
type OutlineEntry = {
index: number;
filename: string;
};
function printUsage(): void {
console.log(`Usage:
npx -y tsx scripts/build-batch.ts --outline outline.md --prompts prompts --output batch.json --images-dir attachments
Options:
--outline <path> Path to outline.md
--prompts <path> Path to prompts directory
--output <path> Path to output batch.json
--images-dir <path> Directory for generated images
--provider <name> Provider for baoyu-image-gen batch tasks (default: replicate)
--model <id> Model for baoyu-image-gen batch tasks (default: google/nano-banana-pro)
--ar <ratio> Aspect ratio for all tasks (default: 16:9)
--quality <level> Quality for all tasks (default: 2k)
--jobs <count> Recommended worker count metadata (optional)
-h, --help Show help`);
}
function parseArgs(argv: string[]): CliArgs {
const args: CliArgs = {
outlinePath: null,
promptsDir: null,
outputPath: null,
imagesDir: null,
provider: "replicate",
model: "google/nano-banana-pro",
aspectRatio: "16:9",
quality: "2k",
jobs: null,
help: false,
};
for (let i = 0; i < argv.length; i++) {
const current = argv[i]!;
if (current === "--outline") args.outlinePath = argv[++i] ?? null;
else if (current === "--prompts") args.promptsDir = argv[++i] ?? null;
else if (current === "--output") args.outputPath = argv[++i] ?? null;
else if (current === "--images-dir") args.imagesDir = argv[++i] ?? null;
else if (current === "--provider") args.provider = argv[++i] ?? args.provider;
else if (current === "--model") args.model = argv[++i] ?? args.model;
else if (current === "--ar") args.aspectRatio = argv[++i] ?? args.aspectRatio;
else if (current === "--quality") args.quality = argv[++i] ?? args.quality;
else if (current === "--jobs") {
const value = argv[++i];
args.jobs = value ? parseInt(value, 10) : null;
} else if (current === "--help" || current === "-h") {
args.help = true;
}
}
return args;
}
function parseOutline(content: string): OutlineEntry[] {
const entries: OutlineEntry[] = [];
const blocks = content.split(/^## Illustration\s+/m).slice(1);
for (const block of blocks) {
const indexMatch = block.match(/^(\d+)/);
const filenameMatch = block.match(/\*\*Filename\*\*:\s*(.+)/);
if (indexMatch && filenameMatch) {
entries.push({
index: parseInt(indexMatch[1]!, 10),
filename: filenameMatch[1]!.trim(),
});
}
}
return entries;
}
async function findPromptFile(promptsDir: string, entry: OutlineEntry): Promise<string | null> {
const files = await readdir(promptsDir);
const prefix = String(entry.index).padStart(2, "0");
const match = files.find((f) => f.startsWith(prefix) && f.endsWith(".md"));
return match ? path.join(promptsDir, match) : null;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
if (args.help) {
printUsage();
return;
}
if (!args.outlinePath) {
console.error("Error: --outline is required");
process.exit(1);
}
if (!args.promptsDir) {
console.error("Error: --prompts is required");
process.exit(1);
}
if (!args.outputPath) {
console.error("Error: --output is required");
process.exit(1);
}
const outlineContent = await readFile(args.outlinePath, "utf8");
const entries = parseOutline(outlineContent);
if (entries.length === 0) {
console.error("No illustration entries found in outline.");
process.exit(1);
}
const tasks = [];
for (const entry of entries) {
const promptFile = await findPromptFile(args.promptsDir, entry);
if (!promptFile) {
console.error(`Warning: No prompt file found for illustration ${entry.index}, skipping.`);
continue;
}
const imageDir = args.imagesDir ?? path.dirname(args.outputPath);
tasks.push({
id: `illustration-${String(entry.index).padStart(2, "0")}`,
promptFiles: [promptFile],
image: path.join(imageDir, entry.filename),
provider: args.provider,
model: args.model,
ar: args.aspectRatio,
quality: args.quality,
});
}
const output: Record<string, unknown> = { tasks };
if (args.jobs) output.jobs = args.jobs;
await writeFile(args.outputPath, JSON.stringify(output, null, 2) + "\n");
console.log(`Batch file written: ${args.outputPath} (${tasks.length} tasks)`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
@@ -0,0 +1,14 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-danger-gemini-web-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
},
},
},
"packages": {
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
}
}
@@ -1,183 +1,76 @@
import { spawn, type ChildProcess } from 'node:child_process';
import fs from 'node:fs';
import { mkdir } from 'node:fs/promises';
import net from 'node:net';
import process from 'node:process'; import process from 'node:process';
import {
CdpConnection,
findChromeExecutable as findChromeExecutableBase,
findExistingChromeDebugPort,
getFreePort,
killChrome,
launchChrome as launchChromeBase,
openPageSession,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
import { Endpoint, Headers } from '../constants.js'; import { Endpoint, Headers } from '../constants.js';
import { logger } from './logger.js'; import { logger } from './logger.js';
import { cookie_header, fetch_with_timeout, sleep } from './http.js'; import { cookie_header, fetch_with_timeout } from './http.js';
import { read_cookie_file, type CookieMap, write_cookie_file } from './cookie-file.js'; import { read_cookie_file, type CookieMap, write_cookie_file } from './cookie-file.js';
import { resolveGeminiWebChromeProfileDir, resolveGeminiWebCookiePath } from './paths.js'; import { resolveGeminiWebChromeProfileDir, resolveGeminiWebCookiePath } from './paths.js';
type CdpSendOptions = { sessionId?: string; timeoutMs?: number }; const GEMINI_APP_URL = 'https://gemini.google.com/app';
class CdpConnection { const CHROME_CANDIDATES_FULL: PlatformCandidates = {
private ws: WebSocket; darwin: [
private nextId = 0; '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
private pending = new Map< '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
number, '/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
{ resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null } '/Applications/Chromium.app/Contents/MacOS/Chromium',
>(); '/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
],
private constructor(ws: WebSocket) { win32: [
this.ws = ws; 'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
this.ws.addEventListener('message', (event) => { 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
try { 'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer); 'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
const msg = JSON.parse(data) as { id?: number; result?: unknown; error?: { message?: string } }; ],
if (msg.id) { default: [
const p = this.pending.get(msg.id); '/usr/bin/google-chrome',
if (p) { '/usr/bin/google-chrome-stable',
this.pending.delete(msg.id); '/usr/bin/chromium',
if (p.timer) clearTimeout(p.timer); '/usr/bin/chromium-browser',
if (msg.error?.message) p.reject(new Error(msg.error.message)); '/snap/bin/chromium',
else p.resolve(msg.result); '/usr/bin/microsoft-edge',
} ],
} };
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, p] of this.pending.entries()) {
this.pending.delete(id);
if (p.timer) clearTimeout(p.timer);
p.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const t = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => {
clearTimeout(t);
resolve();
});
ws.addEventListener('error', () => {
clearTimeout(t);
reject(new Error('CDP connection failed.'));
});
});
return new CdpConnection(ws);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, opts?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const msg: Record<string, unknown> = { id, method };
if (params) msg.params = params;
if (opts?.sessionId) msg.sessionId = opts.sessionId;
const timeoutMs = opts?.timeoutMs ?? 15_000;
const out = await new Promise<unknown>((resolve, reject) => {
const t =
timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer: t });
this.ws.send(JSON.stringify(msg));
});
return out as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
async function get_free_port(): Promise<number> { async function get_free_port(): Promise<number> {
const fixed = parseInt(process.env.GEMINI_WEB_DEBUG_PORT || '', 10); return await getFreePort('GEMINI_WEB_DEBUG_PORT');
if (fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const srv = net.createServer();
srv.unref();
srv.on('error', reject);
srv.listen(0, '127.0.0.1', () => {
const addr = srv.address();
if (!addr || typeof addr === 'string') {
srv.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = addr.port;
srv.close((err) => (err ? reject(err) : resolve(port)));
});
});
} }
function find_chrome_executable(): string | null { function find_chrome_executable(): string | null {
const override = process.env.GEMINI_WEB_CHROME_PATH?.trim(); return findChromeExecutableBase({
if (override && fs.existsSync(override)) return override; candidates: CHROME_CANDIDATES_FULL,
envNames: ['GEMINI_WEB_CHROME_PATH'],
const candidates: string[] = []; }) ?? null;
switch (process.platform) {
case 'darwin':
candidates.push(
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
);
break;
case 'win32':
candidates.push(
'C:\\\\Program Files\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe',
'C:\\\\Program Files (x86)\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe',
'C:\\\\Program Files\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe',
'C:\\\\Program Files (x86)\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe',
);
break;
default:
candidates.push(
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
);
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return null;
} }
async function wait_for_chrome_debug_port(port: number, timeoutMs: number): Promise<string> { async function find_existing_chrome_debug_port(profileDir: string): Promise<number | null> {
const start = Date.now(); return await findExistingChromeDebugPort({ profileDir });
while (Date.now() - start < timeoutMs) {
try {
const res = await fetch_with_timeout(`http://127.0.0.1:${port}/json/version`, { timeout_ms: 5_000 });
if (!res.ok) throw new Error(`status=${res.status}`);
const j = (await res.json()) as { webSocketDebuggerUrl?: string };
if (j.webSocketDebuggerUrl) return j.webSocketDebuggerUrl;
} catch {}
await sleep(200);
}
throw new Error('Chrome debug port not ready');
} }
async function launch_chrome(profileDir: string, port: number): Promise<ChildProcess> { async function launch_chrome(profileDir: string, port: number) {
const chrome = find_chrome_executable(); const chromePath = find_chrome_executable();
if (!chrome) throw new Error('Chrome executable not found.'); if (!chromePath) throw new Error('Chrome executable not found.');
const args = [ return await launchChromeBase({
`--remote-debugging-port=${port}`, chromePath,
`--user-data-dir=${profileDir}`, profileDir,
'--no-first-run', port,
'--no-default-browser-check', url: GEMINI_APP_URL,
'--disable-popup-blocking', extraArgs: ['--disable-popup-blocking'],
'https://gemini.google.com/app', });
];
return spawn(chrome, args, { stdio: 'ignore' });
} }
async function is_gemini_session_ready(cookies: CookieMap, verbose: boolean): Promise<boolean> { async function is_gemini_session_ready(cookies: CookieMap, verbose: boolean): Promise<boolean> {
@@ -209,27 +102,33 @@ async function fetch_google_cookies_via_cdp(
timeoutMs: number, timeoutMs: number,
verbose: boolean, verbose: boolean,
): Promise<CookieMap> { ): Promise<CookieMap> {
await mkdir(profileDir, { recursive: true }); const existingPort = await find_existing_chrome_debug_port(profileDir);
const reusing = existingPort !== null;
const port = await get_free_port(); const port = existingPort ?? await get_free_port();
const chrome = await launch_chrome(profileDir, port); const chrome = reusing ? null : await launch_chrome(profileDir, port);
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await wait_for_chrome_debug_port(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 15_000); cdp = await CdpConnection.connect(wsUrl, 15_000);
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', {
url: 'https://gemini.google.com/app',
newWindow: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId, flatten: true });
await cdp.send('Network.enable', {}, { sessionId });
if (verbose) { if (verbose) {
logger.info('Chrome opened. If needed, complete Google login in the window. Waiting for a valid Gemini session...'); logger.info(reusing
? `Reusing existing Chrome on port ${port}. Waiting for a valid Gemini session...`
: 'Chrome opened. If needed, complete Google login in the window. Waiting for a valid Gemini session...');
} }
const page = await openPageSession({
cdp,
reusing,
url: GEMINI_APP_URL,
matchTarget: (target) => target.type === 'page' && target.url.includes('gemini.google.com'),
enableNetwork: true,
});
const { sessionId } = page;
targetId = page.targetId;
const start = Date.now(); const start = Date.now();
let last: CookieMap = {}; let last: CookieMap = {};
@@ -240,14 +139,14 @@ async function fetch_google_cookies_via_cdp(
{ sessionId, timeoutMs: 10_000 }, { sessionId, timeoutMs: 10_000 },
); );
const m: CookieMap = {}; const cookieMap: CookieMap = {};
for (const c of cookies) { for (const cookie of cookies) {
if (c?.name && typeof c.value === 'string') m[c.name] = c.value; if (cookie?.name && typeof cookie.value === 'string') cookieMap[cookie.name] = cookie.value;
} }
last = m; last = cookieMap;
if (await is_gemini_session_ready(m, verbose)) { if (await is_gemini_session_ready(cookieMap, verbose)) {
return m; return cookieMap;
} }
await sleep(1000); await sleep(1000);
@@ -256,22 +155,19 @@ async function fetch_google_cookies_via_cdp(
throw new Error(`Timed out waiting for a valid Gemini session. Last keys: ${Object.keys(last).join(', ')}`); throw new Error(`Timed out waiting for a valid Gemini session. Last keys: ${Object.keys(last).join(', ')}`);
} finally { } finally {
if (cdp) { if (cdp) {
try { if (reusing && targetId) {
await cdp.send('Browser.close', {}, { timeoutMs: 5_000 }); try {
} catch {} await cdp.send('Target.closeTarget', { targetId }, { timeoutMs: 5_000 });
} catch {}
} else {
try {
await cdp.send('Browser.close', {}, { timeoutMs: 5_000 });
} catch {}
}
cdp.close(); cdp.close();
} }
try { if (chrome) killChrome(chrome);
chrome.kill('SIGTERM');
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill('SIGKILL');
} catch {}
}
}, 2_000).unref?.();
} }
} }
@@ -286,8 +182,8 @@ export async function load_browser_cookies(domain_name: string = '', verbose: bo
const cookies = await fetch_google_cookies_via_cdp(profileDir, 120_000, verbose); const cookies = await fetch_google_cookies_via_cdp(profileDir, 120_000, verbose);
const filtered: CookieMap = {}; const filtered: CookieMap = {};
for (const [k, v] of Object.entries(cookies)) { for (const [key, value] of Object.entries(cookies)) {
if (typeof v === 'string' && v.length > 0) filtered[k] = v; if (typeof value === 'string' && value.length > 0) filtered[key] = value;
} }
await write_cookie_file(filtered, resolveGeminiWebCookiePath(), 'cdp'); await write_cookie_file(filtered, resolveGeminiWebCookiePath(), 'cdp');
@@ -0,0 +1,8 @@
{
"name": "baoyu-danger-gemini-web-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
@@ -0,0 +1,14 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-danger-x-to-markdown-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
},
},
},
"packages": {
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
}
}
@@ -1,7 +1,16 @@
import { spawn, type ChildProcess } from "node:child_process"; import {
import fs from "node:fs"; CdpConnection,
import { mkdir } from "node:fs/promises"; findChromeExecutable as findChromeExecutableBase,
import net from "node:net"; findExistingChromeDebugPort,
getFreePort,
killChrome,
launchChrome as launchChromeBase,
openPageSession,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from "baoyu-chrome-cdp";
import process from "node:process"; import process from "node:process";
import { read_cookie_file, write_cookie_file } from "./cookie-file.js"; import { read_cookie_file, write_cookie_file } from "./cookie-file.js";
@@ -9,201 +18,47 @@ import { resolveXToMarkdownCookiePath } from "./paths.js";
import { X_COOKIE_NAMES, X_REQUIRED_COOKIES, X_LOGIN_URL, X_USER_DATA_DIR } from "./constants.js"; import { X_COOKIE_NAMES, X_REQUIRED_COOKIES, X_LOGIN_URL, X_USER_DATA_DIR } from "./constants.js";
import type { CookieLike } from "./types.js"; import type { CookieLike } from "./types.js";
type CdpSendOptions = { sessionId?: string; timeoutMs?: number }; const CHROME_CANDIDATES_FULL: PlatformCandidates = {
darwin: [
function sleep(ms: number): Promise<void> { "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
return new Promise((resolve) => setTimeout(resolve, ms)); "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
} "/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
async function fetchWithTimeout( "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
url: string, ],
init: RequestInit & { timeoutMs?: number } = {} win32: [
): Promise<Response> { "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
const { timeoutMs, ...rest } = init; "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
if (!timeoutMs || timeoutMs <= 0) return fetch(url, rest); "C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe",
"C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
const ctl = new AbortController(); ],
const t = setTimeout(() => ctl.abort(), timeoutMs); default: [
try { "/usr/bin/google-chrome",
return await fetch(url, { ...rest, signal: ctl.signal }); "/usr/bin/google-chrome-stable",
} finally { "/usr/bin/chromium",
clearTimeout(t); "/usr/bin/chromium-browser",
} "/snap/bin/chromium",
} "/usr/bin/microsoft-edge",
],
class CdpConnection { };
private ws: WebSocket;
private nextId = 0;
private pending = new Map<
number,
{ resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }
>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener("message", (event) => {
try {
const data =
typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; result?: unknown; error?: { message?: string } };
if (msg.id) {
const p = this.pending.get(msg.id);
if (p) {
this.pending.delete(msg.id);
if (p.timer) clearTimeout(p.timer);
if (msg.error?.message) p.reject(new Error(msg.error.message));
else p.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, p] of this.pending.entries()) {
this.pending.delete(id);
if (p.timer) clearTimeout(p.timer);
p.reject(new Error("CDP connection closed."));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const t = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(t);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(t);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws);
}
async send<T = unknown>(
method: string,
params?: Record<string, unknown>,
opts?: CdpSendOptions
): Promise<T> {
const id = ++this.nextId;
const msg: Record<string, unknown> = { id, method };
if (params) msg.params = params;
if (opts?.sessionId) msg.sessionId = opts.sessionId;
const timeoutMs = opts?.timeoutMs ?? 15_000;
const out = await new Promise<unknown>((resolve, reject) => {
const t =
timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer: t });
this.ws.send(JSON.stringify(msg));
});
return out as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.X_DEBUG_PORT || "", 10);
if (fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const srv = net.createServer();
srv.unref();
srv.on("error", reject);
srv.listen(0, "127.0.0.1", () => {
const addr = srv.address();
if (!addr || typeof addr === "string") {
srv.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = addr.port;
srv.close((err) => (err ? reject(err) : resolve(port)));
});
});
}
function findChromeExecutable(): string | null { function findChromeExecutable(): string | null {
const override = process.env.X_CHROME_PATH?.trim(); return findChromeExecutableBase({
if (override && fs.existsSync(override)) return override; candidates: CHROME_CANDIDATES_FULL,
envNames: ["X_CHROME_PATH"],
const candidates: string[] = []; }) ?? null;
switch (process.platform) {
case "darwin":
candidates.push(
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
"/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge"
);
break;
case "win32":
candidates.push(
"C:\\\\Program Files\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe",
"C:\\\\Program Files (x86)\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe",
"C:\\\\Program Files\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe",
"C:\\\\Program Files (x86)\\\\Microsoft\\\\Edge\\\\Application\\\\msedge.exe"
);
break;
default:
candidates.push(
"/usr/bin/google-chrome",
"/usr/bin/google-chrome-stable",
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
"/snap/bin/chromium",
"/usr/bin/microsoft-edge"
);
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return null;
} }
async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> { async function launchChrome(profileDir: string, port: number) {
const start = Date.now(); const chromePath = findChromeExecutable();
while (Date.now() - start < timeoutMs) { if (!chromePath) throw new Error("Chrome executable not found.");
try { return await launchChromeBase({
const res = await fetchWithTimeout(`http://127.0.0.1:${port}/json/version`, { timeoutMs: 5_000 }); chromePath,
if (!res.ok) throw new Error(`status=${res.status}`); profileDir,
const j = (await res.json()) as { webSocketDebuggerUrl?: string }; port,
if (j.webSocketDebuggerUrl) return j.webSocketDebuggerUrl; url: X_LOGIN_URL,
} catch {} extraArgs: ["--disable-popup-blocking"],
await sleep(200); });
}
throw new Error("Chrome debug port not ready");
}
async function launchChrome(profileDir: string, port: number): Promise<ChildProcess> {
const chrome = findChromeExecutable();
if (!chrome) throw new Error("Chrome executable not found.");
const args = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
"--no-first-run",
"--no-default-browser-check",
"--disable-popup-blocking",
X_LOGIN_URL,
];
return spawn(chrome, args, { stdio: "ignore" });
} }
async function fetchXCookiesViaCdp( async function fetchXCookiesViaCdp(
@@ -212,25 +67,33 @@ async function fetchXCookiesViaCdp(
verbose: boolean, verbose: boolean,
log?: (message: string) => void log?: (message: string) => void
): Promise<Record<string, string>> { ): Promise<Record<string, string>> {
await mkdir(profileDir, { recursive: true }); const existingPort = await findExistingChromeDebugPort({ profileDir });
const reusing = existingPort !== null;
const port = await getFreePort(); const port = existingPort ?? await getFreePort("X_DEBUG_PORT");
const chrome = await launchChrome(profileDir, port); const chrome = reusing ? null : await launchChrome(profileDir, port);
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 15_000); cdp = await CdpConnection.connect(wsUrl, 15_000);
const { targetId } = await cdp.send<{ targetId: string }>("Target.createTarget", { const page = await openPageSession({
cdp,
reusing,
url: X_LOGIN_URL, url: X_LOGIN_URL,
newWindow: true, matchTarget: (target) => target.type === "page" && (
target.url.includes("x.com") || target.url.includes("twitter.com")
),
enableNetwork: true,
}); });
const { sessionId } = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true }); const { sessionId } = page;
await cdp.send("Network.enable", {}, { sessionId }); targetId = page.targetId;
if (verbose) { if (verbose) {
log?.("[x-cookies] Chrome opened. If needed, complete X login in the window. Waiting for cookies..."); log?.(reusing
? `[x-cookies] Reusing existing Chrome on port ${port}. Waiting for cookies...`
: "[x-cookies] Chrome opened. If needed, complete X login in the window. Waiting for cookies...");
} }
const start = Date.now(); const start = Date.now();
@@ -255,22 +118,19 @@ async function fetchXCookiesViaCdp(
throw new Error(`Timed out waiting for X cookies. Last keys: ${Object.keys(last).join(", ")}`); throw new Error(`Timed out waiting for X cookies. Last keys: ${Object.keys(last).join(", ")}`);
} finally { } finally {
if (cdp) { if (cdp) {
try { if (reusing && targetId) {
await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); try {
} catch {} await cdp.send("Target.closeTarget", { targetId }, { timeoutMs: 5_000 });
} catch {}
} else {
try {
await cdp.send("Browser.close", {}, { timeoutMs: 5_000 });
} catch {}
}
cdp.close(); cdp.close();
} }
try { if (chrome) killChrome(chrome);
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
} }
} }
@@ -0,0 +1,8 @@
{
"name": "baoyu-danger-x-to-markdown-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
+74 -29
View File
@@ -1,6 +1,6 @@
--- ---
name: baoyu-image-gen name: baoyu-image-gen
description: AI image generation with OpenAI, Google, DashScope and Replicate APIs. Supports text-to-image, reference images, aspect ratios. Sequential by default; parallel generation available on request. Use when user asks to generate, create, or draw images. description: AI image generation with OpenAI, Google, DashScope and Replicate APIs. Supports text-to-image, reference images, aspect ratios, and batch generation from saved prompt files. Sequential by default; use batch parallel generation when the user already has multiple prompts or wants stable multi-image throughput. Use when user asks to generate, create, or draw images.
version: 1.56.1 version: 1.56.1
metadata: metadata:
openclaw: openclaw:
@@ -55,7 +55,7 @@ if (Test-Path "$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md") { "user" }
| `.baoyu-skills/baoyu-image-gen/EXTEND.md` | Project directory | | `.baoyu-skills/baoyu-image-gen/EXTEND.md` | Project directory |
| `$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md` | User home | | `$HOME/.baoyu-skills/baoyu-image-gen/EXTEND.md` | User home |
**EXTEND.md Supports**: Default provider | Default quality | Default aspect ratio | Default image size | Default models **EXTEND.md Supports**: Default provider | Default quality | Default aspect ratio | Default image size | Default models | Batch worker cap | Provider-specific batch limits
Schema: `references/config/preferences-schema.md` Schema: `references/config/preferences-schema.md`
@@ -91,22 +91,57 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
# Replicate with specific model # Replicate with specific model
${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana
# Batch mode with saved prompt files
${BUN_X} {baseDir}/scripts/main.ts --batchfile batch.json
# Batch mode with explicit worker count
${BUN_X} {baseDir}/scripts/main.ts --batchfile batch.json --jobs 4 --json
``` ```
### Batch File Format
```json
{
"jobs": 4,
"tasks": [
{
"id": "hero",
"promptFiles": ["prompts/hero.md"],
"image": "out/hero.png",
"provider": "replicate",
"model": "google/nano-banana-pro",
"ar": "16:9",
"quality": "2k"
},
{
"id": "diagram",
"promptFiles": ["prompts/diagram.md"],
"image": "out/diagram.png",
"ref": ["references/original.png"]
}
]
}
```
Paths in `promptFiles`, `image`, and `ref` are resolved relative to the batch file's directory. `jobs` is optional (overridden by CLI `--jobs`). Top-level array format (without `jobs` wrapper) is also accepted.
## Options ## Options
| Option | Description | | Option | Description |
|--------|-------------| |--------|-------------|
| `--prompt <text>`, `-p` | Prompt text | | `--prompt <text>`, `-p` | Prompt text |
| `--promptfiles <files...>` | Read prompt from files (concatenated) | | `--promptfiles <files...>` | Read prompt from files (concatenated) |
| `--image <path>` | Output image path (required) | | `--image <path>` | Output image path (required in single-image mode) |
| `--provider google\|openai\|dashscope\|replicate` | Force provider (default: google) | | `--batchfile <path>` | JSON batch file for multi-image generation |
| `--model <id>`, `-m` | Model ID (Google: `gemini-3-pro-image-preview`, `gemini-3.1-flash-image-preview`; OpenAI: `gpt-image-1.5`) | | `--jobs <count>` | Worker count for batch mode (default: auto, max from config, built-in default 10) |
| `--provider google\|openai\|dashscope\|replicate` | Force provider (default: auto-detect) |
| `--model <id>`, `-m` | Model ID (Google: `gemini-3-pro-image-preview`, `gemini-3.1-flash-image-preview`; OpenAI: `gpt-image-1.5`, `gpt-image-1`) |
| `--ar <ratio>` | Aspect ratio (e.g., `16:9`, `1:1`, `4:3`) | | `--ar <ratio>` | Aspect ratio (e.g., `16:9`, `1:1`, `4:3`) |
| `--size <WxH>` | Size (e.g., `1024x1024`) | | `--size <WxH>` | Size (e.g., `1024x1024`) |
| `--quality normal\|2k` | Quality preset (default: 2k) | | `--quality normal\|2k` | Quality preset (default: `2k`) |
| `--imageSize 1K\|2K\|4K` | Image size for Google (default: from quality) | | `--imageSize 1K\|2K\|4K` | Image size for Google (default: from quality) |
| `--ref <files...>` | Reference images. Supported by Google multimodal (`gemini-3-pro-image-preview`, `gemini-3-flash-preview`, `gemini-3.1-flash-image-preview`) and OpenAI edits (GPT Image models). If provider omitted: Google first, then OpenAI | | `--ref <files...>` | Reference images. Supported by Google multimodal, OpenAI GPT Image edits, and Replicate |
| `--n <count>` | Number of images | | `--n <count>` | Number of images |
| `--json` | JSON output | | `--json` | JSON output |
@@ -126,6 +161,9 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
| `GOOGLE_BASE_URL` | Custom Google endpoint | | `GOOGLE_BASE_URL` | Custom Google endpoint |
| `DASHSCOPE_BASE_URL` | Custom DashScope endpoint | | `DASHSCOPE_BASE_URL` | Custom DashScope endpoint |
| `REPLICATE_BASE_URL` | Custom Replicate endpoint | | `REPLICATE_BASE_URL` | Custom Replicate endpoint |
| `BAOYU_IMAGE_GEN_MAX_WORKERS` | Override batch worker cap |
| `BAOYU_IMAGE_GEN_<PROVIDER>_CONCURRENCY` | Override provider concurrency, e.g. `BAOYU_IMAGE_GEN_REPLICATE_CONCURRENCY` |
| `BAOYU_IMAGE_GEN_<PROVIDER>_START_INTERVAL_MS` | Override provider start gap, e.g. `BAOYU_IMAGE_GEN_REPLICATE_START_INTERVAL_MS` |
**Load Priority**: CLI args > EXTEND.md > env vars > `<cwd>/.baoyu-skills/.env` > `~/.baoyu-skills/.env` **Load Priority**: CLI args > EXTEND.md > env vars > `<cwd>/.baoyu-skills/.env` > `~/.baoyu-skills/.env`
@@ -170,10 +208,10 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
## Quality Presets ## Quality Presets
| Preset | Google imageSize | OpenAI Size | Use Case | | Preset | Google imageSize | OpenAI Size | Replicate resolution | Use Case |
|--------|------------------|-------------|----------| |--------|------------------|-------------|----------------------|----------|
| `normal` | 1K | 1024px | Quick previews | | `normal` | 1K | 1024px | 1K | Quick previews |
| `2k` (default) | 2K | 2048px | Covers, illustrations, infographics | | `2k` (default) | 2K | 2048px | 2K | Covers, illustrations, infographics |
**Google imageSize**: Can be overridden with `--imageSize 1K|2K|4K` **Google imageSize**: Can be overridden with `--imageSize 1K|2K|4K`
@@ -182,41 +220,48 @@ ${BUN_X} {baseDir}/scripts/main.ts --prompt "A cat" --image out.png --provider r
Supported: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `2.35:1` Supported: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `2.35:1`
- Google multimodal: uses `imageConfig.aspectRatio` - Google multimodal: uses `imageConfig.aspectRatio`
- Google Imagen: uses `aspectRatio` parameter
- OpenAI: maps to closest supported size - OpenAI: maps to closest supported size
- Replicate: passes `aspect_ratio` to model; when `--ref` is provided without `--ar`, defaults to `match_input_image`
## Generation Mode ## Generation Mode
**Default**: Sequential generation (one image at a time). This ensures stable output and easier debugging. **Default**: Sequential generation.
**Parallel Generation**: Only use when user explicitly requests parallel/concurrent generation. **Batch Parallel Generation**: When `--batchfile` contains 2 or more pending tasks, the script automatically enables parallel generation.
| Mode | When to Use | | Mode | When to Use |
|------|-------------| |------|-------------|
| Sequential (default) | Normal usage, single images, small batches | | Sequential (default) | Normal usage, single images, small batches |
| Parallel | User explicitly requests, large batches (10+) | | Parallel batch | Batch mode with 2+ tasks |
**Parallel Settings** (when requested): Execution choice:
| Setting | Value | | Situation | Preferred approach | Why |
|---------|-------| |-----------|--------------------|-----|
| Recommended concurrency | 4 subagents | | One image, or 1-2 simple images | Sequential | Lower coordination overhead and easier debugging |
| Max concurrency | 8 subagents | | Multiple images already have saved prompt files | Batch (`--batchfile`) | Reuses finalized prompts, applies shared throttling/retries, and gives predictable throughput |
| Use case | Large batch generation when user requests parallel | | Each image still needs separate reasoning, prompt writing, or style exploration | Subagents | The work is still exploratory, so each image may need independent analysis before generation |
| Output comes from `baoyu-article-illustrator` with `outline.md` + `prompts/` | Batch (`build-batch.ts` -> `--batchfile`) | That workflow already produces prompt files, so direct batch execution is the intended path |
**Agent Implementation** (parallel mode only): Rule of thumb:
```
# Launch multiple generations in parallel using Task tool - Prefer batch over subagents once prompt files are already saved and the task is "generate all of these"
# Each Task runs as background subagent with run_in_background=true - Use subagents only when generation is coupled with per-image thinking, rewriting, or divergent creative exploration
# Collect results via TaskOutput when all complete
``` Parallel behavior:
- Default worker count is automatic, capped by config, built-in default 10
- Provider-specific throttling is applied only in batch mode, and the built-in defaults are tuned for faster throughput while still avoiding obvious RPM bursts
- You can override worker count with `--jobs <count>`
- Each image retries automatically up to 3 attempts
- Final output includes success count, failure count, and per-image failure reasons
## Error Handling ## Error Handling
- Missing API key → error with setup instructions - Missing API key → error with setup instructions
- Generation failure → auto-retry once - Generation failure → auto-retry up to 3 attempts per image
- Invalid aspect ratio → warning, proceed with default - Invalid aspect ratio → warning, proceed with default
- Reference images with unsupported provider/model → error with fix hint (switch to Google multimodal: `gemini-3-pro-image-preview`, `gemini-3.1-flash-image-preview`; or OpenAI GPT Image edits) - Reference images with unsupported provider/model → error with fix hint
## Extension Support ## Extension Support
@@ -21,9 +21,25 @@ default_image_size: null # 1K|2K|4K|null (Google only, overrides quality)
default_model: default_model:
google: null # e.g., "gemini-3-pro-image-preview", "gemini-3.1-flash-image-preview" google: null # e.g., "gemini-3-pro-image-preview", "gemini-3.1-flash-image-preview"
openai: null # e.g., "gpt-image-1.5" openai: null # e.g., "gpt-image-1.5", "gpt-image-1"
dashscope: null # e.g., "z-image-turbo" dashscope: null # e.g., "z-image-turbo"
replicate: null # e.g., "google/nano-banana-pro" replicate: null # e.g., "google/nano-banana-pro"
batch:
max_workers: 10
provider_limits:
replicate:
concurrency: 5
start_interval_ms: 700
google:
concurrency: 3
start_interval_ms: 1100
openai:
concurrency: 3
start_interval_ms: 1100
dashscope:
concurrency: 3
start_interval_ms: 1100
--- ---
``` ```
@@ -40,6 +56,9 @@ default_model:
| `default_model.openai` | string\|null | null | OpenAI default model | | `default_model.openai` | string\|null | null | OpenAI default model |
| `default_model.dashscope` | string\|null | null | DashScope default model | | `default_model.dashscope` | string\|null | null | DashScope default model |
| `default_model.replicate` | string\|null | null | Replicate default model | | `default_model.replicate` | string\|null | null | Replicate default model |
| `batch.max_workers` | int\|null | 10 | Batch worker cap |
| `batch.provider_limits.<provider>.concurrency` | int\|null | provider default | Max simultaneous requests per provider |
| `batch.provider_limits.<provider>.start_interval_ms` | int\|null | provider default | Minimum gap between request starts per provider |
## Examples ## Examples
@@ -65,5 +84,11 @@ default_model:
openai: "gpt-image-1.5" openai: "gpt-image-1.5"
dashscope: "z-image-turbo" dashscope: "z-image-turbo"
replicate: "google/nano-banana-pro" replicate: "google/nano-banana-pro"
batch:
max_workers: 10
provider_limits:
replicate:
concurrency: 5
start_interval_ms: 700
--- ---
``` ```
+560 -105
View File
@@ -2,34 +2,108 @@ import path from "node:path";
import process from "node:process"; import process from "node:process";
import { homedir } from "node:os"; import { homedir } from "node:os";
import { access, mkdir, readFile, writeFile } from "node:fs/promises"; import { access, mkdir, readFile, writeFile } from "node:fs/promises";
import type { CliArgs, Provider, ExtendConfig } from "./types"; import type {
BatchFile,
BatchTaskInput,
CliArgs,
ExtendConfig,
Provider,
} from "./types";
type ProviderModule = {
getDefaultModel: () => string;
generateImage: (prompt: string, model: string, args: CliArgs) => Promise<Uint8Array>;
};
type PreparedTask = {
id: string;
prompt: string;
args: CliArgs;
provider: Provider;
model: string;
outputPath: string;
providerModule: ProviderModule;
};
type TaskResult = {
id: string;
provider: Provider;
model: string;
outputPath: string;
success: boolean;
attempts: number;
error: string | null;
};
type ProviderRateLimit = {
concurrency: number;
startIntervalMs: number;
};
type LoadedBatchTasks = {
tasks: BatchTaskInput[];
jobs: number | null;
batchDir: string;
};
const MAX_ATTEMPTS = 3;
const DEFAULT_MAX_WORKERS = 10;
const POLL_WAIT_MS = 250;
const DEFAULT_PROVIDER_RATE_LIMITS: Record<Provider, ProviderRateLimit> = {
replicate: { concurrency: 5, startIntervalMs: 700 },
google: { concurrency: 3, startIntervalMs: 1100 },
openai: { concurrency: 3, startIntervalMs: 1100 },
dashscope: { concurrency: 3, startIntervalMs: 1100 },
};
function printUsage(): void { function printUsage(): void {
console.log(`Usage: console.log(`Usage:
npx -y bun scripts/main.ts --prompt "A cat" --image cat.png npx -y bun scripts/main.ts --prompt "A cat" --image cat.png
npx -y bun scripts/main.ts --prompt "A landscape" --image landscape.png --ar 16:9
npx -y bun scripts/main.ts --promptfiles system.md content.md --image out.png npx -y bun scripts/main.ts --promptfiles system.md content.md --image out.png
npx -y bun scripts/main.ts --batchfile batch.json
Options: Options:
-p, --prompt <text> Prompt text -p, --prompt <text> Prompt text
--promptfiles <files...> Read prompt from files (concatenated) --promptfiles <files...> Read prompt from files (concatenated)
--image <path> Output image path (required) --image <path> Output image path (required in single-image mode)
--batchfile <path> JSON batch file for multi-image generation
--jobs <count> Worker count for batch mode (default: auto, max from config, built-in default 10)
--provider google|openai|dashscope|replicate Force provider (auto-detect by default) --provider google|openai|dashscope|replicate Force provider (auto-detect by default)
-m, --model <id> Model ID -m, --model <id> Model ID
--ar <ratio> Aspect ratio (e.g., 16:9, 1:1, 4:3) --ar <ratio> Aspect ratio (e.g., 16:9, 1:1, 4:3)
--size <WxH> Size (e.g., 1024x1024) --size <WxH> Size (e.g., 1024x1024)
--quality normal|2k Quality preset (default: 2k) --quality normal|2k Quality preset (default: 2k)
--imageSize 1K|2K|4K Image size for Google (default: from quality) --imageSize 1K|2K|4K Image size for Google (default: from quality)
--ref <files...> Reference images (Google multimodal or OpenAI edits) --ref <files...> Reference images (Google multimodal, OpenAI GPT Image edits, or Replicate)
--n <count> Number of images (default: 1) --n <count> Number of images for the current task (default: 1)
--json JSON output --json JSON output
-h, --help Show help -h, --help Show help
Batch file format:
{
"jobs": 4,
"tasks": [
{
"id": "hero",
"promptFiles": ["prompts/hero.md"],
"image": "out/hero.png",
"provider": "replicate",
"model": "google/nano-banana-pro",
"ar": "16:9"
}
]
}
Behavior:
- Batch mode automatically runs in parallel when pending tasks >= 2
- Each image retries automatically up to 3 attempts
- Batch summary reports success count, failure count, and per-image errors
Environment variables: Environment variables:
OPENAI_API_KEY OpenAI API key OPENAI_API_KEY OpenAI API key
GOOGLE_API_KEY Google API key GOOGLE_API_KEY Google API key
GEMINI_API_KEY Gemini API key (alias for GOOGLE_API_KEY) GEMINI_API_KEY Gemini API key (alias for GOOGLE_API_KEY)
DASHSCOPE_API_KEY DashScope API key (阿里云通义万象) DASHSCOPE_API_KEY DashScope API key
REPLICATE_API_TOKEN Replicate API token REPLICATE_API_TOKEN Replicate API token
OPENAI_IMAGE_MODEL Default OpenAI model (gpt-image-1.5) OPENAI_IMAGE_MODEL Default OpenAI model (gpt-image-1.5)
GOOGLE_IMAGE_MODEL Default Google model (gemini-3-pro-image-preview) GOOGLE_IMAGE_MODEL Default Google model (gemini-3-pro-image-preview)
@@ -40,6 +114,9 @@ Environment variables:
GOOGLE_BASE_URL Custom Google endpoint GOOGLE_BASE_URL Custom Google endpoint
DASHSCOPE_BASE_URL Custom DashScope endpoint DASHSCOPE_BASE_URL Custom DashScope endpoint
REPLICATE_BASE_URL Custom Replicate endpoint REPLICATE_BASE_URL Custom Replicate endpoint
BAOYU_IMAGE_GEN_MAX_WORKERS Override batch worker cap
BAOYU_IMAGE_GEN_<PROVIDER>_CONCURRENCY Override provider concurrency
BAOYU_IMAGE_GEN_<PROVIDER>_START_INTERVAL_MS Override provider start gap in ms
Env file load order: CLI args > EXTEND.md > process.env > <cwd>/.baoyu-skills/.env > ~/.baoyu-skills/.env`); Env file load order: CLI args > EXTEND.md > process.env > <cwd>/.baoyu-skills/.env > ~/.baoyu-skills/.env`);
} }
@@ -57,6 +134,8 @@ function parseArgs(argv: string[]): CliArgs {
imageSize: null, imageSize: null,
referenceImages: [], referenceImages: [],
n: 1, n: 1,
batchFile: null,
jobs: null,
json: false, json: false,
help: false, help: false,
}; };
@@ -110,9 +189,26 @@ function parseArgs(argv: string[]): CliArgs {
continue; continue;
} }
if (a === "--batchfile") {
const v = argv[++i];
if (!v) throw new Error("Missing value for --batchfile");
out.batchFile = v;
continue;
}
if (a === "--jobs") {
const v = argv[++i];
if (!v) throw new Error("Missing value for --jobs");
out.jobs = parseInt(v, 10);
if (isNaN(out.jobs) || out.jobs < 1) throw new Error(`Invalid worker count: ${v}`);
continue;
}
if (a === "--provider") { if (a === "--provider") {
const v = argv[++i]; const v = argv[++i];
if (v !== "google" && v !== "openai" && v !== "dashscope" && v !== "replicate") throw new Error(`Invalid provider: ${v}`); if (v !== "google" && v !== "openai" && v !== "dashscope" && v !== "replicate") {
throw new Error(`Invalid provider: ${v}`);
}
out.provider = v; out.provider = v;
continue; continue;
} }
@@ -228,9 +324,11 @@ function parseSimpleYaml(yaml: string): Partial<ExtendConfig> {
const config: Partial<ExtendConfig> = {}; const config: Partial<ExtendConfig> = {};
const lines = yaml.split("\n"); const lines = yaml.split("\n");
let currentKey: string | null = null; let currentKey: string | null = null;
let currentProvider: Provider | null = null;
for (const line of lines) { for (const line of lines) {
const trimmed = line.trim(); const trimmed = line.trim();
const indent = line.match(/^\s*/)?.[0].length ?? 0;
if (!trimmed || trimmed.startsWith("#")) continue; if (!trimmed || trimmed.startsWith("#")) continue;
if (trimmed.includes(":") && !trimmed.startsWith("-")) { if (trimmed.includes(":") && !trimmed.startsWith("-")) {
@@ -247,18 +345,57 @@ function parseSimpleYaml(yaml: string): Partial<ExtendConfig> {
} else if (key === "default_provider") { } else if (key === "default_provider") {
config.default_provider = value === "null" ? null : (value as Provider); config.default_provider = value === "null" ? null : (value as Provider);
} else if (key === "default_quality") { } else if (key === "default_quality") {
config.default_quality = value === "null" ? null : (value as "normal" | "2k"); config.default_quality = value === "null" ? null : value as "normal" | "2k";
} else if (key === "default_aspect_ratio") { } else if (key === "default_aspect_ratio") {
const cleaned = value.replace(/['"]/g, ""); const cleaned = value.replace(/['"]/g, "");
config.default_aspect_ratio = cleaned === "null" ? null : cleaned; config.default_aspect_ratio = cleaned === "null" ? null : cleaned;
} else if (key === "default_image_size") { } else if (key === "default_image_size") {
config.default_image_size = value === "null" ? null : (value as "1K" | "2K" | "4K"); config.default_image_size = value === "null" ? null : value as "1K" | "2K" | "4K";
} else if (key === "default_model") { } else if (key === "default_model") {
config.default_model = { google: null, openai: null, dashscope: null, replicate: null }; config.default_model = { google: null, openai: null, dashscope: null, replicate: null };
currentKey = "default_model"; currentKey = "default_model";
} else if (currentKey === "default_model" && (key === "google" || key === "openai" || key === "dashscope" || key === "replicate")) { currentProvider = null;
} else if (key === "batch") {
config.batch = {};
currentKey = "batch";
currentProvider = null;
} else if (currentKey === "batch" && indent >= 2 && key === "max_workers") {
config.batch ??= {};
config.batch.max_workers = value === "null" ? null : parseInt(value, 10);
} else if (currentKey === "batch" && indent >= 2 && key === "provider_limits") {
config.batch ??= {};
config.batch.provider_limits ??= {};
currentKey = "provider_limits";
currentProvider = null;
} else if (
currentKey === "provider_limits" &&
indent >= 4 &&
(key === "google" || key === "openai" || key === "dashscope" || key === "replicate")
) {
config.batch ??= {};
config.batch.provider_limits ??= {};
config.batch.provider_limits[key] ??= {};
currentProvider = key;
} else if (
currentKey === "default_model" &&
(key === "google" || key === "openai" || key === "dashscope" || key === "replicate")
) {
const cleaned = value.replace(/['"]/g, ""); const cleaned = value.replace(/['"]/g, "");
config.default_model![key] = cleaned === "null" ? null : cleaned; config.default_model![key] = cleaned === "null" ? null : cleaned;
} else if (
currentKey === "provider_limits" &&
currentProvider &&
indent >= 6 &&
(key === "concurrency" || key === "start_interval_ms")
) {
config.batch ??= {};
config.batch.provider_limits ??= {};
const providerLimit = (config.batch.provider_limits[currentProvider] ??= {});
if (key === "concurrency") {
providerLimit.concurrency = value === "null" ? null : parseInt(value, 10);
} else {
providerLimit.start_interval_ms = value === "null" ? null : parseInt(value, 10);
}
} }
} }
} }
@@ -280,7 +417,6 @@ async function loadExtendConfig(): Promise<Partial<ExtendConfig>> {
const content = await readFile(p, "utf8"); const content = await readFile(p, "utf8");
const yaml = extractYamlFrontMatter(content); const yaml = extractYamlFrontMatter(content);
if (!yaml) continue; if (!yaml) continue;
return parseSimpleYaml(yaml); return parseSimpleYaml(yaml);
} catch { } catch {
continue; continue;
@@ -300,6 +436,57 @@ function mergeConfig(args: CliArgs, extend: Partial<ExtendConfig>): CliArgs {
}; };
} }
function parsePositiveInt(value: string | undefined): number | null {
if (!value) return null;
const parsed = parseInt(value, 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
}
function parsePositiveBatchInt(value: unknown): number | null {
if (value === null || value === undefined) return null;
if (typeof value === "number") {
return Number.isInteger(value) && value > 0 ? value : null;
}
if (typeof value === "string") {
return parsePositiveInt(value);
}
return null;
}
function getConfiguredMaxWorkers(extendConfig: Partial<ExtendConfig>): number {
const envValue = parsePositiveInt(process.env.BAOYU_IMAGE_GEN_MAX_WORKERS);
const configValue = extendConfig.batch?.max_workers ?? null;
return Math.max(1, envValue ?? configValue ?? DEFAULT_MAX_WORKERS);
}
function getConfiguredProviderRateLimits(
extendConfig: Partial<ExtendConfig>
): Record<Provider, ProviderRateLimit> {
const configured: Record<Provider, ProviderRateLimit> = {
replicate: { ...DEFAULT_PROVIDER_RATE_LIMITS.replicate },
google: { ...DEFAULT_PROVIDER_RATE_LIMITS.google },
openai: { ...DEFAULT_PROVIDER_RATE_LIMITS.openai },
dashscope: { ...DEFAULT_PROVIDER_RATE_LIMITS.dashscope },
};
for (const provider of ["replicate", "google", "openai", "dashscope"] as Provider[]) {
const envPrefix = `BAOYU_IMAGE_GEN_${provider.toUpperCase()}`;
const extendLimit = extendConfig.batch?.provider_limits?.[provider];
configured[provider] = {
concurrency:
parsePositiveInt(process.env[`${envPrefix}_CONCURRENCY`]) ??
extendLimit?.concurrency ??
configured[provider].concurrency,
startIntervalMs:
parsePositiveInt(process.env[`${envPrefix}_START_INTERVAL_MS`]) ??
extendLimit?.start_interval_ms ??
configured[provider].startIntervalMs,
};
}
return configured;
}
async function readPromptFromFiles(files: string[]): Promise<string> { async function readPromptFromFiles(files: string[]): Promise<string> {
const parts: string[] = []; const parts: string[] = [];
for (const f of files) { for (const f of files) {
@@ -311,9 +498,12 @@ async function readPromptFromFiles(files: string[]): Promise<string> {
async function readPromptFromStdin(): Promise<string | null> { async function readPromptFromStdin(): Promise<string | null> {
if (process.stdin.isTTY) return null; if (process.stdin.isTTY) return null;
try { try {
const t = await Bun.stdin.text(); const chunks: Buffer[] = [];
const v = t.trim(); for await (const chunk of process.stdin) {
return v.length > 0 ? v : null; chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
}
const value = Buffer.concat(chunks).toString("utf8").trim();
return value.length > 0 ? value : null;
} catch { } catch {
return null; return null;
} }
@@ -327,7 +517,13 @@ function normalizeOutputImagePath(p: string): string {
} }
function detectProvider(args: CliArgs): Provider { function detectProvider(args: CliArgs): Provider {
if (args.referenceImages.length > 0 && args.provider && args.provider !== "google" && args.provider !== "openai" && args.provider !== "replicate") { if (
args.referenceImages.length > 0 &&
args.provider &&
args.provider !== "google" &&
args.provider !== "openai" &&
args.provider !== "replicate"
) {
throw new Error( throw new Error(
"Reference images require a ref-capable provider. Use --provider google (Gemini multimodal), --provider openai (GPT Image edits), or --provider replicate." "Reference images require a ref-capable provider. Use --provider google (Gemini multimodal), --provider openai (GPT Image edits), or --provider replicate."
); );
@@ -349,7 +545,12 @@ function detectProvider(args: CliArgs): Provider {
); );
} }
const available = [hasGoogle && "google", hasOpenai && "openai", hasDashscope && "dashscope", hasReplicate && "replicate"].filter(Boolean) as Provider[]; const available = [
hasGoogle && "google",
hasOpenai && "openai",
hasDashscope && "dashscope",
hasReplicate && "replicate",
].filter(Boolean) as Provider[];
if (available.length === 1) return available[0]!; if (available.length === 1) return available[0]!;
if (available.length > 1) return available[0]!; if (available.length > 1) return available[0]!;
@@ -371,11 +572,6 @@ async function validateReferenceImages(referenceImages: string[]): Promise<void>
} }
} }
type ProviderModule = {
getDefaultModel: () => string;
generateImage: (prompt: string, model: string, args: CliArgs) => Promise<Uint8Array>;
};
function isRetryableGenerationError(error: unknown): boolean { function isRetryableGenerationError(error: unknown): boolean {
const msg = error instanceof Error ? error.message : String(error); const msg = error instanceof Error ? error.message : String(error);
const nonRetryableMarkers = [ const nonRetryableMarkers = [
@@ -384,26 +580,353 @@ function isRetryableGenerationError(error: unknown): boolean {
"only supported", "only supported",
"No API key found", "No API key found",
"is required", "is required",
"Invalid ",
"Unexpected ",
"API error (400)",
"API error (401)",
"API error (402)",
"API error (403)",
"API error (404)",
"temporarily disabled",
]; ];
return !nonRetryableMarkers.some((marker) => msg.includes(marker)); return !nonRetryableMarkers.some((marker) => msg.includes(marker));
} }
async function loadProviderModule(provider: Provider): Promise<ProviderModule> { async function loadProviderModule(provider: Provider): Promise<ProviderModule> {
if (provider === "google") { if (provider === "google") return (await import("./providers/google")) as ProviderModule;
return (await import("./providers/google")) as ProviderModule; if (provider === "dashscope") return (await import("./providers/dashscope")) as ProviderModule;
} if (provider === "replicate") return (await import("./providers/replicate")) as ProviderModule;
if (provider === "dashscope") {
return (await import("./providers/dashscope")) as ProviderModule;
}
if (provider === "replicate") {
return (await import("./providers/replicate")) as ProviderModule;
}
return (await import("./providers/openai")) as ProviderModule; return (await import("./providers/openai")) as ProviderModule;
} }
async function loadPromptForArgs(args: CliArgs): Promise<string | null> {
let prompt: string | null = args.prompt;
if (!prompt && args.promptFiles.length > 0) {
prompt = await readPromptFromFiles(args.promptFiles);
}
return prompt;
}
function getModelForProvider(
provider: Provider,
requestedModel: string | null,
extendConfig: Partial<ExtendConfig>,
providerModule: ProviderModule
): string {
if (requestedModel) return requestedModel;
if (extendConfig.default_model) {
if (provider === "google" && extendConfig.default_model.google) return extendConfig.default_model.google;
if (provider === "openai" && extendConfig.default_model.openai) return extendConfig.default_model.openai;
if (provider === "dashscope" && extendConfig.default_model.dashscope) return extendConfig.default_model.dashscope;
if (provider === "replicate" && extendConfig.default_model.replicate) return extendConfig.default_model.replicate;
}
return providerModule.getDefaultModel();
}
async function prepareSingleTask(args: CliArgs, extendConfig: Partial<ExtendConfig>): Promise<PreparedTask> {
if (!args.quality) args.quality = "2k";
const prompt = (await loadPromptForArgs(args)) ?? (await readPromptFromStdin());
if (!prompt) throw new Error("Prompt is required");
if (!args.imagePath) throw new Error("--image is required");
if (args.referenceImages.length > 0) await validateReferenceImages(args.referenceImages);
const provider = detectProvider(args);
const providerModule = await loadProviderModule(provider);
const model = getModelForProvider(provider, args.model, extendConfig, providerModule);
return {
id: "single",
prompt,
args,
provider,
model,
outputPath: normalizeOutputImagePath(args.imagePath),
providerModule,
};
}
async function loadBatchTasks(batchFilePath: string): Promise<LoadedBatchTasks> {
const resolvedBatchFilePath = path.resolve(batchFilePath);
const content = await readFile(resolvedBatchFilePath, "utf8");
const parsed = JSON.parse(content.replace(/^\uFEFF/, "")) as BatchFile;
const batchDir = path.dirname(resolvedBatchFilePath);
if (Array.isArray(parsed)) {
return {
tasks: parsed,
jobs: null,
batchDir,
};
}
if (parsed && typeof parsed === "object" && Array.isArray(parsed.tasks)) {
const jobs = parsePositiveBatchInt(parsed.jobs);
if (parsed.jobs !== undefined && parsed.jobs !== null && jobs === null) {
throw new Error("Invalid batch file. jobs must be a positive integer when provided.");
}
return {
tasks: parsed.tasks,
jobs,
batchDir,
};
}
throw new Error("Invalid batch file. Expected an array of tasks or an object with a tasks array.");
}
function resolveBatchPath(batchDir: string, filePath: string): string {
return path.isAbsolute(filePath) ? filePath : path.resolve(batchDir, filePath);
}
function createTaskArgs(baseArgs: CliArgs, task: BatchTaskInput, batchDir: string): CliArgs {
return {
...baseArgs,
prompt: task.prompt ?? null,
promptFiles: task.promptFiles ? task.promptFiles.map((filePath) => resolveBatchPath(batchDir, filePath)) : [],
imagePath: task.image ? resolveBatchPath(batchDir, task.image) : null,
provider: task.provider ?? baseArgs.provider ?? null,
model: task.model ?? baseArgs.model ?? null,
aspectRatio: task.ar ?? baseArgs.aspectRatio ?? null,
size: task.size ?? baseArgs.size ?? null,
quality: task.quality ?? baseArgs.quality ?? null,
imageSize: task.imageSize ?? baseArgs.imageSize ?? null,
referenceImages: task.ref ? task.ref.map((filePath) => resolveBatchPath(batchDir, filePath)) : [],
n: task.n ?? baseArgs.n,
batchFile: null,
jobs: baseArgs.jobs,
json: baseArgs.json,
help: false,
};
}
async function prepareBatchTasks(
args: CliArgs,
extendConfig: Partial<ExtendConfig>
): Promise<{ tasks: PreparedTask[]; jobs: number | null }> {
if (!args.batchFile) throw new Error("--batchfile is required in batch mode");
const { tasks: taskInputs, jobs: batchJobs, batchDir } = await loadBatchTasks(args.batchFile);
if (taskInputs.length === 0) throw new Error("Batch file does not contain any tasks.");
const prepared: PreparedTask[] = [];
for (let i = 0; i < taskInputs.length; i++) {
const task = taskInputs[i]!;
const taskArgs = createTaskArgs(args, task, batchDir);
const prompt = await loadPromptForArgs(taskArgs);
if (!prompt) throw new Error(`Task ${i + 1} is missing prompt or promptFiles.`);
if (!taskArgs.imagePath) throw new Error(`Task ${i + 1} is missing image output path.`);
if (taskArgs.referenceImages.length > 0) await validateReferenceImages(taskArgs.referenceImages);
const provider = detectProvider(taskArgs);
const providerModule = await loadProviderModule(provider);
const model = getModelForProvider(provider, taskArgs.model, extendConfig, providerModule);
prepared.push({
id: task.id || `task-${String(i + 1).padStart(2, "0")}`,
prompt,
args: taskArgs,
provider,
model,
outputPath: normalizeOutputImagePath(taskArgs.imagePath),
providerModule,
});
}
return {
tasks: prepared,
jobs: args.jobs ?? batchJobs,
};
}
async function writeImage(outputPath: string, imageData: Uint8Array): Promise<void> {
await mkdir(path.dirname(outputPath), { recursive: true });
await writeFile(outputPath, imageData);
}
async function generatePreparedTask(task: PreparedTask): Promise<TaskResult> {
console.error(`Using ${task.provider} / ${task.model} for ${task.id}`);
console.error(
`Switch model: --model <id> | EXTEND.md default_model.${task.provider} | env ${task.provider.toUpperCase()}_IMAGE_MODEL`
);
let attempts = 0;
while (attempts < MAX_ATTEMPTS) {
attempts += 1;
try {
const imageData = await task.providerModule.generateImage(task.prompt, task.model, task.args);
await writeImage(task.outputPath, imageData);
return {
id: task.id,
provider: task.provider,
model: task.model,
outputPath: task.outputPath,
success: true,
attempts,
error: null,
};
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
const canRetry = attempts < MAX_ATTEMPTS && isRetryableGenerationError(error);
if (canRetry) {
console.error(`[${task.id}] Attempt ${attempts}/${MAX_ATTEMPTS} failed, retrying...`);
continue;
}
return {
id: task.id,
provider: task.provider,
model: task.model,
outputPath: task.outputPath,
success: false,
attempts,
error: message,
};
}
}
return {
id: task.id,
provider: task.provider,
model: task.model,
outputPath: task.outputPath,
success: false,
attempts: MAX_ATTEMPTS,
error: "Unknown failure",
};
}
function createProviderGate(providerRateLimits: Record<Provider, ProviderRateLimit>) {
const state = new Map<Provider, { active: number; lastStartedAt: number }>();
return async function acquire(provider: Provider): Promise<() => void> {
const limit = providerRateLimits[provider];
while (true) {
const current = state.get(provider) ?? { active: 0, lastStartedAt: 0 };
const now = Date.now();
const enoughCapacity = current.active < limit.concurrency;
const enoughGap = now - current.lastStartedAt >= limit.startIntervalMs;
if (enoughCapacity && enoughGap) {
state.set(provider, { active: current.active + 1, lastStartedAt: now });
return () => {
const latest = state.get(provider) ?? { active: 1, lastStartedAt: now };
state.set(provider, {
active: Math.max(0, latest.active - 1),
lastStartedAt: latest.lastStartedAt,
});
};
}
await new Promise((resolve) => setTimeout(resolve, POLL_WAIT_MS));
}
};
}
function getWorkerCount(taskCount: number, jobs: number | null, maxWorkers: number): number {
const requested = jobs ?? Math.min(taskCount, maxWorkers);
return Math.max(1, Math.min(requested, taskCount, maxWorkers));
}
async function runBatchTasks(
tasks: PreparedTask[],
jobs: number | null,
extendConfig: Partial<ExtendConfig>
): Promise<TaskResult[]> {
if (tasks.length === 1) {
return [await generatePreparedTask(tasks[0]!)];
}
const maxWorkers = getConfiguredMaxWorkers(extendConfig);
const providerRateLimits = getConfiguredProviderRateLimits(extendConfig);
const acquireProvider = createProviderGate(providerRateLimits);
const workerCount = getWorkerCount(tasks.length, jobs, maxWorkers);
console.error(`Batch mode: ${tasks.length} tasks, ${workerCount} workers, parallel mode enabled.`);
for (const provider of ["replicate", "google", "openai", "dashscope"] as Provider[]) {
const limit = providerRateLimits[provider];
console.error(`- ${provider}: concurrency=${limit.concurrency}, startIntervalMs=${limit.startIntervalMs}`);
}
let nextIndex = 0;
const results: TaskResult[] = new Array(tasks.length);
const worker = async (): Promise<void> => {
while (true) {
const currentIndex = nextIndex;
nextIndex += 1;
if (currentIndex >= tasks.length) return;
const task = tasks[currentIndex]!;
const release = await acquireProvider(task.provider);
try {
results[currentIndex] = await generatePreparedTask(task);
} finally {
release();
}
}
};
await Promise.all(Array.from({ length: workerCount }, () => worker()));
return results;
}
function printBatchSummary(results: TaskResult[]): void {
const successCount = results.filter((result) => result.success).length;
const failureCount = results.length - successCount;
console.error("");
console.error("Batch generation summary:");
console.error(`- Total: ${results.length}`);
console.error(`- Succeeded: ${successCount}`);
console.error(`- Failed: ${failureCount}`);
if (failureCount > 0) {
console.error("Failure reasons:");
for (const result of results.filter((item) => !item.success)) {
console.error(`- ${result.id}: ${result.error}`);
}
}
}
function emitJson(payload: unknown): void {
console.log(JSON.stringify(payload, null, 2));
}
async function runSingleMode(args: CliArgs, extendConfig: Partial<ExtendConfig>): Promise<void> {
const task = await prepareSingleTask(args, extendConfig);
const result = await generatePreparedTask(task);
if (!result.success) {
throw new Error(result.error || "Generation failed");
}
if (args.json) {
emitJson({
savedImage: result.outputPath,
provider: result.provider,
model: result.model,
attempts: result.attempts,
prompt: task.prompt.slice(0, 200),
});
return;
}
console.log(result.outputPath);
}
async function runBatchMode(args: CliArgs, extendConfig: Partial<ExtendConfig>): Promise<void> {
const { tasks, jobs } = await prepareBatchTasks(args, extendConfig);
const results = await runBatchTasks(tasks, jobs, extendConfig);
printBatchSummary(results);
if (args.json) {
emitJson({
mode: "batch",
total: results.length,
succeeded: results.filter((item) => item.success).length,
failed: results.filter((item) => !item.success).length,
results,
});
}
if (results.some((item) => !item.success)) {
process.exitCode = 1;
}
}
async function main(): Promise<void> { async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2)); const args = parseArgs(process.argv.slice(2));
if (args.help) { if (args.help) {
printUsage(); printUsage();
return; return;
@@ -412,86 +935,18 @@ async function main(): Promise<void> {
await loadEnv(); await loadEnv();
const extendConfig = await loadExtendConfig(); const extendConfig = await loadExtendConfig();
const mergedArgs = mergeConfig(args, extendConfig); const mergedArgs = mergeConfig(args, extendConfig);
if (!mergedArgs.quality) mergedArgs.quality = "2k"; if (!mergedArgs.quality) mergedArgs.quality = "2k";
let prompt: string | null = mergedArgs.prompt; if (mergedArgs.batchFile) {
if (!prompt && mergedArgs.promptFiles.length > 0) prompt = await readPromptFromFiles(mergedArgs.promptFiles); await runBatchMode(mergedArgs, extendConfig);
if (!prompt) prompt = await readPromptFromStdin();
if (!prompt) {
console.error("Error: Prompt is required");
printUsage();
process.exitCode = 1;
return; return;
} }
if (!mergedArgs.imagePath) { await runSingleMode(mergedArgs, extendConfig);
console.error("Error: --image is required");
printUsage();
process.exitCode = 1;
return;
}
if (mergedArgs.referenceImages.length > 0) {
await validateReferenceImages(mergedArgs.referenceImages);
}
const provider = detectProvider(mergedArgs);
const providerModule = await loadProviderModule(provider);
let model = mergedArgs.model;
if (!model && extendConfig.default_model) {
if (provider === "google") model = extendConfig.default_model.google ?? null;
if (provider === "openai") model = extendConfig.default_model.openai ?? null;
if (provider === "dashscope") model = extendConfig.default_model.dashscope ?? null;
if (provider === "replicate") model = extendConfig.default_model.replicate ?? null;
}
model = model || providerModule.getDefaultModel();
const outputPath = normalizeOutputImagePath(mergedArgs.imagePath);
let imageData: Uint8Array;
let retried = false;
while (true) {
try {
imageData = await providerModule.generateImage(prompt, model, mergedArgs);
break;
} catch (e) {
if (!retried && isRetryableGenerationError(e)) {
retried = true;
console.error("Generation failed, retrying...");
continue;
}
throw e;
}
}
const dir = path.dirname(outputPath);
await mkdir(dir, { recursive: true });
await writeFile(outputPath, imageData);
if (mergedArgs.json) {
console.log(
JSON.stringify(
{
savedImage: outputPath,
provider,
model,
prompt: prompt.slice(0, 200),
},
null,
2
)
);
} else {
console.log(outputPath);
}
} }
main().catch((e) => { main().catch((error) => {
const msg = e instanceof Error ? e.message : String(e); const message = error instanceof Error ? error.message : String(error);
console.error(msg); console.error(message);
process.exit(1); process.exit(1);
}); });
@@ -68,7 +68,11 @@ export async function generateImage(
const baseURL = process.env.OPENAI_BASE_URL || "https://api.openai.com/v1"; const baseURL = process.env.OPENAI_BASE_URL || "https://api.openai.com/v1";
const apiKey = process.env.OPENAI_API_KEY; const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) throw new Error("OPENAI_API_KEY is required"); if (!apiKey) {
throw new Error(
"OPENAI_API_KEY is required. Codex/ChatGPT desktop login does not automatically grant OpenAI Images API access to this script."
);
}
if (process.env.OPENAI_IMAGE_USE_CHAT === "true") { if (process.env.OPENAI_IMAGE_USE_CHAT === "true") {
return generateWithChatCompletions(baseURL, apiKey, prompt, model); return generateWithChatCompletions(baseURL, apiKey, prompt, model);
@@ -36,22 +36,24 @@ function buildInput(prompt: string, args: CliArgs, referenceImages: string[]): R
if (args.aspectRatio) { if (args.aspectRatio) {
input.aspect_ratio = args.aspectRatio; input.aspect_ratio = args.aspectRatio;
} else if (referenceImages.length > 0) {
input.aspect_ratio = "match_input_image";
} }
if (args.n > 1) { if (args.n > 1) {
input.number_of_images = args.n; input.number_of_images = args.n;
} }
if (args.quality === "normal") {
input.resolution = "1K";
} else if (args.quality === "2k") {
input.resolution = "2K";
}
input.output_format = "png"; input.output_format = "png";
if (referenceImages.length > 0) { if (referenceImages.length > 0) {
if (referenceImages.length === 1) { input.image_input = referenceImages;
input.image = referenceImages[0];
} else {
for (let i = 0; i < referenceImages.length; i++) {
input[`image${i > 0 ? i + 1 : ""}`] = referenceImages[i];
}
}
} }
return input; return input;
+36
View File
@@ -13,10 +13,34 @@ export type CliArgs = {
imageSize: string | null; imageSize: string | null;
referenceImages: string[]; referenceImages: string[];
n: number; n: number;
batchFile: string | null;
jobs: number | null;
json: boolean; json: boolean;
help: boolean; help: boolean;
}; };
export type BatchTaskInput = {
id?: string;
prompt?: string | null;
promptFiles?: string[];
image?: string;
provider?: Provider | null;
model?: string | null;
ar?: string | null;
size?: string | null;
quality?: Quality | null;
imageSize?: "1K" | "2K" | "4K" | null;
ref?: string[];
n?: number;
};
export type BatchFile =
| BatchTaskInput[]
| {
tasks: BatchTaskInput[];
jobs?: number | null;
};
export type ExtendConfig = { export type ExtendConfig = {
version: number; version: number;
default_provider: Provider | null; default_provider: Provider | null;
@@ -29,4 +53,16 @@ export type ExtendConfig = {
dashscope: string | null; dashscope: string | null;
replicate: string | null; replicate: string | null;
}; };
batch?: {
max_workers?: number | null;
provider_limits?: Partial<
Record<
Provider,
{
concurrency?: number | null;
start_interval_ms?: number | null;
}
>
>;
};
}; };
@@ -0,0 +1,14 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-post-to-wechat-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
},
},
},
"packages": {
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
}
}
+155 -215
View File
@@ -1,172 +1,83 @@
import { spawn } from 'node:child_process'; import { execSync, type ChildProcess } from 'node:child_process';
import fs from 'node:fs';
import { mkdir } from 'node:fs/promises';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
export function sleep(ms: number): Promise<void> { import {
return new Promise((resolve) => setTimeout(resolve, ms)); CdpConnection,
findChromeExecutable as findChromeExecutableBase,
findExistingChromeDebugPort as findExistingChromeDebugPortBase,
getFreePort as getFreePortBase,
launchChrome as launchChromeBase,
resolveSharedChromeProfileDir,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
export { CdpConnection, sleep, waitForChromeDebugPort };
const CHROME_CANDIDATES_FULL: PlatformCandidates = {
darwin: [
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
],
win32: [
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
],
default: [
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
],
};
let wslHome: string | null | undefined;
function getWslWindowsHome(): string | null {
if (wslHome !== undefined) return wslHome;
if (!process.env.WSL_DISTRO_NAME) {
wslHome = null;
return null;
}
try {
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', {
encoding: 'utf-8',
timeout: 5_000,
}).trim().replace(/\r/g, '');
wslHome = execSync(`wslpath -u "${raw}"`, {
encoding: 'utf-8',
timeout: 5_000,
}).trim() || null;
} catch {
wslHome = null;
}
return wslHome;
} }
export async function getFreePort(): Promise<number> { export async function getFreePort(): Promise<number> {
return new Promise((resolve, reject) => { return await getFreePortBase('WECHAT_BROWSER_DEBUG_PORT');
const server = net.createServer(); }
server.unref();
server.on('error', reject); export function findChromeExecutable(chromePathOverride?: string): string | undefined {
server.listen(0, '127.0.0.1', () => { if (chromePathOverride?.trim()) return chromePathOverride.trim();
const address = server.address(); return findChromeExecutableBase({
if (!address || typeof address === 'string') { candidates: CHROME_CANDIDATES_FULL,
server.close(() => reject(new Error('Unable to allocate a free TCP port.'))); envNames: ['WECHAT_BROWSER_CHROME_PATH'],
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
}); });
} }
export function findChromeExecutable(): string | undefined {
const override = process.env.WECHAT_BROWSER_CHROME_PATH?.trim();
if (override && fs.existsSync(override)) return override;
const candidates: string[] = [];
switch (process.platform) {
case 'darwin':
candidates.push(
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
);
break;
case 'win32':
candidates.push(
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
);
break;
default:
candidates.push('/usr/bin/google-chrome', '/usr/bin/chromium');
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return undefined;
}
export function getDefaultProfileDir(): string { export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim(); return resolveSharedChromeProfileDir({
if (override) return path.resolve(override); envNames: ['BAOYU_CHROME_PROFILE_DIR', 'WECHAT_BROWSER_PROFILE_DIR'],
const base = process.platform === 'darwin' wslWindowsHome: getWslWindowsHome(),
? path.join(os.homedir(), 'Library', 'Application Support') });
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
async function fetchJson<T = unknown>(url: string): Promise<T> {
const res = await fetch(url, { redirect: 'follow' });
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
}
async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error('Missing webSocketDebuggerUrl');
} catch (error) {
lastError = error;
}
await sleep(200);
}
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, { resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) handlers.forEach((h) => h(msg.params));
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) this.eventHandlers.set(method, new Set());
this.eventHandlers.get(method)!.add(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? 15_000;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0 ? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs) : null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
} }
export interface ChromeSession { export interface ChromeSession {
@@ -177,56 +88,38 @@ export interface ChromeSession {
export async function tryConnectExisting(port: number): Promise<CdpConnection | null> { export async function tryConnectExisting(port: number): Promise<CdpConnection | null> {
try { try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`); const wsUrl = await waitForChromeDebugPort(port, 5_000, { includeLastError: true });
if (version.webSocketDebuggerUrl) { return await CdpConnection.connect(wsUrl, 5_000);
const cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, 5_000); } catch {
return cdp; return null;
} }
} catch {}
return null;
} }
export async function findExistingChromeDebugPort(): Promise<number | null> { export async function findExistingChromeDebugPort(profileDir = getDefaultProfileDir()): Promise<number | null> {
if (process.platform !== 'darwin' && process.platform !== 'linux') return null; return await findExistingChromeDebugPortBase({ profileDir });
try {
const { execSync } = await import('node:child_process');
const cmd = process.platform === 'darwin'
? `lsof -nP -iTCP -sTCP:LISTEN 2>/dev/null | grep -i 'google\\|chrome' | awk '{print $9}' | sed 's/.*://'`
: `ss -tlnp 2>/dev/null | grep -i chrome | awk '{print $4}' | sed 's/.*://'`;
const output = execSync(cmd, { encoding: 'utf-8', timeout: 5_000 }).trim();
if (!output) return null;
const ports = output.split('\n').map(p => parseInt(p, 10)).filter(p => !isNaN(p) && p > 0);
for (const port of ports) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return port;
} catch {}
}
} catch {}
return null;
} }
export async function launchChrome(url: string, profileDir?: string): Promise<{ cdp: CdpConnection; chrome: ReturnType<typeof spawn> }> { export async function launchChrome(
const chromePath = findChromeExecutable(); url: string,
profileDir?: string,
chromePathOverride?: string,
): Promise<{ cdp: CdpConnection; chrome: ChildProcess }> {
const chromePath = findChromeExecutable(chromePathOverride);
if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.');
const profile = profileDir ?? getDefaultProfileDir(); const profile = profileDir ?? getDefaultProfileDir();
await mkdir(profile, { recursive: true });
const port = await getFreePort(); const port = await getFreePort();
console.log(`[cdp] Launching Chrome (profile: ${profile})`); console.log(`[cdp] Launching Chrome (profile: ${profile})`);
const chrome = spawn(chromePath, [ const chrome = await launchChromeBase({
`--remote-debugging-port=${port}`, chromePath,
`--user-data-dir=${profile}`, profileDir: profile,
'--no-first-run', port,
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
url, url,
], { stdio: 'ignore' }); extraArgs: ['--disable-blink-features=AutomationControlled', '--start-maximized'],
});
const wsUrl = await waitForChromeDebugPort(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
const cdp = await CdpConnection.connect(wsUrl, 30_000); const cdp = await CdpConnection.connect(wsUrl, 30_000);
return { cdp, chrome }; return { cdp, chrome };
@@ -234,11 +127,14 @@ export async function launchChrome(url: string, profileDir?: string): Promise<{
export async function getPageSession(cdp: CdpConnection, urlPattern: string): Promise<ChromeSession> { export async function getPageSession(cdp: CdpConnection, urlPattern: string): Promise<ChromeSession> {
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes(urlPattern)); const pageTarget = targets.targetInfos.find((target) => target.type === 'page' && target.url.includes(urlPattern));
if (!pageTarget) throw new Error(`Page not found: ${urlPattern}`); if (!pageTarget) throw new Error(`Page not found: ${urlPattern}`);
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', {
targetId: pageTarget.targetId,
flatten: true,
});
await cdp.send('Page.enable', {}, { sessionId }); await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId }); await cdp.send('Runtime.enable', {}, { sessionId });
@@ -247,11 +143,20 @@ export async function getPageSession(cdp: CdpConnection, urlPattern: string): Pr
return { cdp, sessionId, targetId: pageTarget.targetId }; return { cdp, sessionId, targetId: pageTarget.targetId };
} }
export async function waitForNewTab(cdp: CdpConnection, initialIds: Set<string>, urlPattern: string, timeoutMs = 30_000): Promise<string> { export async function waitForNewTab(
cdp: CdpConnection,
initialIds: Set<string>,
urlPattern: string,
timeoutMs = 30_000,
): Promise<string> {
const start = Date.now(); const start = Date.now();
while (Date.now() - start < timeoutMs) { while (Date.now() - start < timeoutMs) {
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
const newTab = targets.targetInfos.find(t => t.type === 'page' && !initialIds.has(t.targetId) && t.url.includes(urlPattern)); const newTab = targets.targetInfos.find((target) => (
target.type === 'page' &&
!initialIds.has(target.targetId) &&
target.url.includes(urlPattern)
));
if (newTab) return newTab.targetId; if (newTab) return newTab.targetId;
await sleep(500); await sleep(500);
} }
@@ -259,7 +164,7 @@ export async function waitForNewTab(cdp: CdpConnection, initialIds: Set<string>,
} }
export async function clickElement(session: ChromeSession, selector: string): Promise<void> { export async function clickElement(session: ChromeSession, selector: string): Promise<void> {
const posResult = await session.cdp.send<{ result: { value: string } }>('Runtime.evaluate', { const position = await session.cdp.send<{ result: { value: string } }>('Runtime.evaluate', {
expression: ` expression: `
(function() { (function() {
const el = document.querySelector('${selector}'); const el = document.querySelector('${selector}');
@@ -272,23 +177,46 @@ export async function clickElement(session: ChromeSession, selector: string): Pr
returnByValue: true, returnByValue: true,
}, { sessionId: session.sessionId }); }, { sessionId: session.sessionId });
if (posResult.result.value === 'null') throw new Error(`Element not found: ${selector}`); if (position.result.value === 'null') throw new Error(`Element not found: ${selector}`);
const pos = JSON.parse(posResult.result.value); const pos = JSON.parse(position.result.value);
await session.cdp.send('Input.dispatchMouseEvent', { type: 'mousePressed', x: pos.x, y: pos.y, button: 'left', clickCount: 1 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchMouseEvent', {
type: 'mousePressed',
x: pos.x,
y: pos.y,
button: 'left',
clickCount: 1,
}, { sessionId: session.sessionId });
await sleep(50); await sleep(50);
await session.cdp.send('Input.dispatchMouseEvent', { type: 'mouseReleased', x: pos.x, y: pos.y, button: 'left', clickCount: 1 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchMouseEvent', {
type: 'mouseReleased',
x: pos.x,
y: pos.y,
button: 'left',
clickCount: 1,
}, { sessionId: session.sessionId });
} }
export async function typeText(session: ChromeSession, text: string): Promise<void> { export async function typeText(session: ChromeSession, text: string): Promise<void> {
const lines = text.split('\n'); const lines = text.split('\n');
for (let i = 0; i < lines.length; i++) { for (let index = 0; index < lines.length; index += 1) {
if (lines[i].length > 0) { const line = lines[index];
await session.cdp.send('Input.insertText', { text: lines[i] }, { sessionId: session.sessionId }); if (line.length > 0) {
await session.cdp.send('Input.insertText', { text: line }, { sessionId: session.sessionId });
} }
if (i < lines.length - 1) { if (index < lines.length - 1) {
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyDown', key: 'Enter', code: 'Enter', windowsVirtualKeyCode: 13 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchKeyEvent', {
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyUp', key: 'Enter', code: 'Enter', windowsVirtualKeyCode: 13 }, { sessionId: session.sessionId }); type: 'keyDown',
key: 'Enter',
code: 'Enter',
windowsVirtualKeyCode: 13,
}, { sessionId: session.sessionId });
await session.cdp.send('Input.dispatchKeyEvent', {
type: 'keyUp',
key: 'Enter',
code: 'Enter',
windowsVirtualKeyCode: 13,
}, { sessionId: session.sessionId });
} }
await sleep(30); await sleep(30);
} }
@@ -296,8 +224,20 @@ export async function typeText(session: ChromeSession, text: string): Promise<vo
export async function pasteFromClipboard(session: ChromeSession): Promise<void> { export async function pasteFromClipboard(session: ChromeSession): Promise<void> {
const modifiers = process.platform === 'darwin' ? 4 : 2; const modifiers = process.platform === 'darwin' ? 4 : 2;
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyDown', key: 'v', code: 'KeyV', modifiers, windowsVirtualKeyCode: 86 }, { sessionId: session.sessionId }); await session.cdp.send('Input.dispatchKeyEvent', {
await session.cdp.send('Input.dispatchKeyEvent', { type: 'keyUp', key: 'v', code: 'KeyV', modifiers, windowsVirtualKeyCode: 86 }, { sessionId: session.sessionId }); type: 'keyDown',
key: 'v',
code: 'KeyV',
modifiers,
windowsVirtualKeyCode: 86,
}, { sessionId: session.sessionId });
await session.cdp.send('Input.dispatchKeyEvent', {
type: 'keyUp',
key: 'v',
code: 'KeyV',
modifiers,
windowsVirtualKeyCode: 86,
}, { sessionId: session.sessionId });
} }
export async function evaluate<T = unknown>(session: ChromeSession, expression: string): Promise<T> { export async function evaluate<T = unknown>(session: ChromeSession, expression: string): Promise<T> {
@@ -0,0 +1,8 @@
{
"name": "baoyu-post-to-wechat-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
@@ -1,11 +1,16 @@
import { execSync, spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir, readdir } from 'node:fs/promises'; import { readdir } from 'node:fs/promises';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
import {
CdpConnection,
findChromeExecutable,
getDefaultProfileDir,
launchChrome,
sleep,
} from './cdp.ts';
const WECHAT_URL = 'https://mp.weixin.qq.com/'; const WECHAT_URL = 'https://mp.weixin.qq.com/';
interface MarkdownMeta { interface MarkdownMeta {
@@ -104,195 +109,6 @@ async function loadImagesFromDir(dir: string): Promise<string[]> {
return images; return images;
} }
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.WECHAT_BROWSER_DEBUG_PORT || '', 10);
if (fixed > 0) return fixed;
return new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
function findChromeExecutable(): string | undefined {
const override = process.env.WECHAT_BROWSER_CHROME_PATH?.trim();
if (override && fs.existsSync(override)) return override;
const candidates: string[] = [];
switch (process.platform) {
case 'darwin':
candidates.push(
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
);
break;
case 'win32':
candidates.push(
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
);
break;
default:
candidates.push(
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
);
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return undefined;
}
let _wslHome: string | null | undefined;
function getWslWindowsHome(): string | null {
if (_wslHome !== undefined) return _wslHome;
if (!process.env.WSL_DISTRO_NAME) { _wslHome = null; return null; }
try {
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', { encoding: 'utf-8', timeout: 5000 }).trim().replace(/\r/g, '');
_wslHome = execSync(`wslpath -u "${raw}"`, { encoding: 'utf-8', timeout: 5000 }).trim() || null;
} catch { _wslHome = null; }
return _wslHome;
}
function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.WECHAT_BROWSER_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
const wslHome = getWslWindowsHome();
if (wslHome) return path.join(wslHome, '.local', 'share', 'baoyu-skills', 'chrome-profile');
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
async function fetchJson<T = unknown>(url: string): Promise<T> {
const res = await fetch(url, { redirect: 'follow' });
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
}
async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error('Missing webSocketDebuggerUrl');
} catch (error) {
lastError = error;
}
await sleep(200);
}
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, { resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) handlers.forEach((h) => h(msg.params));
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) this.eventHandlers.set(method, new Set());
this.eventHandlers.get(method)!.add(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? 15_000;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0 ? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs) : null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
}
interface WeChatBrowserOptions { interface WeChatBrowserOptions {
title?: string; title?: string;
content?: string; content?: string;
@@ -348,29 +164,18 @@ export async function postToWeChat(options: WeChatBrowserOptions): Promise<void>
if (!fs.existsSync(img)) throw new Error(`Image not found: ${img}`); if (!fs.existsSync(img)) throw new Error(`Image not found: ${img}`);
} }
const chromePath = options.chromePath ?? findChromeExecutable(); const chromePath = findChromeExecutable(options.chromePath);
if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WECHAT_BROWSER_CHROME_PATH env var.');
await mkdir(profileDir, { recursive: true });
const port = await getFreePort();
console.log(`[wechat-browser] Launching Chrome (profile: ${profileDir})`); console.log(`[wechat-browser] Launching Chrome (profile: ${profileDir})`);
const chrome = spawn(chromePath, [ const launched = await launchChrome(WECHAT_URL, profileDir, chromePath);
`--remote-debugging-port=${port}`, const chrome = launched.chrome;
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
WECHAT_URL,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000); cdp = launched.cdp;
cdp = await CdpConnection.connect(wsUrl, 30_000);
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('mp.weixin.qq.com')); let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('mp.weixin.qq.com'));
@@ -0,0 +1,31 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-post-to-weibo-scripts",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2",
"highlight.js": "^11.11.1",
"marked": "^15.0.6",
},
},
},
"packages": {
"argparse": ["argparse@1.0.10", "", { "dependencies": { "sprintf-js": "~1.0.2" } }, "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg=="],
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
"esprima": ["esprima@4.0.1", "", { "bin": { "esparse": "./bin/esparse.js", "esvalidate": "./bin/esvalidate.js" } }, "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A=="],
"front-matter": ["front-matter@4.0.2", "", { "dependencies": { "js-yaml": "^3.13.1" } }, "sha512-I8ZuJ/qG92NWX8i5x1Y8qyj3vizhXS31OxjKDu3LKP+7/qBgfIKValiZIEwoVoJKUHlhWtYrktkxV1XsX+pPlg=="],
"highlight.js": ["highlight.js@11.11.1", "", {}, "sha512-Xwwo44whKBVCYoliBQwaPvtd/2tYFkRQtXDWj1nackaV2JPXx3L0+Jvd8/qCJ2p+ML0/XVkJ2q+Mr+UVdpJK5w=="],
"js-yaml": ["js-yaml@3.14.2", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg=="],
"marked": ["marked@15.0.12", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-8dD6FusOQSrpv9Z1rdNMdlSgQOIP880DHqnohobOmYLElGEqAL/JvxvuxZO16r4HtjTlfPRDC1hbvxC9dPN2nA=="],
"sprintf-js": ["sprintf-js@1.0.3", "", {}, "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g=="],
}
}
@@ -446,7 +446,9 @@ Options:
} }
} }
await main().catch((err) => { if (import.meta.main ?? (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename ?? ''))) {
console.error(`Error: ${err instanceof Error ? err.message : String(err)}`); await main().catch((err) => {
process.exit(1); console.error(`Error: ${err instanceof Error ? err.message : String(err)}`);
}); process.exit(1);
});
}
@@ -0,0 +1,11 @@
{
"name": "baoyu-post-to-weibo-scripts",
"private": true,
"type": "module",
"dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2",
"highlight.js": "^11.11.1",
"marked": "^15.0.6"
}
}
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
@@ -1,9 +1,7 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir, writeFile } from 'node:fs/promises'; import { mkdir, writeFile } from 'node:fs/promises';
import os from 'node:os'; import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process';
import { import {
CdpConnection, CdpConnection,
copyHtmlToClipboard, copyHtmlToClipboard,
@@ -11,7 +9,7 @@ import {
findChromeExecutable, findChromeExecutable,
findExistingChromeDebugPort, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, launchChrome,
pasteFromClipboard, pasteFromClipboard,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
@@ -74,36 +72,17 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
// Try reusing an existing Chrome instance with the same profile // Try reusing an existing Chrome instance with the same profile
const existingPort = findExistingChromeDebugPort(profileDir); const existingPort = await findExistingChromeDebugPort(profileDir);
let port: number; let port: number;
let launched = false;
if (existingPort) { if (existingPort) {
console.log(`[weibo-article] Found existing Chrome on port ${existingPort}, reusing...`); console.log(`[weibo-article] Found existing Chrome on port ${existingPort}, reusing...`);
port = existingPort; port = existingPort;
} else { } else {
const chromePath = options.chromePath ?? findChromeExecutable(); const chromePath = findChromeExecutable(options.chromePath);
if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
port = await getFreePort(); port = await launchChrome(WEIBO_ARTICLE_URL, profileDir, chromePath);
console.log(`[weibo-article] Launching Chrome...`);
const chromeArgs = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
WEIBO_ARTICLE_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
launched = true;
} }
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
@@ -1,4 +1,3 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import path from 'node:path'; import path from 'node:path';
@@ -8,8 +7,8 @@ import {
findChromeExecutable, findChromeExecutable,
findExistingChromeDebugPort, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort,
killChromeByProfile, killChromeByProfile,
launchChrome as launchWeiboChrome,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
} from './weibo-utils.js'; } from './weibo-utils.js';
@@ -37,32 +36,11 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const chromePath = options.chromePath ?? findChromeExecutable(); const chromePath = findChromeExecutable(options.chromePath);
if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.'); if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
const launchChrome = async (): Promise<number> => {
const port = await getFreePort();
console.log(`[weibo-post] Launching Chrome (profile: ${profileDir})`);
const chromeArgs = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
WEIBO_HOME_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
return port;
};
let port: number; let port: number;
const existingPort = findExistingChromeDebugPort(profileDir); const existingPort = await findExistingChromeDebugPort(profileDir);
if (existingPort) { if (existingPort) {
console.log(`[weibo-post] Found existing Chrome on port ${existingPort}, checking health...`); console.log(`[weibo-post] Found existing Chrome on port ${existingPort}, checking health...`);
@@ -77,10 +55,10 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
console.log('[weibo-post] Existing Chrome unresponsive, restarting...'); console.log('[weibo-post] Existing Chrome unresponsive, restarting...');
killChromeByProfile(profileDir); killChromeByProfile(profileDir);
await sleep(2000); await sleep(2000);
port = await launchChrome(); port = await launchWeiboChrome(WEIBO_HOME_URL, profileDir, chromePath);
} }
} else { } else {
port = await launchChrome(); port = await launchWeiboChrome(WEIBO_HOME_URL, profileDir, chromePath);
} }
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
+69 -160
View File
@@ -1,12 +1,23 @@
import { spawnSync } from 'node:child_process'; import { execSync, spawnSync } from 'node:child_process';
import fs from 'node:fs';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
export const CHROME_CANDIDATES = { import {
CdpConnection,
findChromeExecutable as findChromeExecutableBase,
findExistingChromeDebugPort as findExistingChromeDebugPortBase,
getFreePort as getFreePortBase,
launchChrome as launchChromeBase,
resolveSharedChromeProfileDir,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
export { CdpConnection, sleep, waitForChromeDebugPort };
export const CHROME_CANDIDATES: PlatformCandidates = {
darwin: [ darwin: [
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary', '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
@@ -23,183 +34,81 @@ export const CHROME_CANDIDATES = {
], ],
}; };
export function findChromeExecutable(): string | undefined { let wslHome: string | null | undefined;
const override = process.env.WEIBO_BROWSER_CHROME_PATH?.trim(); function getWslWindowsHome(): string | null {
if (override && fs.existsSync(override)) return override; if (wslHome !== undefined) return wslHome;
if (!process.env.WSL_DISTRO_NAME) {
const candidates = process.platform === 'darwin' wslHome = null;
? CHROME_CANDIDATES.darwin return null;
: process.platform === 'win32'
? CHROME_CANDIDATES.win32
: CHROME_CANDIDATES.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
} }
return undefined; try {
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', {
encoding: 'utf-8',
timeout: 5_000,
}).trim().replace(/\r/g, '');
wslHome = execSync(`wslpath -u "${raw}"`, {
encoding: 'utf-8',
timeout: 5_000,
}).trim() || null;
} catch {
wslHome = null;
}
return wslHome;
} }
export function findExistingChromeDebugPort(profileDir: string): number | null { export function findChromeExecutable(chromePathOverride?: string): string | undefined {
try { if (chromePathOverride?.trim()) return chromePathOverride.trim();
const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5000 }); return findChromeExecutableBase({
if (result.status !== 0 || !result.stdout) return null; candidates: CHROME_CANDIDATES,
const lines = result.stdout.split('\n'); envNames: ['WEIBO_BROWSER_CHROME_PATH'],
for (const line of lines) { });
if (!line.includes('--remote-debugging-port=') || !line.includes(profileDir)) continue; }
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
if (portMatch) return Number(portMatch[1]); export async function findExistingChromeDebugPort(profileDir: string): Promise<number | null> {
} return await findExistingChromeDebugPortBase({ profileDir });
} catch {}
return null;
} }
export function killChromeByProfile(profileDir: string): void { export function killChromeByProfile(profileDir: string): void {
try { try {
const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5000 }); const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return; if (result.status !== 0 || !result.stdout) return;
for (const line of result.stdout.split('\n')) { for (const line of result.stdout.split('\n')) {
if (!line.includes(profileDir) || !line.includes('--remote-debugging-port=')) continue; if (!line.includes(profileDir) || !line.includes('--remote-debugging-port=')) continue;
const pidMatch = line.trim().split(/\s+/)[1]; const pid = line.trim().split(/\s+/)[1];
if (pidMatch) { if (pid) {
try { process.kill(Number(pidMatch), 'SIGTERM'); } catch {} try {
process.kill(Number(pid), 'SIGTERM');
} catch {}
} }
} }
} catch {} } catch {}
} }
export function getDefaultProfileDir(): string { export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.WEIBO_BROWSER_PROFILE_DIR?.trim(); return resolveSharedChromeProfileDir({
if (override) return path.resolve(override); envNames: ['BAOYU_CHROME_PROFILE_DIR', 'WEIBO_BROWSER_PROFILE_DIR'],
const base = process.platform === 'darwin' wslWindowsHome: getWslWindowsHome(),
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.WEIBO_BROWSER_DEBUG_PORT || '', 10);
if (fixed > 0) return fixed;
return new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
}); });
} }
async function fetchJson<T = unknown>(url: string): Promise<T> { export async function getFreePort(): Promise<number> {
const res = await fetch(url, { redirect: 'follow' }); return await getFreePortBase('WEIBO_BROWSER_DEBUG_PORT');
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
} }
export async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> { export async function launchChrome(url: string, profileDir: string, chromePathOverride?: string): Promise<number> {
const start = Date.now(); const chromePath = findChromeExecutable(chromePathOverride);
let lastError: unknown = null; if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
while (Date.now() - start < timeoutMs) { const port = await getFreePort();
try { console.log(`[weibo-cdp] Launching Chrome (profile: ${profileDir})`);
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`); await launchChromeBase({
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl; chromePath,
lastError = new Error('Missing webSocketDebuggerUrl'); profileDir,
} catch (error) { port,
lastError = error; url,
} extraArgs: ['--disable-blink-features=AutomationControlled', '--start-maximized'],
await sleep(200); });
} return port;
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, options?: { defaultTimeoutMs?: number }) {
this.ws = ws;
this.defaultTimeoutMs = options?.defaultTimeoutMs ?? 15_000;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number, options?: { defaultTimeoutMs?: number }): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws, options);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
} }
export function getScriptDir(): string { export function getScriptDir(): string {
+3
View File
@@ -4,6 +4,7 @@
"": { "": {
"name": "baoyu-post-to-x-scripts", "name": "baoyu-post-to-x-scripts",
"dependencies": { "dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2", "front-matter": "^4.0.2",
"highlight.js": "^11.11.1", "highlight.js": "^11.11.1",
"marked": "^15.0.6", "marked": "^15.0.6",
@@ -27,6 +28,8 @@
"bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="], "bail": ["bail@2.0.2", "", {}, "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw=="],
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
"character-entities": ["character-entities@2.0.2", "", {}, "sha512-shx7oQ0Awen/BRIdkjkvz54PnEEI/EjwXDSIZp86/KKdbafHh1Df/RYGBhn4hbe2+uKC9FnT5UCEdyPz3ai9hQ=="], "character-entities": ["character-entities@2.0.2", "", {}, "sha512-shx7oQ0Awen/BRIdkjkvz54PnEEI/EjwXDSIZp86/KKdbafHh1Df/RYGBhn4hbe2+uKC9FnT5UCEdyPz3ai9hQ=="],
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="], "debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
@@ -3,6 +3,7 @@
"private": true, "private": true,
"type": "module", "type": "module",
"dependencies": { "dependencies": {
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"front-matter": "^4.0.2", "front-matter": "^4.0.2",
"highlight.js": "^11.11.1", "highlight.js": "^11.11.1",
"marked": "^15.0.6", "marked": "^15.0.6",
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}
+19 -58
View File
@@ -1,9 +1,7 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir, writeFile } from 'node:fs/promises'; import { mkdir, writeFile } from 'node:fs/promises';
import os from 'node:os'; import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process';
import { parseMarkdown } from './md-to-html.js'; import { parseMarkdown } from './md-to-html.js';
import { import {
@@ -11,9 +9,10 @@ import {
CdpConnection, CdpConnection,
copyHtmlToClipboard, copyHtmlToClipboard,
copyImageToClipboard, copyImageToClipboard,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, launchChrome,
openPageSession,
pasteFromClipboard, pasteFromClipboard,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
@@ -61,25 +60,6 @@ interface ArticleOptions {
chromePath?: string; chromePath?: string;
} }
async function findExistingDebugPort(profileDir: string): Promise<number | null> {
const portFile = path.join(profileDir, 'DevToolsActivePort');
if (!fs.existsSync(portFile)) return null;
try {
const content = fs.readFileSync(portFile, 'utf-8').trim();
if (!content) return null;
const [portLine] = content.split(/\r?\n/);
const port = Number(portLine);
if (!Number.isFinite(port) || port <= 0) return null;
// Verify the port is actually active.
await waitForChromeDebugPort(port, 1500, { includeLastError: true });
return port;
} catch {
return null;
}
}
export async function publishArticle(options: ArticleOptions): Promise<void> { export async function publishArticle(options: ArticleOptions): Promise<void> {
const { markdownPath, submit = false, profileDir = getDefaultProfileDir() } = options; const { markdownPath, submit = false, profileDir = getDefaultProfileDir() } = options;
@@ -98,32 +78,17 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
await writeFile(htmlPath, parsed.html, 'utf-8'); await writeFile(htmlPath, parsed.html, 'utf-8');
console.log(`[x-article] HTML saved to: ${htmlPath}`); console.log(`[x-article] HTML saved to: ${htmlPath}`);
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_BASIC);
if (!chromePath) throw new Error('Chrome not found');
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const existingPort = await findExistingDebugPort(profileDir); const existingPort = await findExistingChromeDebugPort(profileDir);
const port = existingPort ?? await getFreePort(); const reusing = existingPort !== null;
let port = existingPort ?? 0;
if (existingPort) { if (reusing) {
console.log(`[x-article] Reusing existing Chrome instance on port ${port}`); console.log(`[x-article] Reusing existing Chrome instance on port ${port}`);
} else { } else {
console.log(`[x-article] Launching Chrome...`); console.log(`[x-article] Launching Chrome...`);
const chromeArgs = [ const launched = await launchChrome(X_ARTICLES_URL, profileDir, CHROME_CANDIDATES_BASIC, options.chromePath);
`--remote-debugging-port=${port}`, port = launched.port;
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
X_ARTICLES_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
} }
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
@@ -132,20 +97,16 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 60_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 60_000 });
// Get page target const page = await openPageSession({
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); cdp,
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.startsWith(X_ARTICLES_URL)); reusing,
url: X_ARTICLES_URL,
if (!pageTarget) { matchTarget: (target) => target.type === 'page' && target.url.startsWith(X_ARTICLES_URL),
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: X_ARTICLES_URL }); enablePage: true,
pageTarget = { targetId, url: X_ARTICLES_URL, type: 'page' }; enableRuntime: true,
} enableDom: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = page;
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('DOM.enable', {}, { sessionId });
console.log('[x-article] Waiting for articles page...'); console.log('[x-article] Waiting for articles page...');
await sleep(1000); await sleep(1000);
+26 -30
View File
@@ -1,4 +1,3 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import process from 'node:process'; import process from 'node:process';
@@ -6,9 +5,10 @@ import {
CHROME_CANDIDATES_FULL, CHROME_CANDIDATES_FULL,
CdpConnection, CdpConnection,
copyImageToClipboard, copyImageToClipboard,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, launchChrome,
openPageSession,
pasteFromClipboard, pasteFromClipboard,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
@@ -28,23 +28,20 @@ interface XBrowserOptions {
export async function postToX(options: XBrowserOptions): Promise<void> { export async function postToX(options: XBrowserOptions): Promise<void> {
const { text, images = [], submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options; const { text, images = [], submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_FULL);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const port = await getFreePort(); const existingPort = await findExistingChromeDebugPort(profileDir);
console.log(`[x-browser] Launching Chrome (profile: ${profileDir})`); const reusing = existingPort !== null;
let port = existingPort ?? 0;
let chrome: Awaited<ReturnType<typeof launchChrome>>['chrome'] | null = null;
if (!reusing) {
const launched = await launchChrome(X_COMPOSE_URL, profileDir, CHROME_CANDIDATES_FULL, options.chromePath);
port = launched.port;
chrome = launched.chrome;
}
const chrome = spawn(chromePath, [ if (reusing) console.log(`[x-browser] Reusing existing Chrome on port ${port}`);
`--remote-debugging-port=${port}`, else console.log(`[x-browser] Launching Chrome (profile: ${profileDir})`);
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
X_COMPOSE_URL,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
@@ -52,18 +49,15 @@ export async function postToX(options: XBrowserOptions): Promise<void> {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 });
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const page = await openPageSession({
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('x.com')); cdp,
reusing,
if (!pageTarget) { url: X_COMPOSE_URL,
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: X_COMPOSE_URL }); matchTarget: (target) => target.type === 'page' && target.url.includes('x.com'),
pageTarget = { targetId, url: X_COMPOSE_URL, type: 'page' }; enablePage: true,
} enableRuntime: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = page;
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId }); await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId });
console.log('[x-browser] Waiting for X editor...'); console.log('[x-browser] Waiting for X editor...');
@@ -193,7 +187,9 @@ export async function postToX(options: XBrowserOptions): Promise<void> {
if (cdp) { if (cdp) {
cdp.close(); cdp.close();
} }
chrome.unref(); if (chrome) {
chrome.unref();
}
} }
} }
+32 -35
View File
@@ -1,12 +1,13 @@
import { spawn } from 'node:child_process';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import process from 'node:process'; import process from 'node:process';
import { import {
CHROME_CANDIDATES_FULL, CHROME_CANDIDATES_FULL,
CdpConnection, CdpConnection,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, killChrome,
launchChrome,
openPageSession,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
} from './x-utils.js'; } from './x-utils.js';
@@ -31,43 +32,39 @@ interface QuoteOptions {
export async function quotePost(options: QuoteOptions): Promise<void> { export async function quotePost(options: QuoteOptions): Promise<void> {
const { tweetUrl, comment, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options; const { tweetUrl, comment, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_FULL);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const port = await getFreePort(); const existingPort = await findExistingChromeDebugPort(profileDir);
console.log(`[x-quote] Launching Chrome (profile: ${profileDir})`); const reusing = existingPort !== null;
let port = existingPort ?? 0;
console.log(`[x-quote] Opening tweet: ${tweetUrl}`); console.log(`[x-quote] Opening tweet: ${tweetUrl}`);
let chrome: Awaited<ReturnType<typeof launchChrome>>['chrome'] | null = null;
if (!reusing) {
const launched = await launchChrome(tweetUrl, profileDir, CHROME_CANDIDATES_FULL, options.chromePath);
port = launched.port;
chrome = launched.chrome;
}
const chrome = spawn(chromePath, [ if (reusing) console.log(`[x-quote] Reusing existing Chrome on port ${port}`);
`--remote-debugging-port=${port}`, else console.log(`[x-quote] Launching Chrome (profile: ${profileDir})`);
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
tweetUrl,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 15_000 });
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const page = await openPageSession({
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('x.com')); cdp,
reusing,
if (!pageTarget) { url: tweetUrl,
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: tweetUrl }); matchTarget: (target) => target.type === 'page' && target.url.includes('x.com'),
pageTarget = { targetId, url: tweetUrl, type: 'page' }; enablePage: true,
} enableRuntime: true,
});
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); const { sessionId } = page;
targetId = page.targetId;
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
console.log('[x-quote] Waiting for tweet to load...'); console.log('[x-quote] Waiting for tweet to load...');
await sleep(3000); await sleep(3000);
@@ -176,14 +173,14 @@ export async function quotePost(options: QuoteOptions): Promise<void> {
} }
} finally { } finally {
if (cdp) { if (cdp) {
try { await cdp.send('Browser.close', {}, { timeoutMs: 5_000 }); } catch {} if (reusing && targetId) {
try { await cdp.send('Target.closeTarget', { targetId }, { timeoutMs: 5_000 }); } catch {}
} else if (!reusing) {
try { await cdp.send('Browser.close', {}, { timeoutMs: 5_000 }); } catch {}
}
cdp.close(); cdp.close();
} }
if (chrome) killChrome(chrome);
setTimeout(() => {
if (!chrome.killed) try { chrome.kill('SIGKILL'); } catch {}
}, 2_000).unref?.();
try { chrome.kill('SIGTERM'); } catch {}
} }
} }
+44 -164
View File
@@ -1,16 +1,24 @@
import { execSync, spawnSync } from 'node:child_process'; import { execSync, spawnSync } from 'node:child_process';
import fs from 'node:fs';
import net from 'node:net';
import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import process from 'node:process'; import process from 'node:process';
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from 'node:url';
export type PlatformCandidates = { import {
darwin?: string[]; CdpConnection,
win32?: string[]; findChromeExecutable as findChromeExecutableBase,
default: string[]; findExistingChromeDebugPort as findExistingChromeDebugPortBase,
}; getFreePort as getFreePortBase,
killChrome,
launchChrome as launchChromeBase,
openPageSession,
resolveSharedChromeProfileDir,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
export { CdpConnection, killChrome, openPageSession, sleep, waitForChromeDebugPort };
export type { PlatformCandidates } from 'baoyu-chrome-cdp';
export const CHROME_CANDIDATES_BASIC: PlatformCandidates = { export const CHROME_CANDIDATES_BASIC: PlatformCandidates = {
darwin: [ darwin: [
@@ -53,20 +61,11 @@ export const CHROME_CANDIDATES_FULL: PlatformCandidates = {
], ],
}; };
function getCandidatesForPlatform(candidates: PlatformCandidates): string[] {
if (process.platform === 'darwin' && candidates.darwin?.length) return candidates.darwin;
if (process.platform === 'win32' && candidates.win32?.length) return candidates.win32;
return candidates.default;
}
export function findChromeExecutable(candidates: PlatformCandidates): string | undefined { export function findChromeExecutable(candidates: PlatformCandidates): string | undefined {
const override = process.env.X_BROWSER_CHROME_PATH?.trim(); return findChromeExecutableBase({
if (override && fs.existsSync(override)) return override; candidates,
envNames: ['X_BROWSER_CHROME_PATH'],
for (const candidate of getCandidatesForPlatform(candidates)) { });
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
} }
let _wslHome: string | null | undefined; let _wslHome: string | null | undefined;
@@ -81,158 +80,39 @@ function getWslWindowsHome(): string | null {
} }
export function getDefaultProfileDir(): string { export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.X_BROWSER_PROFILE_DIR?.trim(); return resolveSharedChromeProfileDir({
if (override) return path.resolve(override); envNames: ['BAOYU_CHROME_PROFILE_DIR', 'X_BROWSER_PROFILE_DIR'],
const wslHome = getWslWindowsHome(); wslWindowsHome: getWslWindowsHome(),
if (wslHome) return path.join(wslHome, '.local', 'share', 'baoyu-skills', 'chrome-profile');
const base = process.platform === 'darwin'
? path.join(os.homedir(), 'Library', 'Application Support')
: process.env.XDG_DATA_HOME || path.join(os.homedir(), '.local', 'share');
return path.join(base, 'baoyu-skills', 'chrome-profile');
}
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(): Promise<number> {
const fixed = parseInt(process.env.X_BROWSER_DEBUG_PORT || '', 10);
if (fixed > 0) return fixed;
return new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (!address || typeof address === 'string') {
server.close(() => reject(new Error('Unable to allocate a free TCP port.')));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
}); });
} }
async function fetchJson<T = unknown>(url: string): Promise<T> { export async function getFreePort(): Promise<number> {
const res = await fetch(url, { redirect: 'follow' }); return await getFreePortBase('X_BROWSER_DEBUG_PORT');
if (!res.ok) throw new Error(`Request failed: ${res.status} ${res.statusText}`);
return (await res.json()) as T;
} }
export async function waitForChromeDebugPort( export async function findExistingChromeDebugPort(profileDir: string): Promise<number | null> {
port: number, return await findExistingChromeDebugPortBase({ profileDir });
timeoutMs: number,
options?: { includeLastError?: boolean },
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(`http://127.0.0.1:${port}/json/version`);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error('Missing webSocketDebuggerUrl');
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`);
}
throw new Error('Chrome debug port not ready');
} }
type PendingRequest = { export async function launchChrome(
resolve: (value: unknown) => void; url: string,
reject: (error: Error) => void; profileDir: string,
timer: ReturnType<typeof setTimeout> | null; candidates: PlatformCandidates,
}; chromePathOverride?: string,
): Promise<{ chrome: Awaited<ReturnType<typeof launchChromeBase>>; port: number }> {
const chromePath = chromePathOverride?.trim() || findChromeExecutable(candidates);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
export class CdpConnection { const port = await getFreePort();
private ws: WebSocket; const chrome = await launchChromeBase({
private nextId = 0; chromePath,
private pending = new Map<number, PendingRequest>(); profileDir,
private eventHandlers = new Map<string, Set<(params: unknown) => void>>(); port,
private defaultTimeoutMs: number; url,
extraArgs: ['--start-maximized'],
});
private constructor(ws: WebSocket, options?: { defaultTimeoutMs?: number }) { return { chrome, port };
this.ws = ws;
this.defaultTimeoutMs = options?.defaultTimeoutMs ?? 15_000;
this.ws.addEventListener('message', (event) => {
try {
const data = typeof event.data === 'string' ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) handlers.forEach((h) => h(msg.params));
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener('close', () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error('CDP connection closed.'));
}
});
}
static async connect(url: string, timeoutMs: number, options?: { defaultTimeoutMs?: number }): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('CDP connection timeout.')), timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); reject(new Error('CDP connection failed.')); });
});
return new CdpConnection(ws, options);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) this.eventHandlers.set(method, new Set());
this.eventHandlers.get(method)!.add(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: { sessionId?: string; timeoutMs?: number }): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
} }
export function getScriptDir(): string { export function getScriptDir(): string {
+31 -37
View File
@@ -1,4 +1,3 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import { mkdir } from 'node:fs/promises'; import { mkdir } from 'node:fs/promises';
import path from 'node:path'; import path from 'node:path';
@@ -6,9 +5,11 @@ import process from 'node:process';
import { import {
CHROME_CANDIDATES_FULL, CHROME_CANDIDATES_FULL,
CdpConnection, CdpConnection,
findChromeExecutable, findExistingChromeDebugPort,
getDefaultProfileDir, getDefaultProfileDir,
getFreePort, killChrome,
launchChrome,
openPageSession,
sleep, sleep,
waitForChromeDebugPort, waitForChromeDebugPort,
} from './x-utils.js'; } from './x-utils.js';
@@ -27,9 +28,6 @@ interface XVideoOptions {
export async function postVideoToX(options: XVideoOptions): Promise<void> { export async function postVideoToX(options: XVideoOptions): Promise<void> {
const { text, videoPath, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options; const { text, videoPath, submit = false, timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const chromePath = options.chromePath ?? findChromeExecutable(CHROME_CANDIDATES_FULL);
if (!chromePath) throw new Error('Chrome not found. Set X_BROWSER_CHROME_PATH env var.');
if (!fs.existsSync(videoPath)) throw new Error(`Video not found: ${videoPath}`); if (!fs.existsSync(videoPath)) throw new Error(`Video not found: ${videoPath}`);
const absVideoPath = path.resolve(videoPath); const absVideoPath = path.resolve(videoPath);
@@ -37,38 +35,37 @@ export async function postVideoToX(options: XVideoOptions): Promise<void> {
await mkdir(profileDir, { recursive: true }); await mkdir(profileDir, { recursive: true });
const port = await getFreePort(); const existingPort = await findExistingChromeDebugPort(profileDir);
console.log(`[x-video] Launching Chrome (profile: ${profileDir})`); const reusing = existingPort !== null;
let port = existingPort ?? 0;
let chrome: Awaited<ReturnType<typeof launchChrome>>['chrome'] | null = null;
if (!reusing) {
const launched = await launchChrome(X_COMPOSE_URL, profileDir, CHROME_CANDIDATES_FULL, options.chromePath);
port = launched.port;
chrome = launched.chrome;
}
const chrome = spawn(chromePath, [ if (reusing) console.log(`[x-video] Reusing existing Chrome on port ${port}`);
`--remote-debugging-port=${port}`, else console.log(`[x-video] Launching Chrome (profile: ${profileDir})`);
`--user-data-dir=${profileDir}`,
'--no-first-run',
'--no-default-browser-check',
'--disable-blink-features=AutomationControlled',
'--start-maximized',
X_COMPOSE_URL,
], { stdio: 'ignore' });
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true }); const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 30_000 }); cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 30_000 });
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets'); const page = await openPageSession({
let pageTarget = targets.targetInfos.find((t) => t.type === 'page' && t.url.includes('x.com')); cdp,
reusing,
if (!pageTarget) { url: X_COMPOSE_URL,
const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url: X_COMPOSE_URL }); matchTarget: (target) => target.type === 'page' && target.url.includes('x.com'),
pageTarget = { targetId, url: X_COMPOSE_URL, type: 'page' }; enablePage: true,
} enableRuntime: true,
enableDom: true,
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true }); });
const { sessionId } = page;
await cdp.send('Page.enable', {}, { sessionId }); targetId = page.targetId;
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('DOM.enable', {}, { sessionId });
await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId }); await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId });
console.log('[x-video] Waiting for X editor...'); console.log('[x-video] Waiting for X editor...');
@@ -183,15 +180,12 @@ export async function postVideoToX(options: XVideoOptions): Promise<void> {
} }
} finally { } finally {
if (cdp) { if (cdp) {
if (reusing && submit && targetId) {
try { await cdp.send('Target.closeTarget', { targetId }, { timeoutMs: 5_000 }); } catch {}
}
cdp.close(); cdp.close();
} }
// Don't kill Chrome in preview mode, let user review if (chrome && submit) killChrome(chrome);
if (submit) {
setTimeout(() => {
if (!chrome.killed) try { chrome.kill('SIGKILL'); } catch {}
}, 2_000).unref?.();
try { chrome.kill('SIGTERM'); } catch {}
}
} }
} }
+20 -3
View File
@@ -188,7 +188,7 @@ Before translating chunks:
- Splits at markdown block boundaries to preserve structure - Splits at markdown block boundaries to preserve structure
- If a single block exceeds the threshold, falls back to line splitting, then word splitting - If a single block exceeds the threshold, falls back to line splitting, then word splitting
4. **Assemble translation prompt**: 4. **Assemble translation prompt**:
- Main agent reads `01-analysis.md` (if exists) and assembles shared context using Part 1 of [references/subagent-prompt-template.md](references/subagent-prompt-template.md) — inlining content background, merged glossary, and comprehension challenges - Main agent reads `01-analysis.md` (if exists) and assembles shared context using Part 1 of [references/subagent-prompt-template.md](references/subagent-prompt-template.md) — inlining the resolved style preset (from `--style` flag, EXTEND.md `style` setting, or default `storytelling`), content background, merged glossary, and comprehension challenges
- Save as `02-prompt.md` in the output directory (shared context only, no task instructions) - Save as `02-prompt.md` in the output directory (shared context only, no task instructions)
5. **Draft translation via subagents** (if Agent tool available): 5. **Draft translation via subagents** (if Agent tool available):
- Spawn one subagent **per chunk**, all in parallel (Part 2 of the template) - Spawn one subagent **per chunk**, all in parallel (Part 2 of the template)
@@ -212,9 +212,10 @@ Before translating chunks:
- **Natural flow**: Use idiomatic target language word order and sentence patterns; break or restructure sentences freely when the source structure doesn't work naturally in the target language - **Natural flow**: Use idiomatic target language word order and sentence patterns; break or restructure sentences freely when the source structure doesn't work naturally in the target language
- **Terminology**: Use standard translations; annotate with original term in parentheses on first occurrence - **Terminology**: Use standard translations; annotate with original term in parentheses on first occurrence
- **Preserve format**: Keep all markdown formatting (headings, bold, italic, images, links, code blocks) - **Preserve format**: Keep all markdown formatting (headings, bold, italic, images, links, code blocks)
- **Image-language awareness**: Preserve image references exactly during translation, but after the translation is complete, review referenced images and check whether their likely main text language still matches the translated article language
- **Frontmatter transformation**: If the source has YAML frontmatter, preserve it in the translation with these changes: (1) Rename metadata fields that describe the *source* article — `url``sourceUrl`, `title``sourceTitle`, `description``sourceDescription`, `author``sourceAuthor`, `date``sourceDate`, and any similar origin-metadata fields — by adding a `source` prefix (camelCase). (2) Translate the values of text fields (title, description, etc.) and add them as new top-level fields. (3) Keep other fields (tags, categories, custom fields) as-is, translating their values where appropriate - **Frontmatter transformation**: If the source has YAML frontmatter, preserve it in the translation with these changes: (1) Rename metadata fields that describe the *source* article — `url``sourceUrl`, `title``sourceTitle`, `description``sourceDescription`, `author``sourceAuthor`, `date``sourceDate`, and any similar origin-metadata fields — by adding a `source` prefix (camelCase). (2) Translate the values of text fields (title, description, etc.) and add them as new top-level fields. (3) Keep other fields (tags, categories, custom fields) as-is, translating their values where appropriate
- **Respect original**: Maintain original meaning and intent; do not add, remove, or editorialize — but sentence structure and imagery may be adapted freely to serve the meaning - **Respect original**: Maintain original meaning and intent; do not add, remove, or editorialize — but sentence structure and imagery may be adapted freely to serve the meaning
- **Translator's notes**: For terms, concepts, or cultural references that target readers may not understand — due to jargon, cultural gaps, or domain-specific knowledge — add a concise explanatory note in parentheses immediately after the term. The note should explain *what it means* in plain language, not just provide the English original. Format: `译文(English original,通俗解释)`. Calibrate annotation depth to the target audience: general readers need more notes than technical readers. Only add notes where genuinely needed; do not over-annotate obvious terms. - **Translator's notes**: For terms, concepts, or cultural references that target readers may not understand — due to jargon, cultural gaps, or domain-specific knowledge — add a concise explanatory note in parentheses immediately after the term. The note should explain *what it means* in plain language, not just provide the English original. Format: `译文(English original,通俗解释)`. Calibrate annotation depth to the target audience: general readers need more notes than technical readers. For short texts (< 5 sentences), further reduce annotations — only annotate non-common terms that the target audience is unlikely to know; skip terms that are widely recognized or self-explanatory in context. Only add notes where genuinely needed; do not over-annotate obvious terms.
#### Quick Mode #### Quick Mode
@@ -223,7 +224,7 @@ Translate directly → save to `translation.md`. No analysis file, but still app
#### Normal Mode #### Normal Mode
1. **Analyze**`01-analysis.md` (domain, tone, audience, terminology, reader comprehension challenges, figurative language & metaphor mapping) 1. **Analyze**`01-analysis.md` (domain, tone, audience, terminology, reader comprehension challenges, figurative language & metaphor mapping)
2. **Assemble prompt**`02-prompt.md` (translation instructions with inlined context) 2. **Assemble prompt**`02-prompt.md` (translation instructions with inlined style preset, content background, glossary, and comprehension challenges)
3. **Translate** (following `02-prompt.md`) → `translation.md` 3. **Translate** (following `02-prompt.md`) → `translation.md`
After completion, prompt user: "Translation saved. To further review and polish, reply **继续润色** or **refine**." After completion, prompt user: "Translation saved. To further review and polish, reply **继续润色** or **refine**."
@@ -250,6 +251,20 @@ Each step reads the previous step's file and builds on it.
Final translation is always at `translation.md` in the output directory. Final translation is always at `translation.md` in the output directory.
After the final translation is written, do a lightweight image-language pass:
1. Collect image references from the translated article
2. Identify likely text-heavy images such as covers, screenshots, diagrams, charts, frameworks, and infographics
3. If any image likely contains a main text language that does not match the translated article language, proactively remind the user
4. The reminder must be a list only. Do not automatically localize those images unless the user asks
Reminder format (use whatever image syntax the article already uses — standard markdown or wikilink):
```text
Possible image localization needed:
- ![example cover](attachments/example-cover.png): likely still contains source-language text while the article is now in target language
- ![example diagram](attachments/example-diagram.png): likely text-heavy framework graphic, check whether labels need translation
```
Display summary: Display summary:
``` ```
**Translation complete** ({mode} mode) **Translation complete** ({mode} mode)
@@ -261,6 +276,8 @@ Final: {output-dir}/translation.md
Glossary terms applied: {count} Glossary terms applied: {count}
``` ```
If mismatched image-language candidates were found, append a short note after the summary telling the user that some embedded images may still need image-text localization, followed by the candidate list.
## Extension Support ## Extension Support
Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options. Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.
@@ -121,7 +121,7 @@ Implicit assumptions: [unstated premises]
## Step 2: Assemble Translation Prompt ## Step 2: Assemble Translation Prompt
Main agent reads `01-analysis.md` and assembles a complete translation prompt using [references/subagent-prompt-template.md](subagent-prompt-template.md). Inline content background, merged glossary, and comprehension challenges into the prompt. Save to `02-prompt.md`. Main agent reads `01-analysis.md` and assembles a complete translation prompt using [references/subagent-prompt-template.md](subagent-prompt-template.md). Inline the resolved style preset (from `--style` flag, EXTEND.md `style` setting, or default `storytelling`), content background, merged glossary, and comprehension challenges into the prompt. Save to `02-prompt.md`.
This prompt is used by the subagent (chunked) or by the main agent itself (non-chunked). This prompt is used by the subagent (chunked) or by the main agent itself (non-chunked).
@@ -5,6 +5,7 @@
"name": "baoyu-url-to-markdown-scripts", "name": "baoyu-url-to-markdown-scripts",
"dependencies": { "dependencies": {
"@mozilla/readability": "^0.6.0", "@mozilla/readability": "^0.6.0",
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"defuddle": "^0.10.0", "defuddle": "^0.10.0",
"jsdom": "^24.1.3", "jsdom": "^24.1.3",
"linkedom": "^0.18.12", "linkedom": "^0.18.12",
@@ -36,6 +37,8 @@
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="], "asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@file:vendor/baoyu-chrome-cdp", {}],
"boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="], "boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="],
"call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="], "call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="],
+116 -227
View File
@@ -1,209 +1,84 @@
import { spawn, type ChildProcess } from "node:child_process"; import {
import fs from "node:fs"; CdpConnection,
import { mkdir } from "node:fs/promises"; findChromeExecutable as findChromeExecutableBase,
import net from "node:net"; findExistingChromeDebugPort,
import process from "node:process"; getFreePort,
killChrome,
launchChrome as launchChromeBase,
sleep,
waitForChromeDebugPort,
type PlatformCandidates,
} from 'baoyu-chrome-cdp';
import { resolveUrlToMarkdownChromeProfileDir } from "./paths.js"; import { resolveUrlToMarkdownChromeProfileDir } from './paths.js';
import { CDP_CONNECT_TIMEOUT_MS, NETWORK_IDLE_TIMEOUT_MS } from "./constants.js"; import { NETWORK_IDLE_TIMEOUT_MS } from './constants.js';
type CdpSendOptions = { sessionId?: string; timeoutMs?: number }; const CHROME_CANDIDATES_FULL: PlatformCandidates = {
darwin: [
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
'/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta',
'/Applications/Chromium.app/Contents/MacOS/Chromium',
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
],
win32: [
'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
'C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe',
'C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe',
],
default: [
'/usr/bin/google-chrome',
'/usr/bin/google-chrome-stable',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
'/snap/bin/chromium',
'/usr/bin/microsoft-edge',
],
};
function sleep(ms: number): Promise<void> { export { CdpConnection, getFreePort, killChrome, sleep, waitForChromeDebugPort };
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function fetchWithTimeout(url: string, init: RequestInit & { timeoutMs?: number } = {}): Promise<Response> { export async function findExistingChromePort(): Promise<number | null> {
const { timeoutMs, ...rest } = init; return await findExistingChromeDebugPort({
if (!timeoutMs || timeoutMs <= 0) return fetch(url, rest); profileDir: resolveUrlToMarkdownChromeProfileDir(),
const ctl = new AbortController();
const t = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { ...rest, signal: ctl.signal });
} finally {
clearTimeout(t);
}
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, { resolve: (v: unknown) => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> | null }>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private constructor(ws: WebSocket) {
this.ws = ws;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string" ? event.data : new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as { id?: number; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
if (msg.id) {
const p = this.pending.get(msg.id);
if (p) {
this.pending.delete(msg.id);
if (p.timer) clearTimeout(p.timer);
if (msg.error?.message) p.reject(new Error(msg.error.message));
else p.resolve(msg.result);
}
} else if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
for (const h of handlers) h(msg.params);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, p] of this.pending.entries()) {
this.pending.delete(id);
if (p.timer) clearTimeout(p.timer);
p.reject(new Error("CDP connection closed."));
}
});
}
static async connect(url: string, timeoutMs: number): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const t = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => { clearTimeout(t); resolve(); });
ws.addEventListener("error", () => { clearTimeout(t); reject(new Error("CDP connection failed.")); });
});
return new CdpConnection(ws);
}
on(event: string, handler: (params: unknown) => void): void {
let handlers = this.eventHandlers.get(event);
if (!handlers) {
handlers = new Set();
this.eventHandlers.set(event, handlers);
}
handlers.add(handler);
}
off(event: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(event)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, opts?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const msg: Record<string, unknown> = { id, method };
if (params) msg.params = params;
if (opts?.sessionId) msg.sessionId = opts.sessionId;
const timeoutMs = opts?.timeoutMs ?? 15_000;
const out = await new Promise<unknown>((resolve, reject) => {
const t = timeoutMs > 0 ? setTimeout(() => { this.pending.delete(id); reject(new Error(`CDP timeout: ${method}`)); }, timeoutMs) : null;
this.pending.set(id, { resolve, reject, timer: t });
this.ws.send(JSON.stringify(msg));
});
return out as T;
}
close(): void {
try { this.ws.close(); } catch {}
}
}
export async function getFreePort(): Promise<number> {
return await new Promise((resolve, reject) => {
const srv = net.createServer();
srv.unref();
srv.on("error", reject);
srv.listen(0, "127.0.0.1", () => {
const addr = srv.address();
if (!addr || typeof addr === "string") {
srv.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = addr.port;
srv.close((err) => (err ? reject(err) : resolve(port)));
});
}); });
} }
export function findChromeExecutable(): string | null { export function findChromeExecutable(): string | null {
const override = process.env.URL_CHROME_PATH?.trim(); return findChromeExecutableBase({
if (override && fs.existsSync(override)) return override; candidates: CHROME_CANDIDATES_FULL,
envNames: ['URL_CHROME_PATH'],
const candidates: string[] = []; }) ?? null;
switch (process.platform) {
case "darwin":
candidates.push(
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
"/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge"
);
break;
case "win32":
candidates.push(
"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
"C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe",
"C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe"
);
break;
default:
candidates.push(
"/usr/bin/google-chrome",
"/usr/bin/google-chrome-stable",
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
"/snap/bin/chromium",
"/usr/bin/microsoft-edge"
);
break;
}
for (const p of candidates) {
if (fs.existsSync(p)) return p;
}
return null;
} }
export async function waitForChromeDebugPort(port: number, timeoutMs: number): Promise<string> { export async function launchChrome(url: string, port: number, headless = false) {
const start = Date.now(); const chromePath = findChromeExecutable();
while (Date.now() - start < timeoutMs) { if (!chromePath) throw new Error('Chrome executable not found. Install Chrome or set URL_CHROME_PATH env.');
try {
const res = await fetchWithTimeout(`http://127.0.0.1:${port}/json/version`, { timeoutMs: 5_000 }); return await launchChromeBase({
if (!res.ok) throw new Error(`status=${res.status}`); chromePath,
const j = (await res.json()) as { webSocketDebuggerUrl?: string }; profileDir: resolveUrlToMarkdownChromeProfileDir(),
if (j.webSocketDebuggerUrl) return j.webSocketDebuggerUrl; port,
} catch {} url,
await sleep(200); headless,
} extraArgs: ['--disable-popup-blocking'],
throw new Error("Chrome debug port not ready"); });
} }
export async function launchChrome(url: string, port: number, headless: boolean = false): Promise<ChildProcess> { export async function waitForNetworkIdle(
const chrome = findChromeExecutable(); cdp: CdpConnection,
if (!chrome) throw new Error("Chrome executable not found. Install Chrome or set URL_CHROME_PATH env."); sessionId: string,
const profileDir = resolveUrlToMarkdownChromeProfileDir(); timeoutMs: number = NETWORK_IDLE_TIMEOUT_MS,
await mkdir(profileDir, { recursive: true }); ): Promise<void> {
const args = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
"--no-first-run",
"--no-default-browser-check",
"--disable-popup-blocking",
];
if (headless) args.push("--headless=new");
args.push(url);
return spawn(chrome, args, { stdio: "ignore" });
}
export async function waitForNetworkIdle(cdp: CdpConnection, sessionId: string, timeoutMs: number = NETWORK_IDLE_TIMEOUT_MS): Promise<void> {
return new Promise((resolve) => { return new Promise((resolve) => {
let timer: ReturnType<typeof setTimeout> | null = null; let timer: ReturnType<typeof setTimeout> | null = null;
let pending = 0; let pending = 0;
const cleanup = () => { const cleanup = () => {
if (timer) clearTimeout(timer); if (timer) clearTimeout(timer);
cdp.off("Network.requestWillBeSent", onRequest); cdp.off('Network.requestWillBeSent', onRequest);
cdp.off("Network.loadingFinished", onFinish); cdp.off('Network.loadingFinished', onFinish);
cdp.off("Network.loadingFailed", onFinish); cdp.off('Network.loadingFailed', onFinish);
}; };
const done = () => { cleanup(); resolve(); }; const done = () => { cleanup(); resolve(); };
const resetTimer = () => { const resetTimer = () => {
@@ -212,79 +87,93 @@ export async function waitForNetworkIdle(cdp: CdpConnection, sessionId: string,
}; };
const onRequest = () => { pending++; resetTimer(); }; const onRequest = () => { pending++; resetTimer(); };
const onFinish = () => { pending = Math.max(0, pending - 1); if (pending <= 2) resetTimer(); }; const onFinish = () => { pending = Math.max(0, pending - 1); if (pending <= 2) resetTimer(); };
cdp.on("Network.requestWillBeSent", onRequest); cdp.on('Network.requestWillBeSent', onRequest);
cdp.on("Network.loadingFinished", onFinish); cdp.on('Network.loadingFinished', onFinish);
cdp.on("Network.loadingFailed", onFinish); cdp.on('Network.loadingFailed', onFinish);
resetTimer(); resetTimer();
}); });
} }
export async function waitForPageLoad(cdp: CdpConnection, sessionId: string, timeoutMs: number = 30_000): Promise<void> { export async function waitForPageLoad(
return new Promise((resolve, reject) => { cdp: CdpConnection,
sessionId: string,
timeoutMs: number = 30_000,
): Promise<void> {
void sessionId;
return new Promise((resolve) => {
const timer = setTimeout(() => { const timer = setTimeout(() => {
cdp.off("Page.loadEventFired", handler); cdp.off('Page.loadEventFired', handler);
resolve(); resolve();
}, timeoutMs); }, timeoutMs);
const handler = () => { const handler = () => {
clearTimeout(timer); clearTimeout(timer);
cdp.off("Page.loadEventFired", handler); cdp.off('Page.loadEventFired', handler);
resolve(); resolve();
}; };
cdp.on("Page.loadEventFired", handler); cdp.on('Page.loadEventFired', handler);
}); });
} }
export async function createTargetAndAttach(cdp: CdpConnection, url: string): Promise<{ targetId: string; sessionId: string }> { export async function createTargetAndAttach(
const { targetId } = await cdp.send<{ targetId: string }>("Target.createTarget", { url }); cdp: CdpConnection,
const { sessionId } = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true }); url: string,
await cdp.send("Network.enable", {}, { sessionId }); ): Promise<{ targetId: string; sessionId: string }> {
await cdp.send("Page.enable", {}, { sessionId }); const { targetId } = await cdp.send<{ targetId: string }>('Target.createTarget', { url });
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId, flatten: true });
await cdp.send('Network.enable', {}, { sessionId });
await cdp.send('Page.enable', {}, { sessionId });
return { targetId, sessionId }; return { targetId, sessionId };
} }
export async function navigateAndWait(cdp: CdpConnection, sessionId: string, url: string, timeoutMs: number): Promise<void> { export async function navigateAndWait(
cdp: CdpConnection,
sessionId: string,
url: string,
timeoutMs: number,
): Promise<void> {
const loadPromise = new Promise<void>((resolve, reject) => { const loadPromise = new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("Page load timeout")), timeoutMs); const timer = setTimeout(() => reject(new Error('Page load timeout')), timeoutMs);
const handler = (params: unknown) => { const handler = (params: unknown) => {
const p = params as { name?: string }; const event = params as { name?: string };
if (p.name === "load" || p.name === "DOMContentLoaded") { if (event.name === 'load' || event.name === 'DOMContentLoaded') {
clearTimeout(timer); clearTimeout(timer);
cdp.off("Page.lifecycleEvent", handler); cdp.off('Page.lifecycleEvent', handler);
resolve(); resolve();
} }
}; };
cdp.on("Page.lifecycleEvent", handler); cdp.on('Page.lifecycleEvent', handler);
}); });
await cdp.send("Page.navigate", { url }, { sessionId }); await cdp.send('Page.navigate', { url }, { sessionId });
await loadPromise; await loadPromise;
} }
export async function evaluateScript<T>(cdp: CdpConnection, sessionId: string, expression: string, timeoutMs: number = 30_000): Promise<T> { export async function evaluateScript<T>(
const result = await cdp.send<{ result: { value?: T; type?: string; description?: string } }>( cdp: CdpConnection,
"Runtime.evaluate", sessionId: string,
expression: string,
timeoutMs: number = 30_000,
): Promise<T> {
const result = await cdp.send<{ result: { value?: T } }>(
'Runtime.evaluate',
{ expression, returnByValue: true, awaitPromise: true }, { expression, returnByValue: true, awaitPromise: true },
{ sessionId, timeoutMs } { sessionId, timeoutMs },
); );
return result.result.value as T; return result.result.value as T;
} }
export async function autoScroll(cdp: CdpConnection, sessionId: string, steps: number = 8, waitMs: number = 600): Promise<void> { export async function autoScroll(
let lastHeight = await evaluateScript<number>(cdp, sessionId, "document.body.scrollHeight"); cdp: CdpConnection,
sessionId: string,
steps: number = 8,
waitMs: number = 600,
): Promise<void> {
let lastHeight = await evaluateScript<number>(cdp, sessionId, 'document.body.scrollHeight');
for (let i = 0; i < steps; i++) { for (let i = 0; i < steps; i++) {
await evaluateScript<void>(cdp, sessionId, "window.scrollTo(0, document.body.scrollHeight)"); await evaluateScript<void>(cdp, sessionId, 'window.scrollTo(0, document.body.scrollHeight)');
await sleep(waitMs); await sleep(waitMs);
const newHeight = await evaluateScript<number>(cdp, sessionId, "document.body.scrollHeight"); const newHeight = await evaluateScript<number>(cdp, sessionId, 'document.body.scrollHeight');
if (newHeight === lastHeight) break; if (newHeight === lastHeight) break;
lastHeight = newHeight; lastHeight = newHeight;
} }
await evaluateScript<void>(cdp, sessionId, "window.scrollTo(0, 0)"); await evaluateScript<void>(cdp, sessionId, 'window.scrollTo(0, 0)');
}
export function killChrome(chrome: ChildProcess): void {
try { chrome.kill("SIGTERM"); } catch {}
setTimeout(() => {
if (!chrome.killed) {
try { chrome.kill("SIGKILL"); } catch {}
}
}, 2_000).unref?.();
} }
+37 -14
View File
@@ -3,7 +3,7 @@ import { writeFile, mkdir, access } from "node:fs/promises";
import path from "node:path"; import path from "node:path";
import process from "node:process"; import process from "node:process";
import { CdpConnection, getFreePort, launchChrome, waitForChromeDebugPort, waitForNetworkIdle, waitForPageLoad, autoScroll, evaluateScript, killChrome } from "./cdp.js"; import { CdpConnection, getFreePort, findExistingChromePort, launchChrome, waitForChromeDebugPort, waitForNetworkIdle, waitForPageLoad, autoScroll, evaluateScript, killChrome } from "./cdp.js";
import { absolutizeUrlsScript, extractContent, createMarkdownDocument, type ConversionResult } from "./html-to-markdown.js"; import { absolutizeUrlsScript, extractContent, createMarkdownDocument, type ConversionResult } from "./html-to-markdown.js";
import { localizeMarkdownMedia, countRemoteMedia } from "./media-localizer.js"; import { localizeMarkdownMedia, countRemoteMedia } from "./media-localizer.js";
import { resolveUrlToMarkdownDataDir } from "./paths.js"; import { resolveUrlToMarkdownDataDir } from "./paths.js";
@@ -98,21 +98,37 @@ async function waitForUserSignal(): Promise<void> {
} }
async function captureUrl(args: Args): Promise<ConversionResult> { async function captureUrl(args: Args): Promise<ConversionResult> {
const port = await getFreePort(); const existingPort = await findExistingChromePort();
const chrome = await launchChrome(args.url, port, false); const reusing = existingPort !== null;
const port = existingPort ?? await getFreePort();
const chrome = reusing ? null : await launchChrome(args.url, port, false);
if (reusing) console.log(`Reusing existing Chrome on port ${port}`);
let cdp: CdpConnection | null = null; let cdp: CdpConnection | null = null;
let targetId: string | null = null;
try { try {
const wsUrl = await waitForChromeDebugPort(port, 30_000); const wsUrl = await waitForChromeDebugPort(port, 30_000);
cdp = await CdpConnection.connect(wsUrl, CDP_CONNECT_TIMEOUT_MS); cdp = await CdpConnection.connect(wsUrl, CDP_CONNECT_TIMEOUT_MS);
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; type: string; url: string }> }>("Target.getTargets"); let sessionId: string;
const pageTarget = targets.targetInfos.find(t => t.type === "page" && t.url.startsWith("http")); if (reusing) {
if (!pageTarget) throw new Error("No page target found"); const created = await cdp.send<{ targetId: string }>("Target.createTarget", { url: args.url });
targetId = created.targetId;
const { sessionId } = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId: pageTarget.targetId, flatten: true }); const attached = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true });
await cdp.send("Network.enable", {}, { sessionId }); sessionId = attached.sessionId;
await cdp.send("Page.enable", {}, { sessionId }); await cdp.send("Network.enable", {}, { sessionId });
await cdp.send("Page.enable", {}, { sessionId });
} else {
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; type: string; url: string }> }>("Target.getTargets");
const pageTarget = targets.targetInfos.find(t => t.type === "page" && t.url.startsWith("http"));
if (!pageTarget) throw new Error("No page target found");
targetId = pageTarget.targetId;
const attached = await cdp.send<{ sessionId: string }>("Target.attachToTarget", { targetId, flatten: true });
sessionId = attached.sessionId;
await cdp.send("Network.enable", {}, { sessionId });
await cdp.send("Page.enable", {}, { sessionId });
}
if (args.wait) { if (args.wait) {
await waitForUserSignal(); await waitForUserSignal();
@@ -136,11 +152,18 @@ async function captureUrl(args: Args): Promise<ConversionResult> {
return await extractContent(html, args.url); return await extractContent(html, args.url);
} finally { } finally {
if (cdp) { if (reusing) {
try { await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); } catch {} if (cdp && targetId) {
cdp.close(); try { await cdp.send("Target.closeTarget", { targetId }, { timeoutMs: 5_000 }); } catch {}
}
if (cdp) cdp.close();
} else {
if (cdp) {
try { await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); } catch {}
cdp.close();
}
if (chrome) killChrome(chrome);
} }
killChrome(chrome);
} }
} }
@@ -4,6 +4,7 @@
"type": "module", "type": "module",
"dependencies": { "dependencies": {
"@mozilla/readability": "^0.6.0", "@mozilla/readability": "^0.6.0",
"baoyu-chrome-cdp": "file:./vendor/baoyu-chrome-cdp",
"defuddle": "^0.10.0", "defuddle": "^0.10.0",
"jsdom": "^24.1.3", "jsdom": "^24.1.3",
"linkedom": "^0.18.12", "linkedom": "^0.18.12",
@@ -0,0 +1,9 @@
{
"name": "baoyu-chrome-cdp",
"private": true,
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts"
}
}
@@ -0,0 +1,408 @@
import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import fs from "node:fs";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import process from "node:process";
export type PlatformCandidates = {
darwin?: string[];
win32?: string[];
default: string[];
};
type PendingRequest = {
resolve: (value: unknown) => void;
reject: (error: Error) => void;
timer: ReturnType<typeof setTimeout> | null;
};
type CdpSendOptions = {
sessionId?: string;
timeoutMs?: number;
};
type FetchJsonOptions = {
timeoutMs?: number;
};
type FindChromeExecutableOptions = {
candidates: PlatformCandidates;
envNames?: string[];
};
type ResolveSharedChromeProfileDirOptions = {
envNames?: string[];
appDataDirName?: string;
profileDirName?: string;
wslWindowsHome?: string | null;
};
type FindExistingChromeDebugPortOptions = {
profileDir: string;
timeoutMs?: number;
};
type LaunchChromeOptions = {
chromePath: string;
profileDir: string;
port: number;
url?: string;
headless?: boolean;
extraArgs?: string[];
};
type ChromeTargetInfo = {
targetId: string;
url: string;
type: string;
};
type OpenPageSessionOptions = {
cdp: CdpConnection;
reusing: boolean;
url: string;
matchTarget: (target: ChromeTargetInfo) => boolean;
enablePage?: boolean;
enableRuntime?: boolean;
enableDom?: boolean;
enableNetwork?: boolean;
activateTarget?: boolean;
};
export type PageSession = {
sessionId: string;
targetId: string;
};
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function getFreePort(fixedEnvName?: string): Promise<number> {
const fixed = fixedEnvName ? Number.parseInt(process.env[fixedEnvName] ?? "", 10) : NaN;
if (Number.isInteger(fixed) && fixed > 0) return fixed;
return await new Promise((resolve, reject) => {
const server = net.createServer();
server.unref();
server.on("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close(() => reject(new Error("Unable to allocate a free TCP port.")));
return;
}
const port = address.port;
server.close((err) => {
if (err) reject(err);
else resolve(port);
});
});
});
}
export function findChromeExecutable(options: FindChromeExecutableOptions): string | undefined {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override && fs.existsSync(override)) return override;
}
const candidates = process.platform === "darwin"
? options.candidates.darwin ?? options.candidates.default
: process.platform === "win32"
? options.candidates.win32 ?? options.candidates.default
: options.candidates.default;
for (const candidate of candidates) {
if (fs.existsSync(candidate)) return candidate;
}
return undefined;
}
export function resolveSharedChromeProfileDir(options: ResolveSharedChromeProfileDirOptions = {}): string {
for (const envName of options.envNames ?? []) {
const override = process.env[envName]?.trim();
if (override) return path.resolve(override);
}
const appDataDirName = options.appDataDirName ?? "baoyu-skills";
const profileDirName = options.profileDirName ?? "chrome-profile";
if (options.wslWindowsHome) {
return path.join(options.wslWindowsHome, ".local", "share", appDataDirName, profileDirName);
}
const base = process.platform === "darwin"
? path.join(os.homedir(), "Library", "Application Support")
: process.platform === "win32"
? (process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming"))
: (process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share"));
return path.join(base, appDataDirName, profileDirName);
}
async function fetchWithTimeout(url: string, timeoutMs?: number): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return await fetch(url, { redirect: "follow" });
const ctl = new AbortController();
const timer = setTimeout(() => ctl.abort(), timeoutMs);
try {
return await fetch(url, { redirect: "follow", signal: ctl.signal });
} finally {
clearTimeout(timer);
}
}
async function fetchJson<T = unknown>(url: string, options: FetchJsonOptions = {}): Promise<T> {
const response = await fetchWithTimeout(url, options.timeoutMs);
if (!response.ok) {
throw new Error(`Request failed: ${response.status} ${response.statusText}`);
}
return await response.json() as T;
}
async function isDebugPortReady(port: number, timeoutMs = 3_000): Promise<boolean> {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs }
);
return !!version.webSocketDebuggerUrl;
} catch {
return false;
}
}
export async function findExistingChromeDebugPort(options: FindExistingChromeDebugPortOptions): Promise<number | null> {
const timeoutMs = options.timeoutMs ?? 3_000;
const portFile = path.join(options.profileDir, "DevToolsActivePort");
try {
const content = fs.readFileSync(portFile, "utf-8");
const [portLine] = content.split(/\r?\n/);
const port = Number.parseInt(portLine?.trim() ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
} catch {}
if (process.platform === "win32") return null;
try {
const result = spawnSync("ps", ["aux"], { encoding: "utf-8", timeout: 5_000 });
if (result.status !== 0 || !result.stdout) return null;
const lines = result.stdout
.split("\n")
.filter((line) => line.includes(options.profileDir) && line.includes("--remote-debugging-port="));
for (const line of lines) {
const portMatch = line.match(/--remote-debugging-port=(\d+)/);
const port = Number.parseInt(portMatch?.[1] ?? "", 10);
if (port > 0 && await isDebugPortReady(port, timeoutMs)) return port;
}
} catch {}
return null;
}
export async function waitForChromeDebugPort(
port: number,
timeoutMs: number,
options?: { includeLastError?: boolean }
): Promise<string> {
const start = Date.now();
let lastError: unknown = null;
while (Date.now() - start < timeoutMs) {
try {
const version = await fetchJson<{ webSocketDebuggerUrl?: string }>(
`http://127.0.0.1:${port}/json/version`,
{ timeoutMs: 5_000 }
);
if (version.webSocketDebuggerUrl) return version.webSocketDebuggerUrl;
lastError = new Error("Missing webSocketDebuggerUrl");
} catch (error) {
lastError = error;
}
await sleep(200);
}
if (options?.includeLastError && lastError) {
throw new Error(
`Chrome debug port not ready: ${lastError instanceof Error ? lastError.message : String(lastError)}`
);
}
throw new Error("Chrome debug port not ready");
}
export class CdpConnection {
private ws: WebSocket;
private nextId = 0;
private pending = new Map<number, PendingRequest>();
private eventHandlers = new Map<string, Set<(params: unknown) => void>>();
private defaultTimeoutMs: number;
private constructor(ws: WebSocket, defaultTimeoutMs = 15_000) {
this.ws = ws;
this.defaultTimeoutMs = defaultTimeoutMs;
this.ws.addEventListener("message", (event) => {
try {
const data = typeof event.data === "string"
? event.data
: new TextDecoder().decode(event.data as ArrayBuffer);
const msg = JSON.parse(data) as {
id?: number;
method?: string;
params?: unknown;
result?: unknown;
error?: { message?: string };
};
if (msg.method) {
const handlers = this.eventHandlers.get(msg.method);
if (handlers) {
handlers.forEach((handler) => handler(msg.params));
}
}
if (msg.id) {
const pending = this.pending.get(msg.id);
if (pending) {
this.pending.delete(msg.id);
if (pending.timer) clearTimeout(pending.timer);
if (msg.error?.message) pending.reject(new Error(msg.error.message));
else pending.resolve(msg.result);
}
}
} catch {}
});
this.ws.addEventListener("close", () => {
for (const [id, pending] of this.pending.entries()) {
this.pending.delete(id);
if (pending.timer) clearTimeout(pending.timer);
pending.reject(new Error("CDP connection closed."));
}
});
}
static async connect(
url: string,
timeoutMs: number,
options?: { defaultTimeoutMs?: number }
): Promise<CdpConnection> {
const ws = new WebSocket(url);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("CDP connection timeout.")), timeoutMs);
ws.addEventListener("open", () => {
clearTimeout(timer);
resolve();
});
ws.addEventListener("error", () => {
clearTimeout(timer);
reject(new Error("CDP connection failed."));
});
});
return new CdpConnection(ws, options?.defaultTimeoutMs ?? 15_000);
}
on(method: string, handler: (params: unknown) => void): void {
if (!this.eventHandlers.has(method)) {
this.eventHandlers.set(method, new Set());
}
this.eventHandlers.get(method)?.add(handler);
}
off(method: string, handler: (params: unknown) => void): void {
this.eventHandlers.get(method)?.delete(handler);
}
async send<T = unknown>(method: string, params?: Record<string, unknown>, options?: CdpSendOptions): Promise<T> {
const id = ++this.nextId;
const message: Record<string, unknown> = { id, method };
if (params) message.params = params;
if (options?.sessionId) message.sessionId = options.sessionId;
const timeoutMs = options?.timeoutMs ?? this.defaultTimeoutMs;
const result = await new Promise<unknown>((resolve, reject) => {
const timer = timeoutMs > 0
? setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP timeout: ${method}`));
}, timeoutMs)
: null;
this.pending.set(id, { resolve, reject, timer });
this.ws.send(JSON.stringify(message));
});
return result as T;
}
close(): void {
try {
this.ws.close();
} catch {}
}
}
export async function launchChrome(options: LaunchChromeOptions): Promise<ChildProcess> {
await fs.promises.mkdir(options.profileDir, { recursive: true });
const args = [
`--remote-debugging-port=${options.port}`,
`--user-data-dir=${options.profileDir}`,
"--no-first-run",
"--no-default-browser-check",
...(options.extraArgs ?? []),
];
if (options.headless) args.push("--headless=new");
if (options.url) args.push(options.url);
return spawn(options.chromePath, args, { stdio: "ignore" });
}
export function killChrome(chrome: ChildProcess): void {
try {
chrome.kill("SIGTERM");
} catch {}
setTimeout(() => {
if (!chrome.killed) {
try {
chrome.kill("SIGKILL");
} catch {}
}
}, 2_000).unref?.();
}
export async function openPageSession(options: OpenPageSessionOptions): Promise<PageSession> {
let targetId: string;
if (options.reusing) {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
} else {
const targets = await options.cdp.send<{ targetInfos: ChromeTargetInfo[] }>("Target.getTargets");
const existing = targets.targetInfos.find(options.matchTarget);
if (existing) {
targetId = existing.targetId;
} else {
const created = await options.cdp.send<{ targetId: string }>("Target.createTarget", { url: options.url });
targetId = created.targetId;
}
}
const { sessionId } = await options.cdp.send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true }
);
if (options.activateTarget ?? true) {
await options.cdp.send("Target.activateTarget", { targetId });
}
if (options.enablePage) await options.cdp.send("Page.enable", {}, { sessionId });
if (options.enableRuntime) await options.cdp.send("Runtime.enable", {}, { sessionId });
if (options.enableDom) await options.cdp.send("DOM.enable", {}, { sessionId });
if (options.enableNetwork) await options.cdp.send("Network.enable", {}, { sessionId });
return { sessionId, targetId };
}