Compare commits

..

11 Commits

Author SHA1 Message Date
Jim Liu 宝玉 ce259e4547 chore: release v1.46.0 2026-03-05 14:34:35 -06:00
Jim Liu 宝玉 fff1a54b6b feat(baoyu-url-to-markdown): add --output-dir option for custom output directory
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 14:34:07 -06:00
Jim Liu 宝玉 9437581c48 chore: release v1.45.1 2026-03-05 10:31:54 -06:00
Jim Liu 宝玉 bd4db203f8 refactor(skills): replace hardcoded npx -y bun with ${BUN_X} runtime variable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 10:31:29 -06:00
Jim Liu 宝玉 51387498a5 refactor(project): add BUN_X runtime detection with bun-first fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 10:31:24 -06:00
Jim Liu 宝玉 6d1336d8bd chore: release v1.45.0 2026-03-05 00:55:59 -06:00
Jim Liu 宝玉 42931d6294 feat(baoyu-post-to-x): add post-composition verification and improve stability
Add automatic check after article images are inserted: verifies remaining
XIMGPH placeholders and expected vs actual image count. Increase CDP timeout
to 60s and add 3s DOM stabilization delay between image insertions.
2026-03-05 00:55:56 -06:00
Jim Liu 宝玉 4c9e57777b chore: release v1.44.0 2026-03-05 00:37:52 -06:00
Jim Liu 宝玉 210905ef66 feat(baoyu-url-to-markdown): add media download and cover image extraction
- Add --download-media flag to download images/videos to local dirs and rewrite markdown links
- Extract coverImage from page meta (og:image) into YAML front matter
- Handle data-src lazy loading for WeChat and similar sites
- Add EXTEND.md preferences with first-time setup for download_media setting
- Add media-localizer module adapted from x-to-markdown
2026-03-05 00:37:09 -06:00
Jim Liu 宝玉 bd4de7b995 chore: release v1.43.2 2026-03-05 00:12:44 -06:00
Jim Liu 宝玉 832f06e86e refactor(baoyu-url-to-markdown): replace custom extraction with defuddle library 2026-03-05 00:11:59 -06:00
26 changed files with 830 additions and 924 deletions
+1 -1
View File
@@ -6,7 +6,7 @@
},
"metadata": {
"description": "Skills shared by Baoyu for improving daily work efficiency",
"version": "1.43.1"
"version": "1.46.0"
},
"plugins": [
{
+30
View File
@@ -2,6 +2,36 @@
English | [中文](./CHANGELOG.zh.md)
## 1.46.0 - 2026-03-05
### Features
- Add `--output-dir` option to url-to-markdown for custom output directory with auto-generated filenames
## 1.45.1 - 2026-03-05
### Refactor
- Replace hardcoded `npx -y bun` with `${BUN_X}` runtime variable across all skills — prefers native `bun`, falls back to `npx -y bun`
- Add Runtime Detection section to CLAUDE.md and Script Directory instructions in all SKILL.md files
## 1.45.0 - 2026-03-05
### Features
- `baoyu-post-to-x`: add post-composition verification for X Articles — automatically checks remaining placeholders and image count after all images are inserted
- `baoyu-post-to-x`: increase CDP timeout to 60s and add 3s DOM stabilization delay between image insertions for long articles
## 1.44.0 - 2026-03-05
### Features
- `baoyu-url-to-markdown`: add `--download-media` flag to download images and videos to local directories, rewriting markdown links to local paths
- `baoyu-url-to-markdown`: extract cover image from page meta (og:image) into YAML front matter `coverImage` field
- `baoyu-url-to-markdown`: handle `data-src` lazy loading for WeChat and similar sites
- `baoyu-url-to-markdown`: add EXTEND.md preferences with first-time setup for media download behavior
## 1.43.2 - 2026-03-05
### Refactor
- `baoyu-url-to-markdown`: replace custom HTML extraction (linkedom + Readability + Turndown) with defuddle library for cleaner content extraction and markdown conversion
## 1.43.1 - 2026-03-02
### Features
+30
View File
@@ -2,6 +2,36 @@
[English](./CHANGELOG.md) | 中文
## 1.46.0 - 2026-03-05
### 新功能
- 为 url-to-markdown 新增 `--output-dir` 选项,支持自定义输出目录并自动生成文件名
## 1.45.1 - 2026-03-05
### 重构
- 将所有技能中硬编码的 `npx -y bun` 替换为 `${BUN_X}` 运行时变量——优先使用原生 `bun`,回退到 `npx -y bun`
- 在 CLAUDE.md 中新增运行时检测章节,在所有 SKILL.md 的脚本目录说明中添加运行时解析步骤
## 1.45.0 - 2026-03-05
### 新功能
- `baoyu-post-to-x`:X 文章发布后自动验证——检查残留占位符和图片数量是否正确
- `baoyu-post-to-x`:增加 CDP 超时至 60 秒,图片插入间隔增加 3 秒 DOM 稳定等待,改善长文章发布稳定性
## 1.44.0 - 2026-03-05
### 新功能
- `baoyu-url-to-markdown`:新增 `--download-media` 参数,支持下载图片和视频到本地目录,并将 Markdown 中的链接改写为本地路径
- `baoyu-url-to-markdown`:从页面 meta 信息(og:image)提取封面图,写入 YAML front matter 的 `coverImage` 字段
- `baoyu-url-to-markdown`:支持 `data-src` 懒加载图片提取(兼容微信公众号等站点)
- `baoyu-url-to-markdown`:新增 EXTEND.md 偏好设置,支持首次使用引导配置媒体下载行为
## 1.43.2 - 2026-03-05
### 重构
- `baoyu-url-to-markdown`:使用 defuddle 库替换自定义 HTML 提取逻辑(linkedom + Readability + Turndown),简化内容提取和 Markdown 转换
## 1.43.1 - 2026-03-02
### 新功能
+35 -10
View File
@@ -43,27 +43,51 @@ Each skill contains:
## Running Skills
All scripts run via Bun (no build step):
All scripts are TypeScript, executed via Bun runtime (no build step).
### Runtime Detection (`${BUN_X}`)
Before running any script, the agent MUST detect the runtime **once per session** and set `${BUN_X}`:
```bash
npx -y bun skills/<skill>/scripts/main.ts [options]
# Detect runtime (run once, reuse result)
if command -v bun &>/dev/null; then
BUN_X="bun"
elif command -v npx &>/dev/null; then
BUN_X="npx -y bun"
else
echo "Error: Neither bun nor npx found. Install bun: curl -fsSL https://bun.sh/install | bash"
exit 1
fi
```
| Priority | Condition | `${BUN_X}` value | Notes |
|----------|-----------|-------------------|-------|
| 1 | `bun` installed | `bun` | Fastest, native execution |
| 2 | `npx` available | `npx -y bun` | Downloads bun on first run via npm |
| 3 | Neither found | Error + install guide | Suggest: `curl -fsSL https://bun.sh/install \| bash` |
### Script Execution
```bash
${BUN_X} skills/<skill>/scripts/main.ts [options]
```
Examples:
```bash
# Text generation
npx -y bun skills/baoyu-danger-gemini-web/scripts/main.ts "Hello"
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts "Hello"
# Image generation
npx -y bun skills/baoyu-danger-gemini-web/scripts/main.ts --prompt "A cat" --image cat.png
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts --prompt "A cat" --image cat.png
# From prompt files
npx -y bun skills/baoyu-danger-gemini-web/scripts/main.ts --promptfiles system.md content.md --image out.png
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts --promptfiles system.md content.md --image out.png
```
## Key Dependencies
- **Bun**: TypeScript runtime (via `npx -y bun`)
- **Bun**: TypeScript runtime (native `bun` preferred, fallback `npx -y bun`)
- **Chrome**: Required for `baoyu-danger-gemini-web` auth and `baoyu-post-to-x` automation
- **No npm packages**: Self-contained TypeScript, no external dependencies
@@ -176,7 +200,8 @@ Every SKILL.md with scripts MUST include this section after Usage:
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Replace all `${SKILL_DIR}` in this document with the actual path
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `${SKILL_DIR}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
@@ -185,7 +210,7 @@ Every SKILL.md with scripts MUST include this section after Usage:
| `scripts/other.ts` | Other functionality |
```
When referencing scripts in workflow sections, use `${SKILL_DIR}/scripts/<name>.ts` so agents can resolve the correct path.
When referencing scripts in workflow sections, use `${BUN_X} ${SKILL_DIR}/scripts/<name>.ts` so agents can resolve the correct runtime and path.
### Progressive Disclosure
@@ -320,13 +345,13 @@ When adding, updating, or deleting styles for `baoyu-comic`, follow this workflo
- Add auto-selection entry if applicable
3. **Generate showcase image**:
```bash
npx -y bun skills/baoyu-danger-gemini-web/scripts/main.ts \
${BUN_X} skills/baoyu-danger-gemini-web/scripts/main.ts \
--prompt "A single comic book page in <style-name> style showing [appropriate scene]. Features: [style characteristics from style definition]. 3:4 portrait aspect ratio comic page." \
--image screenshots/comic-styles/<style-name>.png
```
4. **Compress to WebP**:
```bash
npx -y bun skills/baoyu-compress-image/scripts/main.ts screenshots/comic-styles/<style-name>.png
${BUN_X} skills/baoyu-compress-image/scripts/main.ts screenshots/comic-styles/<style-name>.png
```
5. **Update both READMEs** (`README.md` and `README.zh.md`):
- Add style to `--style` options
+3 -2
View File
@@ -107,6 +107,7 @@ Details: [references/auto-selection.md](references/auto-selection.md)
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Replace all `${SKILL_DIR}` in this document with the actual path
4. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
**Script Reference**:
| Script | Purpose |
@@ -209,7 +210,7 @@ Analyze → [Check Existing?] → [Confirm: Style + Reviews] → Storyboard →
- **Backup rule**: If `characters/characters.png` exists, rename to `characters/characters-backup-YYYYMMDD-HHMMSS.png`
```bash
# Use Reference Sheet Prompt from characters/characters.md
npx -y bun ${SKILL_DIR}/../baoyu-image-gen/scripts/main.ts \
${BUN_X} ${SKILL_DIR}/../baoyu-image-gen/scripts/main.ts \
--promptfiles characters/characters.md \
--image characters/characters.png --ar 4:3
```
@@ -233,7 +234,7 @@ Compress to reduce token usage when used as reference image:
```bash
# Example: ALWAYS include --ref for consistency
npx -y bun ${SKILL_DIR}/../baoyu-image-gen/scripts/main.ts \
${BUN_X} ${SKILL_DIR}/../baoyu-image-gen/scripts/main.ts \
--promptfiles prompts/01-page-xxx.md \
--image 01-page-xxx.png --ar 3:4 \
--ref characters/characters.png
+2 -2
View File
@@ -431,7 +431,7 @@ With confirmed prompts from Step 5/6:
```bash
# Each page generation MUST include --ref
npx -y bun ${SKILL_DIR}/../baoyu-image-gen/scripts/main.ts \
${BUN_X} ${SKILL_DIR}/../baoyu-image-gen/scripts/main.ts \
--promptfiles prompts/01-page-xxx.md \
--image 01-page-xxx.png \
--ar 3:4 \
@@ -474,7 +474,7 @@ If image generation skill supports `--sessionId`:
After all images generated:
```bash
npx -y bun ${SKILL_DIR}/scripts/merge-to-pdf.ts <comic-dir>
${BUN_X} ${SKILL_DIR}/scripts/merge-to-pdf.ts <comic-dir>
```
Creates `{topic-slug}.pdf` with all pages as full-page images.
+6 -6
View File
@@ -9,7 +9,7 @@ Compresses images using best available tool (sips → cwebp → ImageMagick →
## Script Directory
Scripts in `scripts/` subdirectory. Replace `${SKILL_DIR}` with this SKILL.md's directory path.
Scripts in `scripts/` subdirectory. `${SKILL_DIR}` = this SKILL.md's directory path. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun. Replace `${SKILL_DIR}` and `${BUN_X}` with actual values.
| Script | Purpose |
|--------|---------|
@@ -48,7 +48,7 @@ test -f "$HOME/.baoyu-skills/baoyu-compress-image/EXTEND.md" && echo "user"
## Usage
```bash
npx -y bun ${SKILL_DIR}/scripts/main.ts <input> [options]
${BUN_X} ${SKILL_DIR}/scripts/main.ts <input> [options]
```
## Options
@@ -67,16 +67,16 @@ npx -y bun ${SKILL_DIR}/scripts/main.ts <input> [options]
```bash
# Single file → WebP (replaces original)
npx -y bun ${SKILL_DIR}/scripts/main.ts image.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts image.png
# Keep PNG format
npx -y bun ${SKILL_DIR}/scripts/main.ts image.png -f png --keep
${BUN_X} ${SKILL_DIR}/scripts/main.ts image.png -f png --keep
# Directory recursive
npx -y bun ${SKILL_DIR}/scripts/main.ts ./images/ -r -q 75
${BUN_X} ${SKILL_DIR}/scripts/main.ts ./images/ -r -q 75
# JSON output
npx -y bun ${SKILL_DIR}/scripts/main.ts image.png --json
${BUN_X} ${SKILL_DIR}/scripts/main.ts image.png --json
```
**Output**:
+11 -10
View File
@@ -14,7 +14,8 @@ Text/image generation via Gemini Web API. Supports reference images and multi-tu
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Replace all `${SKILL_DIR}` in this document with the actual path
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `${SKILL_DIR}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
@@ -75,23 +76,23 @@ test -f "$HOME/.baoyu-skills/baoyu-danger-gemini-web/EXTEND.md" && echo "user"
```bash
# Text generation
npx -y bun ${SKILL_DIR}/scripts/main.ts "Your prompt"
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "Your prompt" --model gemini-3-flash
${BUN_X} ${SKILL_DIR}/scripts/main.ts "Your prompt"
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "Your prompt" --model gemini-3-flash
# Image generation
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cute cat" --image cat.png
npx -y bun ${SKILL_DIR}/scripts/main.ts --promptfiles system.md content.md --image out.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cute cat" --image cat.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --promptfiles system.md content.md --image out.png
# Vision input (reference images)
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "Describe this" --reference image.png
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "Create variation" --reference a.png --image out.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "Describe this" --reference image.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "Create variation" --reference a.png --image out.png
# Multi-turn conversation
npx -y bun ${SKILL_DIR}/scripts/main.ts "Remember: 42" --sessionId session-abc
npx -y bun ${SKILL_DIR}/scripts/main.ts "What number?" --sessionId session-abc
${BUN_X} ${SKILL_DIR}/scripts/main.ts "Remember: 42" --sessionId session-abc
${BUN_X} ${SKILL_DIR}/scripts/main.ts "What number?" --sessionId session-abc
# JSON output
npx -y bun ${SKILL_DIR}/scripts/main.ts "Hello" --json
${BUN_X} ${SKILL_DIR}/scripts/main.ts "Hello" --json
```
## Options
+5 -4
View File
@@ -16,6 +16,7 @@ Scripts located in `scripts/` subdirectory.
**Path Resolution**:
1. `SKILL_DIR` = this SKILL.md's directory
2. Script path = `${SKILL_DIR}/scripts/main.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
## Consent Requirement
@@ -135,10 +136,10 @@ Full reference: [references/config/first-time-setup.md](references/config/first-
## Usage
```bash
npx -y bun ${SKILL_DIR}/scripts/main.ts <url>
npx -y bun ${SKILL_DIR}/scripts/main.ts <url> -o output.md
npx -y bun ${SKILL_DIR}/scripts/main.ts <url> --download-media
npx -y bun ${SKILL_DIR}/scripts/main.ts <url> --json
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url>
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> -o output.md
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> --download-media
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> --json
```
## Options
+6 -6
View File
@@ -9,7 +9,7 @@ Transforms plain text or markdown files into well-structured markdown with prope
## Script Directory
Scripts in `scripts/` subdirectory. Replace `${SKILL_DIR}` with this SKILL.md's directory path.
Scripts in `scripts/` subdirectory. `${SKILL_DIR}` = this SKILL.md's directory path. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun. Replace `${SKILL_DIR}` and `${BUN_X}` with actual values.
| Script | Purpose |
|--------|---------|
@@ -210,7 +210,7 @@ Example:
After saving, **must** run the formatting script:
```bash
npx -y bun ${SKILL_DIR}/scripts/main.ts {output-file-path} [options]
${BUN_X} ${SKILL_DIR}/scripts/main.ts {output-file-path} [options]
```
**Script Options:**
@@ -229,16 +229,16 @@ npx -y bun ${SKILL_DIR}/scripts/main.ts {output-file-path} [options]
```bash
# Default: spacing + emphasis enabled, quotes disabled
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md
# Enable all features including quote replacement
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --quotes
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --quotes
# Only fix emphasis issues, skip spacing
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --no-spacing
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --no-spacing
# Disable all processing except frontmatter formatting
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --no-spacing --no-emphasis
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --no-spacing --no-emphasis
```
**Script performs (based on options):**
+13 -12
View File
@@ -12,6 +12,7 @@ Official API-based image generation. Supports OpenAI, Google, DashScope (阿里
**Agent Execution**:
1. `SKILL_DIR` = this SKILL.md file's directory
2. Script path = `${SKILL_DIR}/scripts/main.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
## Step 0: Load Preferences ⛔ BLOCKING
@@ -44,34 +45,34 @@ Schema: `references/config/preferences-schema.md`
```bash
# Basic
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image cat.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image cat.png
# With aspect ratio
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A landscape" --image out.png --ar 16:9
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A landscape" --image out.png --ar 16:9
# High quality
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --quality 2k
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --quality 2k
# From prompt files
npx -y bun ${SKILL_DIR}/scripts/main.ts --promptfiles system.md content.md --image out.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --promptfiles system.md content.md --image out.png
# With reference images (Google multimodal or OpenAI edits)
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "Make blue" --image out.png --ref source.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "Make blue" --image out.png --ref source.png
# With reference images (explicit provider/model)
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "Make blue" --image out.png --provider google --model gemini-3-pro-image-preview --ref source.png
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "Make blue" --image out.png --provider google --model gemini-3-pro-image-preview --ref source.png
# Specific provider
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider openai
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider openai
# DashScope (阿里通义万象)
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "一只可爱的猫" --image out.png --provider dashscope
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "一只可爱的猫" --image out.png --provider dashscope
# Replicate (google/nano-banana-pro)
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate
# Replicate with specific model
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana
```
## Options
@@ -136,10 +137,10 @@ Examples:
```bash
# Use Replicate default model
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate
# Override model explicitly
npx -y bun ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana
${BUN_X} ${SKILL_DIR}/scripts/main.ts --prompt "A cat" --image out.png --provider replicate --model google/nano-banana
```
## Provider Selection
+8 -8
View File
@@ -9,7 +9,7 @@ Converts Markdown files to beautifully styled HTML with inline CSS, optimized fo
## Script Directory
**Agent Execution**: Determine this SKILL.md directory as `SKILL_DIR`, then use `${SKILL_DIR}/scripts/<name>.ts`.
**Agent Execution**: Determine this SKILL.md directory as `SKILL_DIR`. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun. Replace `${SKILL_DIR}` and `${BUN_X}` with actual values.
| Script | Purpose |
|--------|---------|
@@ -97,7 +97,7 @@ test -f "$HOME/.baoyu-skills/baoyu-post-to-wechat/EXTEND.md" && grep -o 'default
### Step 2: Convert
```bash
npx -y bun ${SKILL_DIR}/scripts/main.ts <markdown_file> --theme <theme>
${BUN_X} ${SKILL_DIR}/scripts/main.ts <markdown_file> --theme <theme>
```
### Step 3: Report Result
@@ -107,7 +107,7 @@ Display the output path from JSON result. If backup was created, mention it.
## Usage
```bash
npx -y bun ${SKILL_DIR}/scripts/main.ts <markdown_file> [options]
${BUN_X} ${SKILL_DIR}/scripts/main.ts <markdown_file> [options]
```
**Options:**
@@ -144,19 +144,19 @@ npx -y bun ${SKILL_DIR}/scripts/main.ts <markdown_file> [options]
```bash
# Basic conversion (uses default theme, removes first heading)
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md
# With specific theme
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --theme grace
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --theme grace
# Theme with custom color
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --theme modern --color red
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --theme modern --color red
# Keep the first heading in content
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --keep-title
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --keep-title
# Override title
npx -y bun ${SKILL_DIR}/scripts/main.ts article.md --title "My Article"
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --title "My Article"
```
## Output
+7 -7
View File
@@ -11,7 +11,7 @@ description: Posts content to WeChat Official Account (微信公众号) via API
## Script Directory
**Agent Execution**: Determine this SKILL.md directory as `SKILL_DIR`, then use `${SKILL_DIR}/scripts/<name>.ts`.
**Agent Execution**: Determine this SKILL.md directory as `SKILL_DIR`, then use `${SKILL_DIR}/scripts/<name>.ts`. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun.
| Script | Purpose |
|--------|---------|
@@ -88,7 +88,7 @@ chrome_profile_path: /path/to/chrome/profile
Before first use, suggest running the environment check. User can skip if they prefer.
```bash
npx -y bun ${SKILL_DIR}/scripts/check-permissions.ts
${BUN_X} ${SKILL_DIR}/scripts/check-permissions.ts
```
Checks: Chrome, profile isolation, Bun, Accessibility, clipboard, paste keystroke, API credentials, Chrome conflicts.
@@ -111,8 +111,8 @@ Checks: Chrome, profile isolation, Bun, Accessibility, clipboard, paste keystrok
For short posts with multiple images (up to 9):
```bash
npx -y bun ${SKILL_DIR}/scripts/wechat-browser.ts --markdown article.md --images ./images/
npx -y bun ${SKILL_DIR}/scripts/wechat-browser.ts --title "标题" --content "内容" --image img.png --submit
${BUN_X} ${SKILL_DIR}/scripts/wechat-browser.ts --markdown article.md --images ./images/
${BUN_X} ${SKILL_DIR}/scripts/wechat-browser.ts --title "标题" --content "内容" --image img.png --submit
```
See [references/image-text-posting.md](references/image-text-posting.md) for details.
@@ -247,7 +247,7 @@ WECHAT_APP_SECRET=<user_input>
**API method** (accepts `.md` or `.html`):
```bash
npx -y bun ${SKILL_DIR}/scripts/wechat-api.ts <file> --theme <theme> [--color <color>] [--title <title>] [--summary <summary>] [--author <author>] [--cover <cover_path>]
${BUN_X} ${SKILL_DIR}/scripts/wechat-api.ts <file> --theme <theme> [--color <color>] [--title <title>] [--summary <summary>] [--author <author>] [--cover <cover_path>]
```
**CRITICAL**: Always include `--theme` parameter. Never omit it, even if using `default`. Only include `--color` if explicitly set by user or EXTEND.md.
@@ -266,8 +266,8 @@ If script parameters do not expose the two comment fields, still ensure final AP
**Browser method** (accepts `--markdown` or `--html`):
```bash
npx -y bun ${SKILL_DIR}/scripts/wechat-article.ts --markdown <markdown_file> --theme <theme> [--color <color>]
npx -y bun ${SKILL_DIR}/scripts/wechat-article.ts --html <html_file>
${BUN_X} ${SKILL_DIR}/scripts/wechat-article.ts --markdown <markdown_file> --theme <theme> [--color <color>]
${BUN_X} ${SKILL_DIR}/scripts/wechat-article.ts --html <html_file>
```
### Step 5: Completion Report
@@ -6,13 +6,13 @@ Post markdown articles to WeChat Official Account with full formatting support.
```bash
# Post markdown article
npx -y bun ./scripts/wechat-article.ts --markdown article.md
${BUN_X} ./scripts/wechat-article.ts --markdown article.md
# With theme
npx -y bun ./scripts/wechat-article.ts --markdown article.md --theme grace
${BUN_X} ./scripts/wechat-article.ts --markdown article.md --theme grace
# With explicit options
npx -y bun ./scripts/wechat-article.ts --markdown article.md --author "作者名" --summary "摘要"
${BUN_X} ./scripts/wechat-article.ts --markdown article.md --author "作者名" --summary "摘要"
```
## Parameters
@@ -8,13 +8,13 @@ Post image-text messages with multiple images to WeChat Official Account.
```bash
# Post with images and markdown file (title/content extracted automatically)
npx -y bun ./scripts/wechat-browser.ts --markdown source.md --images ./images/
${BUN_X} ./scripts/wechat-browser.ts --markdown source.md --images ./images/
# Post with explicit title and content
npx -y bun ./scripts/wechat-browser.ts --title "标题" --content "内容" --image img1.png --image img2.png
${BUN_X} ./scripts/wechat-browser.ts --title "标题" --content "内容" --image img1.png --image img2.png
# Save as draft
npx -y bun ./scripts/wechat-browser.ts --markdown source.md --images ./images/ --submit
${BUN_X} ./scripts/wechat-browser.ts --markdown source.md --images ./images/ --submit
```
## Parameters
+13 -6
View File
@@ -15,6 +15,7 @@ Posts text, images, videos, and long-form articles to X via real Chrome browser
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Replace all `${SKILL_DIR}` in this document with the actual path
4. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
**Script Reference**:
| Script | Purpose |
@@ -69,7 +70,7 @@ test -f "$HOME/.baoyu-skills/baoyu-post-to-x/EXTEND.md" && echo "user"
Before first use, suggest running the environment check. User can skip if they prefer.
```bash
npx -y bun ${SKILL_DIR}/scripts/check-paste-permissions.ts
${BUN_X} ${SKILL_DIR}/scripts/check-paste-permissions.ts
```
Checks: Chrome, profile isolation, Bun, Accessibility, clipboard, paste keystroke, Chrome conflicts.
@@ -98,7 +99,7 @@ Checks: Chrome, profile isolation, Bun, Accessibility, clipboard, paste keystrok
Text + up to 4 images.
```bash
npx -y bun ${SKILL_DIR}/scripts/x-browser.ts "Hello!" --image ./photo.png
${BUN_X} ${SKILL_DIR}/scripts/x-browser.ts "Hello!" --image ./photo.png
```
**Parameters**:
@@ -117,7 +118,7 @@ npx -y bun ${SKILL_DIR}/scripts/x-browser.ts "Hello!" --image ./photo.png
Text + video file.
```bash
npx -y bun ${SKILL_DIR}/scripts/x-video.ts "Check this out!" --video ./clip.mp4
${BUN_X} ${SKILL_DIR}/scripts/x-video.ts "Check this out!" --video ./clip.mp4
```
**Parameters**:
@@ -138,7 +139,7 @@ npx -y bun ${SKILL_DIR}/scripts/x-video.ts "Check this out!" --video ./clip.mp4
Quote an existing tweet with comment.
```bash
npx -y bun ${SKILL_DIR}/scripts/x-quote.ts https://x.com/user/status/123 "Great insight!"
${BUN_X} ${SKILL_DIR}/scripts/x-quote.ts https://x.com/user/status/123 "Great insight!"
```
**Parameters**:
@@ -157,8 +158,8 @@ npx -y bun ${SKILL_DIR}/scripts/x-quote.ts https://x.com/user/status/123 "Great
Long-form Markdown articles (requires X Premium).
```bash
npx -y bun ${SKILL_DIR}/scripts/x-article.ts article.md
npx -y bun ${SKILL_DIR}/scripts/x-article.ts article.md --cover ./cover.jpg
${BUN_X} ${SKILL_DIR}/scripts/x-article.ts article.md
${BUN_X} ${SKILL_DIR}/scripts/x-article.ts article.md --cover ./cover.jpg
```
**Parameters**:
@@ -172,6 +173,12 @@ npx -y bun ${SKILL_DIR}/scripts/x-article.ts article.md --cover ./cover.jpg
**Note**: Script opens browser with article filled in. User reviews and publishes manually.
**Post-Composition Check**: The script automatically verifies after all images are inserted:
- Remaining `XIMGPH_` placeholders in editor content
- Expected vs actual image count
If the check fails (warnings in output), alert the user with the specific issues before they publish.
---
## Troubleshooting
+13 -9
View File
@@ -12,13 +12,13 @@ Publish Markdown articles to X Articles editor with rich text formatting and ima
```bash
# Publish markdown article (preview mode)
npx -y bun ${SKILL_DIR}/scripts/x-article.ts article.md
${BUN_X} ${SKILL_DIR}/scripts/x-article.ts article.md
# With custom cover image
npx -y bun ${SKILL_DIR}/scripts/x-article.ts article.md --cover ./cover.jpg
${BUN_X} ${SKILL_DIR}/scripts/x-article.ts article.md --cover ./cover.jpg
# Actually publish
npx -y bun ${SKILL_DIR}/scripts/x-article.ts article.md --submit
${BUN_X} ${SKILL_DIR}/scripts/x-article.ts article.md --submit
```
## Markdown Format
@@ -76,13 +76,13 @@ Convert markdown and inspect structure:
```bash
# Get JSON with all metadata
npx -y bun ${SKILL_DIR}/scripts/md-to-html.ts article.md
${BUN_X} ${SKILL_DIR}/scripts/md-to-html.ts article.md
# Output HTML only
npx -y bun ${SKILL_DIR}/scripts/md-to-html.ts article.md --html-only
${BUN_X} ${SKILL_DIR}/scripts/md-to-html.ts article.md --html-only
# Save HTML to file
npx -y bun ${SKILL_DIR}/scripts/md-to-html.ts article.md --save-html /tmp/article.html
${BUN_X} ${SKILL_DIR}/scripts/md-to-html.ts article.md --save-html /tmp/article.html
```
JSON output:
@@ -132,8 +132,12 @@ JSON output:
- Select the placeholder
- Copy image to clipboard
- Paste to replace selection
9. **Review**: Browser stays open for 60s preview
10. **Publish**: Only with `--submit` flag
9. **Post-Composition Check** (automatic):
- Scan editor for remaining `XIMGPH_` placeholders
- Compare expected vs actual image count
- Warn if issues found
10. **Review**: Browser stays open for 60s preview
11. **Publish**: Only with `--submit` flag
## Example Session
@@ -157,7 +161,7 @@ Claude:
- **No create button**: Ensure X Premium subscription is active
- **Cover upload fails**: Check file path and format (PNG, JPEG)
- **Images not inserting**: Verify placeholders exist in pasted content
- **Content not pasting**: Check HTML clipboard: `npx -y bun ${SKILL_DIR}/scripts/copy-to-clipboard.ts html --file /tmp/test.html`
- **Content not pasting**: Check HTML clipboard: `${BUN_X} ${SKILL_DIR}/scripts/copy-to-clipboard.ts html --file /tmp/test.html`
## How It Works
@@ -9,20 +9,20 @@ If you prefer step-by-step control:
### Step 1: Copy Image to Clipboard
```bash
npx -y bun ${SKILL_DIR}/scripts/copy-to-clipboard.ts image /path/to/image.png
${BUN_X} ${SKILL_DIR}/scripts/copy-to-clipboard.ts image /path/to/image.png
```
### Step 2: Paste from Clipboard
```bash
# Simple paste to frontmost app
npx -y bun ${SKILL_DIR}/scripts/paste-from-clipboard.ts
${BUN_X} ${SKILL_DIR}/scripts/paste-from-clipboard.ts
# Paste to Chrome with retries
npx -y bun ${SKILL_DIR}/scripts/paste-from-clipboard.ts --app "Google Chrome" --retries 5
${BUN_X} ${SKILL_DIR}/scripts/paste-from-clipboard.ts --app "Google Chrome" --retries 5
# Quick paste with shorter delay
npx -y bun ${SKILL_DIR}/scripts/paste-from-clipboard.ts --delay 200
${BUN_X} ${SKILL_DIR}/scripts/paste-from-clipboard.ts --delay 200
```
### Step 3: Use Playwright MCP (if Chrome session available)
@@ -59,7 +59,7 @@ mcp__playwright__browser_take_screenshot filename="preview.png"
User: /post-to-x "Hello from Claude!" --image ./screenshot.png
Claude:
1. Runs: npx -y bun ${SKILL_DIR}/scripts/x-browser.ts "Hello from Claude!" --image ./screenshot.png
1. Runs: ${BUN_X} ${SKILL_DIR}/scripts/x-browser.ts "Hello from Claude!" --image ./screenshot.png
2. Chrome opens with X compose page
3. Text is typed into editor
4. Image is copied to clipboard and pasted
@@ -72,7 +72,7 @@ Claude:
- **Chrome not found**: Set `X_BROWSER_CHROME_PATH` environment variable
- **Not logged in**: First run opens Chrome - log in manually, cookies are saved
- **Image paste fails**:
- Verify clipboard script: `npx -y bun ${SKILL_DIR}/scripts/copy-to-clipboard.ts image <path>`
- Verify clipboard script: `${BUN_X} ${SKILL_DIR}/scripts/copy-to-clipboard.ts image <path>`
- On macOS, grant "Accessibility" permission to Terminal/iTerm in System Settings > Privacy & Security > Accessibility
- Keep Chrome window visible and in front during paste operations
- **osascript permission denied**: Grant Terminal accessibility permissions in System Preferences
+40 -2
View File
@@ -130,7 +130,7 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
try {
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 30_000 });
cdp = await CdpConnection.connect(wsUrl, 30_000, { defaultTimeoutMs: 60_000 });
// Get page target
const targets = await cdp.send<{ targetInfos: Array<{ targetId: string; url: string; type: string }> }>('Target.getTargets');
@@ -148,7 +148,7 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
await cdp.send('DOM.enable', {}, { sessionId });
console.log('[x-article] Waiting for articles page...');
await sleep(3000);
await sleep(1000);
// Wait for and click "create" button
const waitForElement = async (selector: string, timeoutMs = 60_000): Promise<boolean> => {
@@ -644,6 +644,8 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
if (imgUploadOk) {
console.log(`[x-article] Image upload verified (${expectedImgCount} image block(s))`);
// Wait for DraftEditor DOM to stabilize after image insertion
await sleep(3000);
} else {
console.warn(`[x-article] Image upload not detected after 15s`);
if (i === 0) {
@@ -653,6 +655,42 @@ export async function publishArticle(options: ArticleOptions): Promise<void> {
}
console.log('[x-article] All images processed.');
// Final verification: check placeholder residue and image count
console.log('[x-article] Running post-composition verification...');
const finalEditorContent = await cdp.send<{ result: { value: string } }>('Runtime.evaluate', {
expression: `document.querySelector('.DraftEditor-editorContainer [data-contents="true"]')?.innerText || ''`,
returnByValue: true,
}, { sessionId });
const remainingPlaceholders: string[] = [];
for (const img of parsed.contentImages) {
const regex = new RegExp(img.placeholder + '(?!\\d)');
if (regex.test(finalEditorContent.result.value)) {
remainingPlaceholders.push(img.placeholder);
}
}
const finalImgCount = await cdp.send<{ result: { value: number } }>('Runtime.evaluate', {
expression: `document.querySelectorAll('section[data-block="true"][contenteditable="false"] img[src^="blob:"]').length`,
returnByValue: true,
}, { sessionId });
const expectedCount = parsed.contentImages.length;
const actualCount = finalImgCount.result.value;
if (remainingPlaceholders.length > 0 || actualCount < expectedCount) {
console.warn('[x-article] ⚠ POST-COMPOSITION CHECK FAILED:');
if (remainingPlaceholders.length > 0) {
console.warn(`[x-article] Remaining placeholders: ${remainingPlaceholders.join(', ')}`);
}
if (actualCount < expectedCount) {
console.warn(`[x-article] Image count: expected ${expectedCount}, found ${actualCount}`);
}
console.warn('[x-article] Please check the article before publishing.');
} else {
console.log(`[x-article] ✓ Verification passed: ${actualCount} image(s), no remaining placeholders.`);
}
}
// Before preview: blur editor to trigger save
+3 -2
View File
@@ -24,6 +24,7 @@ Transform content into professional slide deck images.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
| Script | Purpose |
|--------|---------|
@@ -550,8 +551,8 @@ options:
### Step 8: Merge to PPTX and PDF
```bash
npx -y bun ${SKILL_DIR}/scripts/merge-to-pptx.ts <slide-deck-dir>
npx -y bun ${SKILL_DIR}/scripts/merge-to-pdf.ts <slide-deck-dir>
${BUN_X} ${SKILL_DIR}/scripts/merge-to-pptx.ts <slide-deck-dir>
${BUN_X} ${SKILL_DIR}/scripts/merge-to-pdf.ts <slide-deck-dir>
```
### Step 9: Output Summary
+86 -10
View File
@@ -14,7 +14,8 @@ Fetches any URL via Chrome CDP and converts HTML to clean markdown.
**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `SKILL_DIR`
2. Script path = `${SKILL_DIR}/scripts/<script-name>.ts`
3. Replace all `${SKILL_DIR}` in this document with the actual path
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
4. Replace all `${SKILL_DIR}` and `${BUN_X}` in this document with actual values
**Script Reference**:
| Script | Purpose |
@@ -46,10 +47,51 @@ test -f "$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md" && echo "user"
├───────────┼───────────────────────────────────────────────────────────────────────────┤
│ Found │ Read, parse, apply settings │
├───────────┼───────────────────────────────────────────────────────────────────────────┤
│ Not found │ Use defaults
│ Not found │ **MUST** run first-time setup (see below) — do NOT silently create defaults
└───────────┴───────────────────────────────────────────────────────────────────────────┘
**EXTEND.md Supports**: Default output directory | Default capture mode | Timeout settings
**EXTEND.md Supports**: Download media by default | Default output directory | Default capture mode | Timeout settings
### First-Time Setup (BLOCKING)
**CRITICAL**: When EXTEND.md is not found, you **MUST use `AskUserQuestion`** to ask the user for their preferences before creating EXTEND.md. **NEVER** create EXTEND.md with defaults without asking. This is a **BLOCKING** operation — do NOT proceed with any conversion until setup is complete.
Use `AskUserQuestion` with ALL questions in ONE call:
**Question 1** — header: "Media", question: "How to handle images and videos in pages?"
- "Ask each time (Recommended)" — After saving markdown, ask whether to download media
- "Always download" — Always download media to local imgs/ and videos/ directories
- "Never download" — Keep original remote URLs in markdown
**Question 2** — header: "Output", question: "Default output directory?"
- "url-to-markdown (Recommended)" — Save to ./url-to-markdown/{domain}/{slug}.md
- (User may choose "Other" to type a custom path)
**Question 3** — header: "Save", question: "Where to save preferences?"
- "User (Recommended)" — ~/.baoyu-skills/ (all projects)
- "Project" — .baoyu-skills/ (this project only)
After user answers, create EXTEND.md at the chosen location, confirm "Preferences saved to [path]", then continue.
Full reference: [references/config/first-time-setup.md](references/config/first-time-setup.md)
### Supported Keys
| Key | Default | Values | Description |
|-----|---------|--------|-------------|
| `download_media` | `ask` | `ask` / `1` / `0` | `ask` = prompt each time, `1` = always download, `0` = never |
| `default_output_dir` | empty | path or empty | Default output directory (empty = `./url-to-markdown/`) |
**EXTEND.md → CLI mapping**:
| EXTEND.md key | CLI argument | Notes |
|---------------|-------------|-------|
| `download_media: 1` | `--download-media` | |
| `default_output_dir: ./posts/` | `--output-dir ./posts/` | Directory path. Do NOT pass to `-o` (which expects a file path) |
**Value priority**:
1. CLI arguments (`--download-media`, `-o`, `--output-dir`)
2. EXTEND.md
3. Skill defaults
## Features
@@ -57,18 +99,25 @@ test -f "$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md" && echo "user"
- Two capture modes: auto or wait-for-user
- Clean markdown output with metadata
- Handles login-required pages via wait mode
- Download images and videos to local directories
## Usage
```bash
# Auto mode (default) - capture when page loads
npx -y bun ${SKILL_DIR}/scripts/main.ts <url>
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url>
# Wait mode - wait for user signal before capture
npx -y bun ${SKILL_DIR}/scripts/main.ts <url> --wait
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> --wait
# Save to specific file
npx -y bun ${SKILL_DIR}/scripts/main.ts <url> -o output.md
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> -o output.md
# Save to a custom output directory (auto-generates filename)
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> --output-dir ./posts/
# Download images and videos to local directories
${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> --download-media
```
## Options
@@ -76,9 +125,11 @@ npx -y bun ${SKILL_DIR}/scripts/main.ts <url> -o output.md
| Option | Description |
|--------|-------------|
| `<url>` | URL to fetch |
| `-o <path>` | Output file path (default: auto-generated) |
| `-o <path>` | Output file path — must be a **file** path, not directory (default: auto-generated) |
| `--output-dir <dir>` | Base output directory — auto-generates `{dir}/{domain}/{slug}.md` (default: `./url-to-markdown/`) |
| `--wait` | Wait for user signal before capturing |
| `--timeout <ms>` | Page load timeout (default: 30000) |
| `--download-media` | Download image/video assets to local `imgs/` and `videos/`, and rewrite markdown links to local relative paths |
## Capture Modes
@@ -98,13 +149,38 @@ YAML front matter with `url`, `title`, `description`, `author`, `published`, `ca
## Output Directory
```
url-to-markdown/<domain>/<slug>.md
```
Default: `url-to-markdown/<domain>/<slug>.md`
With `--output-dir ./posts/`: `./posts/<domain>/<slug>.md`
- `<slug>`: From page title or URL path (kebab-case, 2-6 words)
- Conflict resolution: Append timestamp `<slug>-YYYYMMDD-HHMMSS.md`
When `--download-media` is enabled:
- Images are saved to `imgs/` next to the markdown file
- Videos are saved to `videos/` next to the markdown file
- Markdown media links are rewritten to local relative paths
## Media Download Workflow
Based on `download_media` setting in EXTEND.md:
| Setting | Behavior |
|---------|----------|
| `1` (always) | Run script with `--download-media` flag |
| `0` (never) | Run script without `--download-media` flag |
| `ask` (default) | Follow the ask-each-time flow below |
### Ask-Each-Time Flow
1. Run script **without** `--download-media` → markdown saved
2. Check saved markdown for remote media URLs (`https://` in image/video links)
3. **If no remote media found** → done, no prompt needed
4. **If remote media found** → use `AskUserQuestion`:
- header: "Media", question: "Download N images/videos to local files?"
- "Yes" — Download to local directories
- "No" — Keep remote URLs
5. If user confirms → run script **again** with `--download-media` (overwrites markdown with localized links)
## Environment Variables
| Variable | Description |
@@ -0,0 +1,106 @@
---
name: first-time-setup
description: First-time setup flow for baoyu-url-to-markdown preferences
---
# First-Time Setup
## Overview
When no EXTEND.md is found, guide user through preference setup.
**BLOCKING OPERATION**: This setup MUST complete before ANY other workflow steps. Do NOT:
- Start converting URLs
- Ask about URLs or output paths
- Proceed to any conversion
ONLY ask the questions in this setup flow, save EXTEND.md, then continue.
## Setup Flow
```
No EXTEND.md found
|
v
+---------------------+
| AskUserQuestion |
| (all questions) |
+---------------------+
|
v
+---------------------+
| Create EXTEND.md |
+---------------------+
|
v
Continue conversion
```
## Questions
**Language**: Use user's input language or saved language preference.
Use AskUserQuestion with ALL questions in ONE call:
### Question 1: Download Media
```yaml
header: "Media"
question: "How to handle images and videos in pages?"
options:
- label: "Ask each time (Recommended)"
description: "After saving markdown, ask whether to download media"
- label: "Always download"
description: "Always download media to local imgs/ and videos/ directories"
- label: "Never download"
description: "Keep original remote URLs in markdown"
```
### Question 2: Default Output Directory
```yaml
header: "Output"
question: "Default output directory?"
options:
- label: "url-to-markdown (Recommended)"
description: "Save to ./url-to-markdown/{domain}/{slug}.md"
```
Note: User will likely choose "Other" to type a custom path.
### Question 3: Save Location
```yaml
header: "Save"
question: "Where to save preferences?"
options:
- label: "User (Recommended)"
description: "~/.baoyu-skills/ (all projects)"
- label: "Project"
description: ".baoyu-skills/ (this project only)"
```
## Save Locations
| Choice | Path | Scope |
|--------|------|-------|
| User | `~/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | All projects |
| Project | `.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | Current project |
## After Setup
1. Create directory if needed
2. Write EXTEND.md
3. Confirm: "Preferences saved to [path]"
4. Continue with conversion using saved preferences
## EXTEND.md Template
```md
download_media: [ask/1/0]
default_output_dir: [path or empty]
```
## Modifying Preferences Later
Users can edit EXTEND.md directly or delete it to trigger setup again.
@@ -1,7 +1,5 @@
import { parseHTML } from "linkedom";
import { Readability } from "@mozilla/readability";
import TurndownService from "turndown";
import { gfm } from "turndown-plugin-gfm";
import { JSDOM } from "jsdom";
import { Defuddle } from "defuddle/node";
export interface PageMetadata {
url: string;
@@ -9,6 +7,7 @@ export interface PageMetadata {
description?: string;
author?: string;
published?: string;
coverImage?: string;
captured_at: string;
}
@@ -17,812 +16,58 @@ export interface ConversionResult {
markdown: string;
}
interface ExtractionCandidate {
title: string | null;
byline: string | null;
excerpt: string | null;
published: string | null;
html: string | null;
textContent: string;
method: string;
}
type AnyRecord = Record<string, unknown>;
const MIN_CONTENT_LENGTH = 120;
const GOOD_CONTENT_LENGTH = 900;
const CONTENT_SELECTORS = [
"article",
"main article",
"[role='main'] article",
"[itemprop='articleBody']",
".article-content",
".article-body",
".post-content",
".entry-content",
".story-body",
"main",
"[role='main']",
"#content",
".content",
];
const REMOVE_SELECTORS = [
"script",
"style",
"noscript",
"template",
"iframe",
"svg",
"path",
"nav",
"aside",
"footer",
"header",
"form",
".advertisement",
".ads",
".social-share",
".related-articles",
".comments",
".newsletter",
".cookie-banner",
".cookie-consent",
"[role='navigation']",
"[aria-label*='cookie' i]",
];
const PUBLISHED_TIME_SELECTORS = [
"meta[property='article:published_time']",
"meta[name='pubdate']",
"meta[name='publishdate']",
"meta[name='date']",
"time[datetime]",
];
const ARTICLE_TYPES = new Set([
"Article",
"NewsArticle",
"BlogPosting",
"WebPage",
"ReportageNewsArticle",
]);
const NEXT_DATA_CONTENT_PATHS = [
"props.pageProps.content.body",
"props.pageProps.article.body",
"props.pageProps.article.content",
"props.pageProps.post.body",
"props.pageProps.post.content",
"props.pageProps.data.body",
"props.pageProps.story.body.content",
];
const cleanupAndExtractScriptBody = String.raw`
(function () {
export const absolutizeUrlsScript = String.raw`
(function() {
const baseUrl = document.baseURI || location.href;
function toAbsolute(url) {
if (!url) return url;
try {
return new URL(url, baseUrl).href;
} catch {
return url;
}
try { return new URL(url, baseUrl).href; } catch { return url; }
}
function absolutizeAttr(selector, attr) {
document.querySelectorAll(selector).forEach((el) => {
const value = el.getAttribute(attr);
if (!value) return;
const abs = toAbsolute(value);
if (abs) el.setAttribute(attr, abs);
function absAttr(sel, attr) {
document.querySelectorAll(sel).forEach(el => {
const v = el.getAttribute(attr);
if (v) { const a = toAbsolute(v); if (a) el.setAttribute(attr, a); }
});
}
function absolutizeSrcset(selector) {
document.querySelectorAll(selector).forEach((el) => {
const srcset = el.getAttribute("srcset");
if (!srcset) return;
const normalized = srcset
.split(",")
.map((part) => {
const trimmed = part.trim();
if (!trimmed) return "";
const pieces = trimmed.split(/\s+/);
const url = pieces[0];
const descriptor = pieces.slice(1).join(" ");
const absoluteUrl = toAbsolute(url);
return descriptor ? absoluteUrl + " " + descriptor : absoluteUrl;
})
.filter(Boolean)
.join(", ");
if (normalized) {
el.setAttribute("srcset", normalized);
}
function absSrcset(sel) {
document.querySelectorAll(sel).forEach(el => {
const s = el.getAttribute("srcset");
if (!s) return;
el.setAttribute("srcset", s.split(",").map(p => {
const t = p.trim(); if (!t) return "";
const [url, ...d] = t.split(/\s+/);
return d.length ? toAbsolute(url) + " " + d.join(" ") : toAbsolute(url);
}).filter(Boolean).join(", "));
});
}
absolutizeAttr("a[href]", "href");
absolutizeAttr("img[src], video[src], audio[src], source[src]", "src");
absolutizeSrcset("img[srcset], source[srcset]");
const removeSelectors = [
"noscript",
"template",
".cookie-banner",
".cookie-consent",
".consent-banner",
"[aria-label*='cookie' i]",
".advertisement",
".ads"
];
for (const sel of removeSelectors) {
try {
document.querySelectorAll(sel).forEach((el) => el.remove());
} catch {}
}
function getMeta(names) {
for (const name of names) {
const el = document.querySelector('meta[name="' + name + '"]') || document.querySelector('meta[property="' + name + '"]');
const content = el && el.getAttribute("content");
if (content && content.trim()) return content.trim();
document.querySelectorAll("img[data-src], video[data-src], audio[data-src], source[data-src]").forEach(el => {
const ds = el.getAttribute("data-src");
if (ds && (!el.getAttribute("src") || el.getAttribute("src") === "" || el.getAttribute("src")?.startsWith("data:"))) {
el.setAttribute("src", ds);
}
return undefined;
}
function flattenJsonLdItems(value) {
if (!value || typeof value !== "object") return [];
if (Array.isArray(value)) {
return value.flatMap(flattenJsonLdItems);
}
const obj = value;
if (Array.isArray(obj["@graph"])) {
return obj["@graph"].flatMap(flattenJsonLdItems);
}
return [obj];
}
function extractJsonLdMeta() {
const scripts = document.querySelectorAll('script[type="application/ld+json"]');
for (const script of scripts) {
try {
const parsed = JSON.parse(script.textContent || "");
const items = flattenJsonLdItems(parsed);
for (const item of items) {
const rawType = Array.isArray(item["@type"]) ? item["@type"][0] : item["@type"];
if (typeof rawType !== "string") continue;
if (!["Article", "NewsArticle", "BlogPosting", "WebPage"].includes(rawType)) continue;
const author = (() => {
if (typeof item.author === "string") return item.author;
if (Array.isArray(item.author) && item.author.length > 0) {
const first = item.author[0];
return first && typeof first === "object" ? first.name : undefined;
}
if (item.author && typeof item.author === "object") {
return item.author.name;
}
return undefined;
})();
return {
title: item.headline || item.name,
description: item.description,
author: typeof author === "string" ? author : undefined,
published: item.datePublished || item.dateCreated,
};
}
} catch {}
}
return {};
}
const jsonLd = extractJsonLdMeta();
const timeEl = document.querySelector("time[datetime]");
const title =
getMeta(["og:title", "twitter:title"]) ||
(typeof jsonLd.title === "string" ? jsonLd.title : undefined) ||
document.querySelector("h1")?.textContent?.trim() ||
document.title?.trim() ||
"";
const description =
getMeta(["description", "og:description", "twitter:description"]) ||
(typeof jsonLd.description === "string" ? jsonLd.description : undefined);
const author =
getMeta(["author", "article:author", "twitter:creator"]) ||
(typeof jsonLd.author === "string" ? jsonLd.author : undefined);
const published =
timeEl?.getAttribute("datetime") ||
getMeta(["article:published_time", "datePublished", "publishdate", "date"]) ||
(typeof jsonLd.published === "string" ? jsonLd.published : undefined);
return {
title,
description,
author,
published,
html: document.documentElement.outerHTML,
};
});
absAttr("a[href]", "href");
absAttr("img[src], video[src], audio[src], source[src]", "src");
absSrcset("img[srcset], source[srcset]");
return { html: document.documentElement.outerHTML };
})()
`;
export const cleanupAndExtractScript = cleanupAndExtractScriptBody;
export async function extractContent(html: string, url: string): Promise<ConversionResult> {
const dom = new JSDOM(html, { url });
const result = await Defuddle(dom, url, { markdown: true });
function pickString(...values: unknown[]): string | null {
for (const value of values) {
if (typeof value === "string") {
const trimmed = value.trim();
if (trimmed) return trimmed;
}
}
return null;
}
function generateExcerpt(excerpt: string | null, textContent: string | null): string | null {
if (excerpt) return excerpt;
if (!textContent) return null;
const trimmed = textContent.trim();
if (!trimmed) return null;
return trimmed.length > 200 ? `${trimmed.slice(0, 200)}...` : trimmed;
}
function extractPublishedTime(document: any): string | null {
for (const selector of PUBLISHED_TIME_SELECTORS) {
const el = document.querySelector(selector);
if (!el) continue;
const value = el.getAttribute("content") ?? el.getAttribute("datetime");
if (value && value.trim()) return value.trim();
}
return null;
}
function extractTitle(document: any): string | null {
const ogTitle = document.querySelector("meta[property='og:title']")?.getAttribute("content");
if (ogTitle && ogTitle.trim()) return ogTitle.trim();
const twitterTitle = document
.querySelector("meta[name='twitter:title']")
?.getAttribute("content");
if (twitterTitle && twitterTitle.trim()) return twitterTitle.trim();
const title = document.querySelector("title")?.textContent?.trim();
if (title) {
const cleaned = title.split(/\s*[-|–—]\s*/)[0]?.trim();
if (cleaned) return cleaned;
}
const h1 = document.querySelector("h1")?.textContent?.trim();
return h1 || null;
}
function extractTextFromHtml(html: string): string {
const { document } = parseHTML(`<!doctype html><html><body>${html}</body></html>`);
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
for (const el of document.querySelectorAll(selector)) {
el.remove();
}
}
return document.body?.textContent?.replace(/\s+/g, " ").trim() ?? "";
}
function parseDocument(html: string): any {
const normalized = /<\s*html[\s>]/i.test(html)
? html
: `<!doctype html><html><body>${html}</body></html>`;
return parseHTML(normalized).document;
}
function sanitizeHtml(html: string): string {
const { document } = parseHTML(`<div id="__root">${html}</div>`);
const root = document.querySelector("#__root");
if (!root) return html;
for (const selector of ["script", "style", "iframe", "noscript", "template", "svg", "path"]) {
for (const el of root.querySelectorAll(selector)) {
el.remove();
}
}
return root.innerHTML;
}
function flattenJsonLdItems(data: unknown): AnyRecord[] {
if (!data || typeof data !== "object") return [];
if (Array.isArray(data)) return data.flatMap(flattenJsonLdItems);
const item = data as AnyRecord;
if (Array.isArray(item["@graph"])) {
return (item["@graph"] as unknown[]).flatMap(flattenJsonLdItems);
}
return [item];
}
function parseJsonLdScripts(document: any): AnyRecord[] {
const results: AnyRecord[] = [];
const scripts = document.querySelectorAll("script[type='application/ld+json']");
for (const script of scripts) {
try {
const data = JSON.parse(script.textContent ?? "");
results.push(...flattenJsonLdItems(data));
} catch {
// ignore malformed blocks
}
}
return results;
}
function isArticleType(item: AnyRecord): boolean {
const value = Array.isArray(item["@type"]) ? item["@type"][0] : item["@type"];
return typeof value === "string" && ARTICLE_TYPES.has(value);
}
function extractAuthorFromJsonLd(authorData: unknown): string | null {
if (typeof authorData === "string") return authorData;
if (!authorData || typeof authorData !== "object") return null;
if (Array.isArray(authorData)) {
const names = authorData
.map((author) => extractAuthorFromJsonLd(author))
.filter((name): name is string => Boolean(name));
return names.length > 0 ? names.join(", ") : null;
}
const author = authorData as AnyRecord;
return typeof author.name === "string" ? author.name : null;
}
function parseJsonLdItem(item: AnyRecord): ExtractionCandidate | null {
if (!isArticleType(item)) return null;
const rawContent =
(typeof item.articleBody === "string" && item.articleBody) ||
(typeof item.text === "string" && item.text) ||
(typeof item.description === "string" && item.description) ||
null;
if (!rawContent) return null;
const content = rawContent.trim();
const htmlLike = /<\/?[a-z][\s\S]*>/i.test(content);
const textContent = htmlLike ? extractTextFromHtml(content) : content;
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(item.headline, item.name),
byline: extractAuthorFromJsonLd(item.author),
excerpt: pickString(item.description),
published: pickString(item.datePublished, item.dateCreated),
html: htmlLike ? content : null,
textContent,
method: "json-ld",
const metadata: PageMetadata = {
url,
title: result.title || "",
description: result.description || undefined,
author: result.author || undefined,
published: result.published || undefined,
coverImage: result.image || undefined,
captured_at: new Date().toISOString(),
};
}
function tryJsonLdExtraction(document: any): ExtractionCandidate | null {
for (const item of parseJsonLdScripts(document)) {
const extracted = parseJsonLdItem(item);
if (extracted) return extracted;
}
return null;
}
function getByPath(value: unknown, path: string): unknown {
let current = value;
for (const part of path.split(".")) {
if (!current || typeof current !== "object") return undefined;
current = (current as AnyRecord)[part];
}
return current;
}
function isContentBlockArray(value: unknown): value is AnyRecord[] {
if (!Array.isArray(value) || value.length === 0) return false;
return value.slice(0, 5).some((item) => {
if (!item || typeof item !== "object") return false;
const obj = item as AnyRecord;
return "type" in obj || "text" in obj || "textHtml" in obj || "content" in obj;
});
}
function extractTextFromContentBlocks(blocks: AnyRecord[]): string {
const parts: string[] = [];
function pushParagraph(text: string): void {
const trimmed = text.trim();
if (!trimmed) return;
parts.push(trimmed, "\n\n");
}
function walk(node: unknown): void {
if (!node || typeof node !== "object") return;
const block = node as AnyRecord;
if (typeof block.text === "string") {
pushParagraph(block.text);
return;
}
if (typeof block.textHtml === "string") {
pushParagraph(extractTextFromHtml(block.textHtml));
return;
}
if (Array.isArray(block.items)) {
for (const item of block.items) {
if (item && typeof item === "object") {
const text = pickString((item as AnyRecord).text);
if (text) parts.push(`- ${text}\n`);
}
}
parts.push("\n");
}
if (Array.isArray(block.components)) {
for (const component of block.components) {
walk(component);
}
}
if (Array.isArray(block.content)) {
for (const child of block.content) {
walk(child);
}
}
}
for (const block of blocks) {
walk(block);
}
return parts.join("").replace(/\n{3,}/g, "\n\n").trim();
}
function tryStringBodyExtraction(
content: string,
meta: AnyRecord,
document: any,
method: string
): ExtractionCandidate | null {
if (!content || content.length < MIN_CONTENT_LENGTH) return null;
const isHtml = /<\/?[a-z][\s\S]*>/i.test(content);
const html = isHtml ? sanitizeHtml(content) : null;
const textContent = isHtml ? extractTextFromHtml(html) : content.trim();
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(meta.headline, meta.title, extractTitle(document)),
byline: pickString(meta.byline, meta.author),
excerpt: pickString(meta.description, meta.excerpt, generateExcerpt(null, textContent)),
published: pickString(meta.datePublished, meta.publishedAt, extractPublishedTime(document)),
html,
textContent,
method,
};
}
function tryNextDataExtraction(document: any): ExtractionCandidate | null {
try {
const script = document.querySelector("script#__NEXT_DATA__");
if (!script?.textContent) return null;
const data = JSON.parse(script.textContent) as AnyRecord;
const pageProps = (getByPath(data, "props.pageProps") ?? {}) as AnyRecord;
for (const path of NEXT_DATA_CONTENT_PATHS) {
const value = getByPath(data, path);
if (typeof value === "string") {
const parentPath = path.split(".").slice(0, -1).join(".");
const parent = (getByPath(data, parentPath) ?? {}) as AnyRecord;
const meta = {
...pageProps,
...parent,
title: parent.title ?? (pageProps.title as string | undefined),
};
const candidate = tryStringBodyExtraction(value, meta, document, "next-data");
if (candidate) return candidate;
}
if (isContentBlockArray(value)) {
const textContent = extractTextFromContentBlocks(value);
if (textContent.length < MIN_CONTENT_LENGTH) continue;
return {
title: pickString(
getByPath(data, "props.pageProps.content.headline"),
getByPath(data, "props.pageProps.article.headline"),
getByPath(data, "props.pageProps.article.title"),
getByPath(data, "props.pageProps.post.title"),
pageProps.title,
extractTitle(document)
),
byline: pickString(
getByPath(data, "props.pageProps.author.name"),
getByPath(data, "props.pageProps.article.author.name")
),
excerpt: pickString(
getByPath(data, "props.pageProps.content.description"),
getByPath(data, "props.pageProps.article.description"),
pageProps.description,
generateExcerpt(null, textContent)
),
published: pickString(
getByPath(data, "props.pageProps.content.datePublished"),
getByPath(data, "props.pageProps.article.datePublished"),
getByPath(data, "props.pageProps.publishedAt"),
extractPublishedTime(document)
),
html: null,
textContent,
method: "next-data",
};
}
}
} catch {
return null;
}
return null;
}
function buildReadabilityCandidate(
article: ReturnType<Readability["parse"]>,
document: any,
method: string
): ExtractionCandidate | null {
const textContent = article?.textContent?.trim() ?? "";
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(article?.title, extractTitle(document)),
byline: pickString(article?.byline),
excerpt: pickString(article?.excerpt, generateExcerpt(null, textContent)),
published: pickString(article?.publishedTime, extractPublishedTime(document)),
html: article?.content ? sanitizeHtml(article.content) : null,
textContent,
method,
};
}
function tryReadability(document: any): ExtractionCandidate | null {
try {
const strictClone = document.cloneNode(true) as Document;
const strictResult = buildReadabilityCandidate(
new Readability(strictClone).parse(),
document,
"readability"
);
if (strictResult) return strictResult;
const relaxedClone = document.cloneNode(true) as Document;
return buildReadabilityCandidate(
new Readability(relaxedClone, { charThreshold: 120 }).parse(),
document,
"readability-relaxed"
);
} catch {
return null;
}
}
function trySelectorExtraction(document: any): ExtractionCandidate | null {
for (const selector of CONTENT_SELECTORS) {
const element = document.querySelector(selector);
if (!element) continue;
const clone = element.cloneNode(true) as Element;
for (const removeSelector of REMOVE_SELECTORS) {
for (const node of clone.querySelectorAll(removeSelector)) {
node.remove();
}
}
const html = sanitizeHtml(clone.innerHTML);
const textContent = extractTextFromHtml(html);
if (textContent.length < MIN_CONTENT_LENGTH) continue;
return {
title: extractTitle(document),
byline: null,
excerpt: generateExcerpt(null, textContent),
published: extractPublishedTime(document),
html,
textContent,
method: `selector:${selector}`,
};
}
return null;
}
function tryBodyExtraction(document: any): ExtractionCandidate | null {
const body = document.body;
if (!body) return null;
const clone = body.cloneNode(true) as Element;
for (const removeSelector of REMOVE_SELECTORS) {
for (const node of clone.querySelectorAll(removeSelector)) {
node.remove();
}
}
const html = sanitizeHtml(clone.innerHTML);
const textContent = extractTextFromHtml(html);
if (!textContent) return null;
return {
title: extractTitle(document),
byline: null,
excerpt: generateExcerpt(null, textContent),
published: extractPublishedTime(document),
html,
textContent,
method: "body-fallback",
};
}
function pickBestCandidate(candidates: ExtractionCandidate[]): ExtractionCandidate | null {
if (candidates.length === 0) return null;
const methodOrder = [
"readability",
"readability-relaxed",
"next-data",
"json-ld",
"selector:",
"body-fallback",
];
function methodRank(method: string): number {
const idx = methodOrder.findIndex((entry) =>
entry.endsWith(":") ? method.startsWith(entry) : method === entry
);
return idx === -1 ? methodOrder.length : idx;
}
const ranked = [...candidates].sort((a, b) => {
const rankA = methodRank(a.method);
const rankB = methodRank(b.method);
if (rankA !== rankB) return rankA - rankB;
return (b.textContent.length ?? 0) - (a.textContent.length ?? 0);
});
for (const candidate of ranked) {
if (candidate.textContent.length >= GOOD_CONTENT_LENGTH) {
return candidate;
}
}
for (const candidate of ranked) {
if (candidate.textContent.length >= MIN_CONTENT_LENGTH) {
return candidate;
}
}
return ranked[0];
}
function extractFromHtml(html: string): ExtractionCandidate | null {
const document = parseDocument(html);
const readabilityCandidate = tryReadability(document);
const nextDataCandidate = tryNextDataExtraction(document);
const jsonLdCandidate = tryJsonLdExtraction(document);
const selectorCandidate = trySelectorExtraction(document);
const bodyCandidate = tryBodyExtraction(document);
const candidates = [
readabilityCandidate,
nextDataCandidate,
jsonLdCandidate,
selectorCandidate,
bodyCandidate,
].filter((candidate): candidate is ExtractionCandidate => Boolean(candidate));
const winner = pickBestCandidate(candidates);
if (!winner) return null;
return {
...winner,
title: winner.title ?? extractTitle(document),
published: winner.published ?? extractPublishedTime(document),
excerpt: winner.excerpt ?? generateExcerpt(null, winner.textContent),
};
}
const turndown = new TurndownService({
headingStyle: "atx",
hr: "---",
bulletListMarker: "-",
codeBlockStyle: "fenced",
emDelimiter: "*",
strongDelimiter: "**",
linkStyle: "inlined",
});
turndown.use(gfm);
turndown.remove(["script", "style", "iframe", "noscript", "template", "svg", "path"]);
turndown.addRule("collapseFigure", {
filter: "figure",
replacement(content) {
return `\n\n${content.trim()}\n\n`;
},
});
turndown.addRule("dropInvisibleAnchors", {
filter(node) {
return node.nodeName === "A" && !(node as Element).textContent?.trim();
},
replacement() {
return "";
},
});
function normalizeMarkdown(markdown: string): string {
return markdown
.replace(/\r\n/g, "\n")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
}
function convertHtmlToMarkdown(html: string): string {
if (!html || !html.trim()) return "";
try {
const sanitized = sanitizeHtml(html);
return turndown.turndown(sanitized);
} catch {
return "";
}
}
function fallbackPlainText(html: string): string {
const document = parseDocument(html);
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
for (const el of document.querySelectorAll(selector)) {
el.remove();
}
}
const text = document.body?.textContent ?? document.documentElement?.textContent ?? "";
return normalizeMarkdown(text.replace(/\s+/g, " "));
}
export function htmlToMarkdown(html: string): string {
if (!html || !html.trim()) return "";
const extracted = extractFromHtml(html);
if (!extracted) {
return fallbackPlainText(html);
}
let markdown = extracted.html ? convertHtmlToMarkdown(extracted.html) : "";
if (!markdown.trim()) {
markdown = extracted.textContent;
}
return normalizeMarkdown(markdown);
return { metadata, markdown: result.content || "" };
}
function escapeYamlValue(value: string): string {
@@ -836,6 +81,7 @@ export function formatMetadataYaml(meta: PageMetadata): string {
if (meta.description) lines.push(`description: "${escapeYamlValue(meta.description)}"`);
if (meta.author) lines.push(`author: "${escapeYamlValue(meta.author)}"`);
if (meta.published) lines.push(`published: "${escapeYamlValue(meta.published)}"`);
if (meta.coverImage) lines.push(`coverImage: "${escapeYamlValue(meta.coverImage)}"`);
lines.push(`captured_at: "${escapeYamlValue(meta.captured_at)}"`);
lines.push("---");
return lines.join("\n");
+41 -19
View File
@@ -4,7 +4,8 @@ import path from "node:path";
import process from "node:process";
import { CdpConnection, getFreePort, launchChrome, waitForChromeDebugPort, waitForNetworkIdle, waitForPageLoad, autoScroll, evaluateScript, killChrome } from "./cdp.js";
import { cleanupAndExtractScript, htmlToMarkdown, createMarkdownDocument, type PageMetadata, type ConversionResult } from "./html-to-markdown.js";
import { absolutizeUrlsScript, extractContent, createMarkdownDocument, type ConversionResult } from "./html-to-markdown.js";
import { localizeMarkdownMedia, countRemoteMedia } from "./media-localizer.js";
import { resolveUrlToMarkdownDataDir } from "./paths.js";
import { DEFAULT_TIMEOUT_MS, CDP_CONNECT_TIMEOUT_MS, NETWORK_IDLE_TIMEOUT_MS, POST_LOAD_DELAY_MS, SCROLL_STEP_WAIT_MS, SCROLL_MAX_STEPS } from "./constants.js";
@@ -24,12 +25,14 @@ async function fileExists(filePath: string): Promise<boolean> {
interface Args {
url: string;
output?: string;
outputDir?: string;
wait: boolean;
timeout: number;
downloadMedia: boolean;
}
function parseArgs(argv: string[]): Args {
const args: Args = { url: "", wait: false, timeout: DEFAULT_TIMEOUT_MS };
const args: Args = { url: "", wait: false, timeout: DEFAULT_TIMEOUT_MS, downloadMedia: false };
for (let i = 2; i < argv.length; i++) {
const arg = argv[i];
if (arg === "--wait" || arg === "-w") {
@@ -38,6 +41,10 @@ function parseArgs(argv: string[]): Args {
args.output = argv[++i];
} else if (arg === "--timeout" || arg === "-t") {
args.timeout = parseInt(argv[++i], 10) || DEFAULT_TIMEOUT_MS;
} else if (arg === "--output-dir") {
args.outputDir = argv[++i];
} else if (arg === "--download-media") {
args.downloadMedia = true;
} else if (!arg.startsWith("-") && !args.url) {
args.url = arg;
}
@@ -62,10 +69,10 @@ function formatTimestamp(): string {
return `${now.getFullYear()}${pad(now.getMonth() + 1)}${pad(now.getDate())}-${pad(now.getHours())}${pad(now.getMinutes())}${pad(now.getSeconds())}`;
}
async function generateOutputPath(url: string, title: string): Promise<string> {
async function generateOutputPath(url: string, title: string, outputDir?: string): Promise<string> {
const domain = new URL(url).hostname.replace(/^www\./, "");
const slug = generateSlug(title, url);
const dataDir = resolveUrlToMarkdownDataDir();
const dataDir = outputDir ? path.resolve(outputDir) : resolveUrlToMarkdownDataDir();
const basePath = path.join(dataDir, domain, `${slug}.md`);
if (!(await fileExists(basePath))) {
@@ -117,21 +124,11 @@ async function captureUrl(args: Args): Promise<ConversionResult> {
}
console.log("Capturing page content...");
const extracted = await evaluateScript<{ title: string; description?: string; author?: string; published?: string; html: string }>(
cdp, sessionId, cleanupAndExtractScript, args.timeout
const { html } = await evaluateScript<{ html: string }>(
cdp, sessionId, absolutizeUrlsScript, args.timeout
);
const metadata: PageMetadata = {
url: args.url,
title: extracted.title || "",
description: extracted.description,
author: extracted.author,
published: extracted.published,
captured_at: new Date().toISOString()
};
const markdown = htmlToMarkdown(extracted.html);
return { metadata, markdown };
return await extractContent(html, args.url);
} finally {
if (cdp) {
try { await cdp.send("Browser.close", {}, { timeoutMs: 5_000 }); } catch {}
@@ -155,15 +152,40 @@ async function main(): Promise<void> {
process.exit(1);
}
if (args.output) {
const stat = await import("node:fs").then(fs => fs.statSync(args.output!, { throwIfNoEntry: false }));
if (stat?.isDirectory()) {
console.error(`Error: -o path is a directory, not a file: ${args.output}`);
process.exit(1);
}
}
console.log(`Fetching: ${args.url}`);
console.log(`Mode: ${args.wait ? "wait" : "auto"}`);
const result = await captureUrl(args);
const outputPath = args.output || await generateOutputPath(args.url, result.metadata.title);
const outputPath = args.output || await generateOutputPath(args.url, result.metadata.title, args.outputDir);
const outputDir = path.dirname(outputPath);
await mkdir(outputDir, { recursive: true });
const document = createMarkdownDocument(result);
let document = createMarkdownDocument(result);
if (args.downloadMedia) {
const mediaResult = await localizeMarkdownMedia(document, {
markdownPath: outputPath,
log: console.log,
});
document = mediaResult.markdown;
if (mediaResult.downloadedImages > 0 || mediaResult.downloadedVideos > 0) {
console.log(`Downloaded: ${mediaResult.downloadedImages} images, ${mediaResult.downloadedVideos} videos`);
}
} else {
const { images, videos } = countRemoteMedia(document);
if (images > 0 || videos > 0) {
console.log(`Remote media found: ${images} images, ${videos} videos`);
}
}
await writeFile(outputPath, document, "utf-8");
console.log(`Saved: ${outputPath}`);
@@ -0,0 +1,317 @@
import path from "node:path";
import { mkdir, writeFile } from "node:fs/promises";
type MediaKind = "image" | "video";
type MediaHint = "image" | "unknown";
type MarkdownLinkCandidate = {
url: string;
hint: MediaHint;
};
export type LocalizeMarkdownMediaOptions = {
markdownPath: string;
log?: (message: string) => void;
};
export type LocalizeMarkdownMediaResult = {
markdown: string;
downloadedImages: number;
downloadedVideos: number;
imageDir: string | null;
videoDir: string | null;
};
const MARKDOWN_LINK_RE = /(!?\[[^\]\n]*\])\((<)?(https?:\/\/[^)\s>]+)(>)?\)/g;
const FRONTMATTER_COVER_RE = /^(coverImage:\s*")(https?:\/\/[^"]+)(")/m;
const IMAGE_EXTENSIONS = new Set([
"jpg",
"jpeg",
"png",
"webp",
"gif",
"bmp",
"avif",
"heic",
"heif",
"svg",
]);
const VIDEO_EXTENSIONS = new Set(["mp4", "m4v", "mov", "webm", "mkv"]);
const MIME_EXTENSION_MAP: Record<string, string> = {
"image/jpeg": "jpg",
"image/jpg": "jpg",
"image/png": "png",
"image/webp": "webp",
"image/gif": "gif",
"image/bmp": "bmp",
"image/avif": "avif",
"image/heic": "heic",
"image/heif": "heif",
"image/svg+xml": "svg",
"video/mp4": "mp4",
"video/webm": "webm",
"video/quicktime": "mov",
"video/x-m4v": "m4v",
};
const DOWNLOAD_USER_AGENT =
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36";
function normalizeContentType(raw: string | null): string {
return raw?.split(";")[0]?.trim().toLowerCase() ?? "";
}
function normalizeExtension(raw: string | undefined | null): string | undefined {
if (!raw) return undefined;
const trimmed = raw.replace(/^\./, "").trim().toLowerCase();
if (!trimmed) return undefined;
if (trimmed === "jpeg") return "jpg";
if (trimmed === "jpg") return "jpg";
return trimmed;
}
function resolveExtensionFromUrl(rawUrl: string): string | undefined {
try {
const parsed = new URL(rawUrl);
const extFromPath = normalizeExtension(path.posix.extname(parsed.pathname));
if (extFromPath) return extFromPath;
const extFromFormat = normalizeExtension(parsed.searchParams.get("format"));
if (extFromFormat) return extFromFormat;
} catch {
return undefined;
}
return undefined;
}
function resolveKindFromContentType(contentType: string): MediaKind | undefined {
if (!contentType) return undefined;
if (contentType.startsWith("image/")) return "image";
if (contentType.startsWith("video/")) return "video";
return undefined;
}
function resolveKindFromExtension(ext: string | undefined): MediaKind | undefined {
if (!ext) return undefined;
if (IMAGE_EXTENSIONS.has(ext)) return "image";
if (VIDEO_EXTENSIONS.has(ext)) return "video";
return undefined;
}
function resolveMediaKind(
rawUrl: string,
contentType: string,
extension: string | undefined,
hint: MediaHint
): MediaKind | undefined {
const kindFromType = resolveKindFromContentType(contentType);
if (kindFromType) return kindFromType;
const kindFromExtension = resolveKindFromExtension(extension);
if (kindFromExtension) return kindFromExtension;
if (contentType && contentType !== "application/octet-stream") {
return undefined;
}
return hint === "image" ? "image" : undefined;
}
function resolveOutputExtension(
contentType: string,
extension: string | undefined,
kind: MediaKind
): string {
const extFromMime = normalizeExtension(MIME_EXTENSION_MAP[contentType]);
if (extFromMime) return extFromMime;
const normalizedExt = normalizeExtension(extension);
if (normalizedExt) return normalizedExt;
return kind === "video" ? "mp4" : "jpg";
}
function safeDecodeURIComponent(value: string): string {
try {
return decodeURIComponent(value);
} catch {
return value;
}
}
function sanitizeFileSegment(input: string): string {
return input
.replace(/[^a-zA-Z0-9_-]+/g, "-")
.replace(/-+/g, "-")
.replace(/^[-_]+|[-_]+$/g, "")
.slice(0, 48);
}
function resolveFileStem(rawUrl: string, extension: string): string {
try {
const parsed = new URL(rawUrl);
const base = path.posix.basename(parsed.pathname);
if (!base) return "";
const decodedBase = safeDecodeURIComponent(base);
const normalizedExt = normalizeExtension(extension);
const stripExt = normalizedExt ? new RegExp(`\\.${normalizedExt}$`, "i") : null;
const rawStem = stripExt ? decodedBase.replace(stripExt, "") : decodedBase;
return sanitizeFileSegment(rawStem);
} catch {
return "";
}
}
function buildFileName(kind: MediaKind, index: number, sourceUrl: string, extension: string): string {
const stem = resolveFileStem(sourceUrl, extension);
const prefix = kind === "image" ? "img" : "video";
const serial = String(index).padStart(3, "0");
const suffix = stem ? `-${stem}` : "";
return `${prefix}-${serial}${suffix}.${extension}`;
}
function collectMarkdownLinkCandidates(markdown: string): MarkdownLinkCandidate[] {
const candidates: MarkdownLinkCandidate[] = [];
const seen = new Set<string>();
const fmMatch = markdown.match(/^---\n([\s\S]*?)\n---/);
if (fmMatch) {
const coverMatch = fmMatch[1]?.match(FRONTMATTER_COVER_RE);
if (coverMatch?.[2] && !seen.has(coverMatch[2])) {
seen.add(coverMatch[2]);
candidates.push({ url: coverMatch[2], hint: "image" });
}
}
MARKDOWN_LINK_RE.lastIndex = 0;
let match: RegExpExecArray | null;
while ((match = MARKDOWN_LINK_RE.exec(markdown))) {
const label = match[1] ?? "";
const rawUrl = match[3] ?? "";
if (!rawUrl || seen.has(rawUrl)) continue;
seen.add(rawUrl);
candidates.push({
url: rawUrl,
hint: label.startsWith("![") ? "image" : "unknown",
});
}
return candidates;
}
function rewriteMarkdownMediaLinks(markdown: string, replacements: Map<string, string>): string {
if (replacements.size === 0) return markdown;
MARKDOWN_LINK_RE.lastIndex = 0;
let result = markdown.replace(MARKDOWN_LINK_RE, (full, label, _openAngle, rawUrl) => {
const localPath = replacements.get(rawUrl);
if (!localPath) return full;
return `${label}(${localPath})`;
});
result = result.replace(FRONTMATTER_COVER_RE, (full, prefix, rawUrl, suffix) => {
const localPath = replacements.get(rawUrl);
if (!localPath) return full;
return `${prefix}${localPath}${suffix}`;
});
return result;
}
export async function localizeMarkdownMedia(
markdown: string,
options: LocalizeMarkdownMediaOptions
): Promise<LocalizeMarkdownMediaResult> {
const log = options.log ?? (() => {});
const markdownDir = path.dirname(options.markdownPath);
const candidates = collectMarkdownLinkCandidates(markdown);
if (candidates.length === 0) {
return {
markdown,
downloadedImages: 0,
downloadedVideos: 0,
imageDir: null,
videoDir: null,
};
}
const replacements = new Map<string, string>();
let downloadedImages = 0;
let downloadedVideos = 0;
for (const candidate of candidates) {
try {
const response = await fetch(candidate.url, {
method: "GET",
redirect: "follow",
headers: {
"user-agent": DOWNLOAD_USER_AGENT,
},
});
if (!response.ok) {
log(`[url-to-markdown] Skip media (${response.status}): ${candidate.url}`);
continue;
}
const sourceUrl = response.url || candidate.url;
const contentType = normalizeContentType(response.headers.get("content-type"));
const extension = resolveExtensionFromUrl(sourceUrl) ?? resolveExtensionFromUrl(candidate.url);
const kind = resolveMediaKind(sourceUrl, contentType, extension, candidate.hint);
if (!kind) {
continue;
}
const outputExtension = resolveOutputExtension(contentType, extension, kind);
const nextIndex = kind === "image" ? downloadedImages + 1 : downloadedVideos + 1;
const dirName = kind === "image" ? "imgs" : "videos";
const targetDir = path.join(markdownDir, dirName);
await mkdir(targetDir, { recursive: true });
const fileName = buildFileName(kind, nextIndex, sourceUrl, outputExtension);
const absolutePath = path.join(targetDir, fileName);
const relativePath = path.posix.join(dirName, fileName);
const bytes = Buffer.from(await response.arrayBuffer());
await writeFile(absolutePath, bytes);
replacements.set(candidate.url, relativePath);
if (kind === "image") {
downloadedImages = nextIndex;
} else {
downloadedVideos = nextIndex;
}
} catch (error) {
const message = error instanceof Error ? error.message : String(error ?? "");
log(`[url-to-markdown] Failed to download media ${candidate.url}: ${message}`);
}
}
return {
markdown: rewriteMarkdownMediaLinks(markdown, replacements),
downloadedImages,
downloadedVideos,
imageDir: downloadedImages > 0 ? path.join(markdownDir, "imgs") : null,
videoDir: downloadedVideos > 0 ? path.join(markdownDir, "videos") : null,
};
}
export function countRemoteMedia(markdown: string): { images: number; videos: number; hasCoverImage: boolean } {
const fmMatch = markdown.match(/^---\n([\s\S]*?)\n---/);
const hasCoverImage = !!(fmMatch?.[1]?.match(FRONTMATTER_COVER_RE)?.[2]);
const candidates = collectMarkdownLinkCandidates(markdown);
let images = 0;
let videos = 0;
for (const c of candidates) {
const ext = resolveExtensionFromUrl(c.url);
const kind = resolveKindFromExtension(ext);
if (kind === "video") {
videos++;
} else if (kind === "image" || c.hint === "image") {
images++;
}
}
return { images, videos, hasCoverImage };
}
@@ -167,7 +167,7 @@ When generating multiple images in a series:
1. **Image 1 (cover)**: Generate without `--ref` — this establishes the visual anchor
2. **Images 2+**: Always pass image 1 as `--ref` to the image generation skill:
```bash
npx -y bun ${SKILL_DIR}/scripts/main.ts \
${BUN_X} ${SKILL_DIR}/scripts/main.ts \
--promptfiles prompts/02-content-xxx.md \
--ref path/to/01-cover-xxx.png \
--image 02-content-xxx.png --ar 3:4 --quality 2k