Compare commits

..

38 Commits

Author SHA1 Message Date
Jim Liu 宝玉 dad8f3a800 chore: release v1.79.2 2026-03-23 22:39:14 -05:00
Jim Liu 宝玉 35298d7c9d fix(baoyu-post-to-weibo): add no-theme rule for article markdown-to-HTML conversion 2026-03-23 22:38:30 -05:00
Jim Liu 宝玉 f22374ab62 fix(baoyu-cover-image): simplify reference image handling based on model capability 2026-03-23 22:38:27 -05:00
Jim Liu 宝玉 d4e80b1bc3 Fix Node-compatible parser tests (#107)
* Fix Node-compatible parser tests

* Add parser test dependencies to root test env
2026-03-23 15:30:42 -05:00
Jim Liu 宝玉 a5761dc71a chore: release v1.79.1 2026-03-23 12:02:44 -05:00
Jim Liu 宝玉 a5189dff37 fix(baoyu-xhs-images): remove opacity from watermark prompt and fix CJK spacing 2026-03-23 12:01:03 -05:00
Jim Liu 宝玉 39fe872bf3 fix(baoyu-comic): fix Doraemon naming spacing and remove opacity from watermark prompt 2026-03-23 12:01:00 -05:00
Jim Liu 宝玉 52813504f8 fix(baoyu-article-illustrator): remove opacity parameter from watermark prompt 2026-03-23 12:00:53 -05:00
Jim Liu 宝玉 a4d4108cd1 docs(project): update documentation to reflect single-plugin architecture 2026-03-23 12:00:38 -05:00
Yizhou Qian 钱亦舟 d7e763f1f5 fix: consolidate to single plugin to prevent duplicate skill registration (#106)
Merge the three plugins (content-skills, ai-generation-skills,
utility-skills) into one plugin entry. Since all three shared the same
source ("./"), Claude Code cached every skill three times. A single
plugin with one source keeps the flat skills/ layout while ensuring
each skill is registered exactly once.
2026-03-23 09:46:50 -05:00
Jim Liu 宝玉 097c09c59b chore: release v1.79.0 2026-03-22 15:42:33 -05:00
Jim Liu 宝玉 e4cd8bfefc feat(baoyu-post-to-wechat): improve credential loading with multi-source resolution and diagnostics 2026-03-22 15:42:08 -05:00
Jim Liu 宝玉 3dc5f2e06f chore: release v1.78.0 2026-03-22 15:19:29 -05:00
Jim Liu 宝玉 e5d6c8ec68 feat(baoyu-url-to-markdown): add URL-specific parser layer for X/Twitter and archive.ph
- New parsers/ module with pluggable rule system for site-specific HTML extraction
- X status parser: extract tweet text, media, quotes, author from data-testid elements
- X article parser: extract long-form article content with inline media
- archive.ph parser: restore original URL and prefer #CONTENT container
- Improved slug generation with stop words and content-aware slugs
- Output path uses subdirectory structure (domain/slug/slug.md)
- Fix: preserve anchor elements containing media in legacy converter
- Fix: smarter title deduplication in markdown document builder
2026-03-22 15:18:46 -05:00
Jim Liu 宝玉 6a4b312146 chore: release v1.77.0 2026-03-22 15:17:12 -05:00
Jim Liu 宝玉 2d6fe533eb Merge pull request #105 from jzOcb/feat/chapter-end-times
feat(youtube-transcript): add end times to chapter data
2026-03-22 15:13:44 -05:00
jzocb f53af25e65 fix: address review feedback on chapter end times
- Guard last chapter end against duration=0: use Math.max(duration, ch.start)
- Remove unnecessary 'as any' cast in backfill
- Check all chapters for missing end (not just first) via .some()
- Skip backfill when needsFetch is true (about to refetch anyway)
- Wrap backfill writeFileSync in try/catch (best-effort persistence)
2026-03-22 16:07:05 -04:00
jzocb c7e32b4590 fix: backfill chapter end times for cached videos
Videos cached before the chapter end-time change would silently
lack the 'end' field when loaded from cache. This adds a migration
that detects missing 'end' fields on cache hit, computes them from
adjacent chapters, and persists the updated meta.json.

This ensures consistent output regardless of whether the data was
freshly fetched or loaded from cache.
2026-03-22 15:58:13 -04:00
jzocb 8d973f2bc5 feat(youtube-transcript): add end times to chapter data
Add 'end' field to Chapter interface and parseChapters output.
Each chapter's end is derived from the next chapter's start time,
with the last chapter ending at the video's total duration.

This makes chapter data complete and ready for downstream consumers
(e.g. video clipping with ffmpeg) without requiring them to compute
end times from adjacent chapters.

Before: { title: 'Overview', start: 0 }
After:  { title: 'Overview', start: 0, end: 21 }
2026-03-22 15:52:30 -04:00
Jim Liu 宝玉 ba20cf89f2 fix(sync-clawhub): skip failed skills instead of aborting 2026-03-21 23:25:45 -05:00
Jim Liu 宝玉 1827be9234 chore: release v1.76.1 2026-03-21 23:17:04 -05:00
Jim Liu 宝玉 93c98dfc3c docs(baoyu-youtube-transcript): fix zsh glob issue for YouTube URLs 2026-03-21 23:16:51 -05:00
Jim Liu 宝玉 fbd9f9b622 chore: release v1.76.0 2026-03-21 23:08:21 -05:00
Jim Liu 宝玉 b6e293d059 fix(baoyu-markdown-to-html): use process.execPath and tsx import in test runner 2026-03-21 23:07:46 -05:00
Jim Liu 宝玉 bb78aab095 feat(baoyu-youtube-transcript): add title heading, description summary, and cover image to markdown output 2026-03-21 23:07:44 -05:00
Jim Liu 宝玉 5071a1d0d0 chore: release v1.75.0 2026-03-21 22:44:00 -05:00
Jim Liu 宝玉 e413ade164 feat(baoyu-youtube-transcript): add sentence segmentation and improve caching 2026-03-21 22:42:43 -05:00
Jim Liu 宝玉 e52f92b193 new skill 2026-03-21 21:03:06 -05:00
Jim Liu 宝玉 603cabaef4 chore: release v1.74.1 2026-03-21 00:03:51 -05:00
Jim Liu 宝玉 7d12526e90 fix(baoyu-image-gen): broaden OpenRouter model detection and aspect ratio validation 2026-03-21 00:03:20 -05:00
Jim Liu 宝玉 e7f9764a49 Merge pull request #101 from cwandev/fix/openrouter
fix(baoyu-image-gen): align `OpenRouter` image generation with current API
2026-03-20 23:32:26 -05:00
Jim Liu 宝玉 e55f91b0ea chore: release v1.74.0 2026-03-20 23:17:14 -05:00
Jim Liu 宝玉 fe3b3d9125 feat(baoyu-markdown-to-html): pass through all rendering options from CLI
- CLI now supports --color, --font-family, --font-size, --code-theme, --mac-code-block, --line-number, --count, --legend
- convertMarkdown accepts full CliOptions instead of limited subset
- Dynamic help text showing available theme/color/font options
2026-03-20 23:17:10 -05:00
Jim Liu 宝玉 105339cf3f fix(baoyu-md): fix CSS custom property regex for quoted values and add theme layering
- Remove quotes from CSS custom property regex character class so values containing quotes are fully stripped
- grace/simple themes now layer default CSS before their own rules
- Add tests for quoted property stripping and theme layering
2026-03-20 23:17:04 -05:00
cwandev e43eec260a fix(baoyu-image-gen): narrow OpenRouter Gemini ratios 2026-03-19 22:09:23 +08:00
cwandev 96ef6e2251 fix(baoyu-image-gen): harden OpenRouter image support 2026-03-19 21:59:41 +08:00
cwandev efb7a1917a fix(baoyu-image-gen): require OpenRouter image parameters 2026-03-19 21:26:19 +08:00
cwandev 1af984a64f fix(baoyu-image-gen): align OpenRouter image generation with current API 2026-03-19 17:36:03 +08:00
57 changed files with 3493 additions and 253 deletions
+16 -31
View File
@@ -6,48 +6,33 @@
},
"metadata": {
"description": "Skills shared by Baoyu for improving daily work efficiency",
"version": "1.73.3"
"version": "1.79.2"
},
"plugins": [
{
"name": "content-skills",
"description": "Content generation and publishing skills",
"name": "baoyu-skills",
"description": "Content generation, AI backends, and utility tools for daily work efficiency",
"source": "./",
"strict": true,
"skills": [
"./skills/baoyu-xhs-images",
"./skills/baoyu-post-to-x",
"./skills/baoyu-post-to-wechat",
"./skills/baoyu-post-to-weibo",
"./skills/baoyu-article-illustrator",
"./skills/baoyu-cover-image",
"./skills/baoyu-slide-deck",
"./skills/baoyu-comic",
"./skills/baoyu-infographic"
]
},
{
"name": "ai-generation-skills",
"description": "AI-powered generation backends",
"source": "./",
"strict": true,
"skills": [
"./skills/baoyu-danger-gemini-web",
"./skills/baoyu-image-gen"
]
},
{
"name": "utility-skills",
"description": "Utility tools for content processing",
"source": "./",
"strict": true,
"skills": [
"./skills/baoyu-danger-x-to-markdown",
"./skills/baoyu-compress-image",
"./skills/baoyu-url-to-markdown",
"./skills/baoyu-cover-image",
"./skills/baoyu-danger-gemini-web",
"./skills/baoyu-danger-x-to-markdown",
"./skills/baoyu-format-markdown",
"./skills/baoyu-image-gen",
"./skills/baoyu-infographic",
"./skills/baoyu-markdown-to-html",
"./skills/baoyu-translate"
"./skills/baoyu-post-to-weibo",
"./skills/baoyu-post-to-wechat",
"./skills/baoyu-post-to-x",
"./skills/baoyu-slide-deck",
"./skills/baoyu-translate",
"./skills/baoyu-url-to-markdown",
"./skills/baoyu-xhs-images",
"./skills/baoyu-youtube-transcript"
]
}
]
+1
View File
@@ -166,3 +166,4 @@ posts/
.clawdhub/
.release-artifacts/
.worktrees/
youtube-transcript/
+75
View File
@@ -2,6 +2,81 @@
English | [中文](./CHANGELOG.zh.md)
## 1.79.2 - 2026-03-23
### Fixes
- `baoyu-cover-image`: simplify reference image handling — use `--ref` when model supports it, only create description files for models without reference image support
- `baoyu-post-to-weibo`: add no-theme rule for article markdown-to-HTML conversion
### Tests
- Fix Node-compatible parser tests and add parser test dependencies
## 1.79.1 - 2026-03-23
### Fixes
- Consolidate to single plugin to prevent duplicate skill registration (by @TyrealQ)
- `baoyu-article-illustrator`: remove opacity parameter from watermark prompt
- `baoyu-comic`: fix Doraemon naming spacing and remove opacity from watermark prompt
- `baoyu-xhs-images`: remove opacity from watermark prompt and fix CJK spacing
### Documentation
- Update project documentation to reflect single-plugin architecture
## 1.79.0 - 2026-03-22
### Features
- `baoyu-post-to-wechat`: improve credential loading with multi-source resolution, priority ordering, and diagnostics for skipped incomplete sources
## 1.78.0 - 2026-03-22
### Features
- `baoyu-url-to-markdown`: add URL-specific parser layer for X/Twitter and archive.ph sites
- `baoyu-url-to-markdown`: improved slug generation with stop words removal and subdirectory output structure
### Fixes
- `baoyu-url-to-markdown`: preserve anchor elements containing media in legacy converter
- `baoyu-url-to-markdown`: smarter title deduplication to avoid redundant headings
## 1.77.0 - 2026-03-22
### Features
- `baoyu-youtube-transcript`: add end times to chapter data (by @jzOcb)
### Fixes
- `sync-clawhub`: skip failed skills instead of aborting
## 1.76.1 - 2026-03-21
### Documentation
- `baoyu-youtube-transcript`: fix zsh glob issue — always single-quote YouTube URLs when running the script
## 1.76.0 - 2026-03-21
### Features
- `baoyu-youtube-transcript`: add title heading, description summary, and cover image to markdown output
### Fixes
- `baoyu-markdown-to-html`: use process.execPath and tsx import in test runner
## 1.75.0 - 2026-03-21
### Features
- `baoyu-youtube-transcript`: new skill — download YouTube video transcripts/subtitles and cover images with multi-language, chapters, and speaker identification support
## 1.74.1 - 2026-03-21
### Fixes
- `baoyu-image-gen`: align OpenRouter image generation with current API, harden image support, and narrow Gemini aspect ratios (by @cwandev)
- `baoyu-image-gen`: broaden OpenRouter model detection and aspect ratio validation
## 1.74.0 - 2026-03-20
### Features
- `baoyu-markdown-to-html`: CLI now supports all rendering options — color, font-family, font-size, code-theme, mac-code-block, line-number, count, legend
### Fixes
- `baoyu-markdown-to-html`: fix CSS custom property regex to handle quoted values; grace/simple themes now layer default CSS
## 1.73.3 - 2026-03-20
### Fixes
+75
View File
@@ -2,6 +2,81 @@
[English](./CHANGELOG.md) | 中文
## 1.79.2 - 2026-03-23
### 修复
- `baoyu-cover-image`:简化参考图片处理流程 — 模型支持 `--ref` 时直接传递,仅在模型不支持参考图时创建描述文件
- `baoyu-post-to-weibo`:文章 Markdown 转 HTML 时不传递 --theme 参数
### 测试
- 修复 Node 兼容的解析器测试,添加解析器测试依赖
## 1.79.1 - 2026-03-23
### 修复
- 合并为单一插件,防止 skill 重复注册 (by @TyrealQ)
- `baoyu-article-illustrator`:移除水印提示词中的不透明度参数
- `baoyu-comic`:修正哆啦 A 梦命名间距,移除水印不透明度参数
- `baoyu-xhs-images`:移除水印不透明度参数,修正中英文间距
### 文档
- 更新项目文档以反映单一插件架构
## 1.79.0 - 2026-03-22
### 新功能
- `baoyu-post-to-wechat`:改进凭据加载机制,支持多来源优先级解析,并提供不完整凭据来源的诊断信息
## 1.78.0 - 2026-03-22
### 新功能
- `baoyu-url-to-markdown`:新增 URL 专用解析层,支持 X/Twitter 和 archive.ph 站点的定制化 HTML 提取
- `baoyu-url-to-markdown`:改进 slug 生成算法,去除停用词并采用子目录输出结构
### 修复
- `baoyu-url-to-markdown`:旧版转换器保留包含媒体元素的锚标签
- `baoyu-url-to-markdown`:更智能的标题去重,避免重复添加标题
## 1.77.0 - 2026-03-22
### 新功能
- `baoyu-youtube-transcript`:为章节数据添加结束时间 (by @jzOcb)
### 修复
- `sync-clawhub`:跳过失败的技能而不是中止同步
## 1.76.1 - 2026-03-21
### 文档
- `baoyu-youtube-transcript`:修复 zsh glob 问题 — 运行脚本时始终对 YouTube URL 使用单引号
## 1.76.0 - 2026-03-21
### 新功能
- `baoyu-youtube-transcript`:Markdown 输出中新增标题、描述摘要和封面图片
### 修复
- `baoyu-markdown-to-html`:测试运行器改用 process.execPath 和 tsx import
## 1.75.0 - 2026-03-21
### 新功能
- `baoyu-youtube-transcript`:新技能 — 下载 YouTube 视频字幕/转录文本和封面图片,支持多语言、章节分段和说话人识别
## 1.74.1 - 2026-03-21
### 修复
- `baoyu-image-gen`:对齐 OpenRouter 图像生成与当前 API,增强图像支持,收窄 Gemini 宽高比范围 (by @cwandev)
- `baoyu-image-gen`:扩展 OpenRouter 模型检测和宽高比验证
## 1.74.0 - 2026-03-20
### 新功能
- `baoyu-markdown-to-html`:CLI 支持全部渲染选项 — color、font-family、font-size、code-theme、mac-code-block、line-number、count、legend
### 修复
- `baoyu-markdown-to-html`:修复 CSS 自定义属性正则无法处理带引号值的问题;grace/simple 主题现在会叠加 default 主题 CSS
## 1.73.3 - 2026-03-20
### 修复
+7 -7
View File
@@ -1,16 +1,16 @@
# CLAUDE.md
Claude Code marketplace plugin providing AI-powered content generation skills. Version: **1.73.3**.
Claude Code marketplace plugin providing AI-powered content generation skills. Version: **1.79.2**.
## Architecture
Skills organized into three categories in `.claude-plugin/marketplace.json` (defines plugin metadata, version, and skill paths):
Skills are exposed through the single `baoyu-skills` plugin in `.claude-plugin/marketplace.json` (which defines plugin metadata, version, and skill paths). The repo docs still group them into three logical areas:
| Category | Description |
|----------|-------------|
| `content-skills` | Generate or publish content (images, slides, comics, posts) |
| `ai-generation-skills` | AI generation backends |
| `utility-skills` | Content processing (conversion, compression, translation) |
| Group | Description |
|-------|-------------|
| Content Skills | Generate or publish content (images, slides, comics, posts) |
| AI Generation Skills | AI generation backends |
| Utility Skills | Content processing (conversion, compression, translation) |
Each skill contains `SKILL.md` (YAML front matter + docs), optional `scripts/`, `references/`, `prompts/`.
+43 -11
View File
@@ -52,16 +52,14 @@ Run the following command in Claude Code:
1. Select **Browse and install plugins**
2. Select **baoyu-skills**
3. Select the plugin(s) you want to install
3. Select the **baoyu-skills** plugin
4. Select **Install now**
**Option 2: Direct Install**
```bash
# Install specific plugin
/plugin install content-skills@baoyu-skills
/plugin install ai-generation-skills@baoyu-skills
/plugin install utility-skills@baoyu-skills
# Install the marketplace's single plugin
/plugin install baoyu-skills@baoyu-skills
```
**Option 3: Ask the Agent**
@@ -70,13 +68,13 @@ Simply tell Claude Code:
> Please install Skills from github.com/JimLiu/baoyu-skills
### Available Plugins
### Available Plugin
| Plugin | Description | Skills |
|--------|-------------|--------|
| **content-skills** | Content generation and publishing | [xhs-images](#baoyu-xhs-images), [infographic](#baoyu-infographic), [cover-image](#baoyu-cover-image), [slide-deck](#baoyu-slide-deck), [comic](#baoyu-comic), [article-illustrator](#baoyu-article-illustrator), [post-to-x](#baoyu-post-to-x), [post-to-wechat](#baoyu-post-to-wechat), [post-to-weibo](#baoyu-post-to-weibo) |
| **ai-generation-skills** | AI-powered generation backends | [image-gen](#baoyu-image-gen), [danger-gemini-web](#baoyu-danger-gemini-web) |
| **utility-skills** | Utility tools for content processing | [url-to-markdown](#baoyu-url-to-markdown), [danger-x-to-markdown](#baoyu-danger-x-to-markdown), [compress-image](#baoyu-compress-image), [format-markdown](#baoyu-format-markdown), [markdown-to-html](#baoyu-markdown-to-html), [translate](#baoyu-translate) |
The marketplace now exposes a single plugin so each skill is registered exactly once.
| Plugin | Description | Includes |
|--------|-------------|----------|
| **baoyu-skills** | Content generation, AI backends, and utility tools for daily work efficiency | All skills in this repository, organized below as Content Skills, AI Generation Skills, and Utility Skills |
## Update Skills
@@ -766,6 +764,40 @@ Interacts with Gemini Web to generate text and images.
Utility tools for content processing.
#### baoyu-youtube-transcript
Download YouTube video transcripts/subtitles and cover images. Supports multiple languages, translation, chapters, and speaker identification. Caches raw data for fast re-formatting.
```bash
# Default: markdown with timestamps
/baoyu-youtube-transcript https://www.youtube.com/watch?v=VIDEO_ID
# Specify languages (priority order)
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --languages zh,en,ja
# With chapters and speaker identification
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --chapters --speakers
# SRT subtitle format
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --format srt
# List available transcripts
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --list
```
**Options**:
| Option | Description | Default |
|--------|-------------|---------|
| `<url-or-id>` | YouTube URL or video ID | Required |
| `--languages <codes>` | Language codes, comma-separated | `en` |
| `--format <fmt>` | Output format: `text`, `srt` | `text` |
| `--translate <code>` | Translate to specified language | |
| `--chapters` | Chapter segmentation from video description | |
| `--speakers` | Speaker identification (requires AI post-processing) | |
| `--no-timestamps` | Disable timestamps | |
| `--list` | List available transcripts | |
| `--refresh` | Force re-fetch, ignore cache | |
#### baoyu-url-to-markdown
Fetch any URL via Chrome CDP and convert to clean markdown. Saves rendered HTML snapshot alongside the markdown, and automatically falls back to a legacy extractor when Defuddle fails.
+41 -9
View File
@@ -52,16 +52,14 @@ clawhub install baoyu-markdown-to-html
1. 选择 **Browse and install plugins**
2. 选择 **baoyu-skills**
3. 选择要安装的插件
3. 选择 **baoyu-skills** 插件
4. 选择 **Install now**
**方式二:直接安装**
```bash
# 安装指定插件
/plugin install content-skills@baoyu-skills
/plugin install ai-generation-skills@baoyu-skills
/plugin install utility-skills@baoyu-skills
# 安装 marketplace 中唯一的插件
/plugin install baoyu-skills@baoyu-skills
```
**方式三:告诉 Agent**
@@ -72,11 +70,11 @@ clawhub install baoyu-markdown-to-html
### 可用插件
| 插件 | 说明 | 包含技能 |
现在 marketplace 只暴露一个插件,这样每个 skill 只会注册一次。
| 插件 | 说明 | 包含内容 |
|------|------|----------|
| **content-skills** | 内容生成和发布 | [xhs-images](#baoyu-xhs-images), [infographic](#baoyu-infographic), [cover-image](#baoyu-cover-image), [slide-deck](#baoyu-slide-deck), [comic](#baoyu-comic), [article-illustrator](#baoyu-article-illustrator), [post-to-x](#baoyu-post-to-x), [post-to-wechat](#baoyu-post-to-wechat), [post-to-weibo](#baoyu-post-to-weibo) |
| **ai-generation-skills** | AI 生成后端 | [image-gen](#baoyu-image-gen), [danger-gemini-web](#baoyu-danger-gemini-web) |
| **utility-skills** | 内容处理工具 | [url-to-markdown](#baoyu-url-to-markdown), [danger-x-to-markdown](#baoyu-danger-x-to-markdown), [compress-image](#baoyu-compress-image), [format-markdown](#baoyu-format-markdown), [markdown-to-html](#baoyu-markdown-to-html), [translate](#baoyu-translate) |
| **baoyu-skills** | 提供内容生成、AI 后端和日常效率工具技能 | 仓库中的全部 skills,仍按下方的内容技能、AI 生成技能、工具技能三个分类展示 |
## 更新技能
@@ -766,6 +764,40 @@ AI 驱动的生成后端。
内容处理工具。
#### baoyu-youtube-transcript
下载 YouTube 视频字幕/转录文本和封面图片。支持多语言、翻译、章节分段和说话人识别。缓存原始数据以便快速重新格式化。
```bash
# 默认:带时间戳的 Markdown
/baoyu-youtube-transcript https://www.youtube.com/watch?v=VIDEO_ID
# 指定语言(按优先级排列)
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --languages zh,en,ja
# 章节分段 + 说话人识别
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --chapters --speakers
# SRT 字幕格式
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --format srt
# 列出可用字幕
/baoyu-youtube-transcript https://youtu.be/VIDEO_ID --list
```
**选项**
| 选项 | 说明 | 默认值 |
|------|------|--------|
| `<url-or-id>` | YouTube URL 或视频 ID | 必填 |
| `--languages <codes>` | 语言代码,逗号分隔 | `en` |
| `--format <fmt>` | 输出格式:`text``srt` | `text` |
| `--translate <code>` | 翻译为指定语言 | |
| `--chapters` | 根据视频描述进行章节分段 | |
| `--speakers` | 说话人识别(需 AI 后处理) | |
| `--no-timestamps` | 禁用时间戳 | |
| `--list` | 列出可用字幕 | |
| `--refresh` | 强制重新获取,忽略缓存 | |
#### baoyu-url-to-markdown
通过 Chrome CDP 抓取任意 URL 并转换为 Markdown。同时保存渲染后的 HTML 快照,Defuddle 失败时自动回退到旧版提取器。
+11 -9
View File
@@ -34,20 +34,22 @@ metadata:
1. Create `skills/baoyu-<name>/SKILL.md` with YAML front matter
2. Add TypeScript in `skills/baoyu-<name>/scripts/` (if applicable)
3. Add prompt templates in `skills/baoyu-<name>/prompts/` if needed
4. Register in `marketplace.json` under appropriate category
4. Register the skill in `.claude-plugin/marketplace.json` under the `baoyu-skills` plugin entry
5. Add Script Directory section to SKILL.md if skill has scripts
6. Add openclaw metadata to frontmatter
## Category Selection
## Skill Grouping
| If your skill... | Use category |
|------------------|--------------|
| Generates visual content (images, slides, comics) | `content-skills` |
| Publishes to platforms (X, WeChat, Weibo) | `content-skills` |
| Provides AI generation backend | `ai-generation-skills` |
| Converts or processes content | `utility-skills` |
All skills are registered under the single `baoyu-skills` plugin. Use these logical groups when deciding where the skill should appear in the docs:
New category: add plugin object to `marketplace.json` with `name`, `description`, `skills[]`.
| If your skill... | Use group |
|------------------|-----------|
| Generates visual content (images, slides, comics) | Content Skills |
| Publishes to platforms (X, WeChat, Weibo) | Content Skills |
| Provides AI generation backend | AI Generation Skills |
| Converts or processes content | Utility Skills |
If you add a new logical group, update the docs that present grouped skills, but keep the skill registered under the single `baoyu-skills` plugin entry.
## Writing Descriptions
+105 -1
View File
@@ -9,7 +9,11 @@
"packages/*"
],
"devDependencies": {
"tsx": "^4.20.5"
"@mozilla/readability": "^0.6.0",
"linkedom": "^0.18.12",
"tsx": "^4.20.5",
"turndown": "^7.2.2",
"turndown-plugin-gfm": "^1.0.2"
}
},
"node_modules/@esbuild/aix-ppc64": {
@@ -454,6 +458,23 @@
"node": ">=18"
}
},
"node_modules/@mixmark-io/domino": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/@mixmark-io/domino/-/domino-2.2.0.tgz",
"integrity": "sha512-Y28PR25bHXUg88kCV7nivXrP2Nj2RueZ3/l/jdx6J9f8J4nsEGcgX0Qe6lt7Pa+J79+kPiJU3LguR6O/6zrLOw==",
"dev": true,
"license": "BSD-2-Clause"
},
"node_modules/@mozilla/readability": {
"version": "0.6.0",
"resolved": "https://registry.npmjs.org/@mozilla/readability/-/readability-0.6.0.tgz",
"integrity": "sha512-juG5VWh4qAivzTAeMzvY9xs9HY5rAcr2E4I7tiSSCokRFi7XIZCAu92ZkSTsIj1OPceCifL3cpfteP3pDT9/QQ==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/@types/debug": {
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.12.tgz",
@@ -615,6 +636,13 @@
"url": "https://github.com/sponsors/fb55"
}
},
"node_modules/cssom": {
"version": "0.5.0",
"resolved": "https://registry.npmjs.org/cssom/-/cssom-0.5.0.tgz",
"integrity": "sha512-iKuQcq+NdHqlAcwUY0o/HL69XQrUaQdMjmStJ8JFmUaiiQErlhrmuigkg/CU4E2J0IyUKUrMAgl36TvN67MqTw==",
"dev": true,
"license": "MIT"
},
"node_modules/debug": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
@@ -896,6 +924,13 @@
"node": ">=12.0.0"
}
},
"node_modules/html-escaper": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-3.0.3.tgz",
"integrity": "sha512-RuMffC89BOWQoY0WKGpIhn5gX3iI54O6nRA0yC124NYVtzjmFWBIiFd8M0x+ZdX0P9R4lADg1mgP8C7PxGOWuQ==",
"dev": true,
"license": "MIT"
},
"node_modules/htmlparser2": {
"version": "9.1.0",
"resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-9.1.0.tgz",
@@ -984,6 +1019,51 @@
"node": ">=18.17"
}
},
"node_modules/linkedom": {
"version": "0.18.12",
"resolved": "https://registry.npmjs.org/linkedom/-/linkedom-0.18.12.tgz",
"integrity": "sha512-jalJsOwIKuQJSeTvsgzPe9iJzyfVaEJiEXl+25EkKevsULHvMJzpNqwvj1jOESWdmgKDiXObyjOYwlUqG7wo1Q==",
"dev": true,
"license": "ISC",
"dependencies": {
"css-select": "^5.1.0",
"cssom": "^0.5.0",
"html-escaper": "^3.0.3",
"htmlparser2": "^10.0.0",
"uhyphen": "^0.2.0"
},
"engines": {
"node": ">=16"
},
"peerDependencies": {
"canvas": ">= 2"
},
"peerDependenciesMeta": {
"canvas": {
"optional": true
}
}
},
"node_modules/linkedom/node_modules/htmlparser2": {
"version": "10.1.0",
"resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.1.0.tgz",
"integrity": "sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ==",
"dev": true,
"funding": [
"https://github.com/fb55/htmlparser2?sponsor=1",
{
"type": "github",
"url": "https://github.com/sponsors/fb55"
}
],
"license": "MIT",
"dependencies": {
"domelementtype": "^2.3.0",
"domhandler": "^5.0.3",
"domutils": "^3.2.2",
"entities": "^7.0.1"
}
},
"node_modules/longest-streak": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/longest-streak/-/longest-streak-3.1.0.tgz",
@@ -1768,6 +1848,30 @@
"fsevents": "~2.3.3"
}
},
"node_modules/turndown": {
"version": "7.2.2",
"resolved": "https://registry.npmjs.org/turndown/-/turndown-7.2.2.tgz",
"integrity": "sha512-1F7db8BiExOKxjSMU2b7if62D/XOyQyZbPKq/nUwopfgnHlqXHqQ0lvfUTeUIr1lZJzOPFn43dODyMSIfvWRKQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@mixmark-io/domino": "^2.2.0"
}
},
"node_modules/turndown-plugin-gfm": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/turndown-plugin-gfm/-/turndown-plugin-gfm-1.0.2.tgz",
"integrity": "sha512-vwz9tfvF7XN/jE0dGoBei3FXWuvll78ohzCZQuOb+ZjWrs3a0XhQVomJEb2Qh4VHTPNRO4GPZh0V7VRbiWwkRg==",
"dev": true,
"license": "MIT"
},
"node_modules/uhyphen": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/uhyphen/-/uhyphen-0.2.0.tgz",
"integrity": "sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA==",
"dev": true,
"license": "ISC"
},
"node_modules/undici": {
"version": "6.24.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-6.24.0.tgz",
+4
View File
@@ -10,6 +10,10 @@
"test:coverage": "node --import tsx --experimental-test-coverage --test"
},
"devDependencies": {
"@mozilla/readability": "^0.6.0",
"linkedom": "^0.18.12",
"turndown": "^7.2.2",
"turndown-plugin-gfm": "^1.0.2",
"tsx": "^4.20.5"
}
}
+34
View File
@@ -9,12 +9,26 @@ import { COLOR_PRESETS, FONT_FAMILY_MAP } from "./constants.ts";
import {
buildMarkdownDocumentMeta,
formatTimestamp,
renderMarkdownDocument,
resolveColorToken,
resolveFontFamilyToken,
resolveMarkdownStyle,
resolveRenderOptions,
} from "./document.ts";
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`);
}
function findInlineStyle(html: string, tagName: string, text: string): string {
const pattern = new RegExp(
`<${tagName}[^>]*style="([^"]*)"[^>]*>${escapeRegExp(text)}</${tagName}>`,
);
const match = html.match(pattern);
assert.ok(match, `Expected inline style for <${tagName}>${text}</${tagName}>`);
return match![1]!;
}
function useCwd(t: TestContext, cwd: string): void {
const previous = process.cwd();
process.chdir(cwd);
@@ -138,3 +152,23 @@ keep_title: true
assert.equal(explicit.fontSize, "18px");
assert.equal(explicit.keepTitle, false);
});
test("renderMarkdownDocument layers default rules into grace theme before CSS inlining", async () => {
const { html } = await renderMarkdownDocument(
`## Section\n\nParagraph with **bold** text.`,
{ keepTitle: true, theme: "grace" },
);
const h2Style = findInlineStyle(html, "h2", "Section");
assert.match(h2Style, /background: #92617E/);
assert.match(h2Style, /box-shadow: 0 4px 6px rgba\(0, 0, 0, 0\.1\)/);
const pMatch = html.match(/<p[^>]*style="([^"]*)"[^>]*>/);
assert.ok(pMatch, "Expected inline style on <p> tag");
assert.match(pMatch![1]!, /color:/);
const strongPattern = /<strong[^>]*style="([^"]*)"[^>]*>bold<\/strong>/;
const strongMatch = html.match(strongPattern);
assert.ok(strongMatch, "Expected inline style for <strong>bold</strong>");
assert.match(strongMatch![1]!, /font-weight:/);
});
@@ -59,6 +59,17 @@ test("normalizeCssText and normalizeInlineCss replace variables and strip declar
assert.doesNotMatch(normalizedHtml, /var\(--md-primary-color\)/);
});
test("normalizeInlineCss removes quoted custom property values without leaving fragments behind", () => {
const normalizedHtml = normalizeInlineCss(
`<html style="--md-font-family: Menlo, Monaco, 'Courier New', monospace; color: var(--md-primary-color)"></html>`,
DEFAULT_STYLE,
);
assert.match(normalizedHtml, /style=" color: #0F4C81"/);
assert.doesNotMatch(normalizedHtml, /Courier New/);
assert.doesNotMatch(normalizedHtml, /--md-font-family/);
});
test("HTML structure helpers hoist nested lists and remove the first heading", () => {
const nestedList = `<ul><li>Parent<ul><li>Child</li></ul></li></ul>`;
assert.equal(
+7 -7
View File
@@ -100,13 +100,13 @@ export function normalizeCssText(cssText: string, style: StyleConfig = DEFAULT_S
.replace(/var\(--md-accent-color\)/g, style.accentColor)
.replace(/var\(--md-container-bg\)/g, style.containerBg)
.replace(/hsl\(var\(--foreground\)\)/g, "#3f3f3f")
.replace(/--md-primary-color:\s*[^;"']+;?/g, "")
.replace(/--md-font-family:\s*[^;"']+;?/g, "")
.replace(/--md-font-size:\s*[^;"']+;?/g, "")
.replace(/--blockquote-background:\s*[^;"']+;?/g, "")
.replace(/--md-accent-color:\s*[^;"']+;?/g, "")
.replace(/--md-container-bg:\s*[^;"']+;?/g, "")
.replace(/--foreground:\s*[^;"']+;?/g, "");
.replace(/--md-primary-color:\s*[^;]+;?/g, "")
.replace(/--md-font-family:\s*[^;]+;?/g, "")
.replace(/--md-font-size:\s*[^;]+;?/g, "")
.replace(/--blockquote-background:\s*[^;]+;?/g, "")
.replace(/--md-accent-color:\s*[^;]+;?/g, "")
.replace(/--md-container-bg:\s*[^;]+;?/g, "")
.replace(/--foreground:\s*[^;]+;?/g, "");
}
export function normalizeInlineCss(html: string, style: StyleConfig = DEFAULT_STYLE): string {
+12 -1
View File
@@ -6,6 +6,7 @@ import type { ThemeName } from "./types.js";
const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
export const THEME_DIR = path.resolve(SCRIPT_DIR, "themes");
const FALLBACK_THEMES: ThemeName[] = ["default", "grace", "simple"];
const THEMES_EXTENDING_DEFAULT = new Set<ThemeName>(["grace", "simple"]);
function stripOutputScope(cssContent: string): string {
let css = cssContent;
@@ -41,6 +42,7 @@ export function loadThemeCss(theme: ThemeName): {
themeCss: string;
} {
const basePath = path.join(THEME_DIR, "base.css");
const defaultThemePath = path.join(THEME_DIR, "default.css");
const themePath = path.join(THEME_DIR, `${theme}.css`);
if (!fs.existsSync(basePath)) {
@@ -51,9 +53,18 @@ export function loadThemeCss(theme: ThemeName): {
throw new Error(`Missing theme CSS for "${theme}": ${themePath}`);
}
const layeredThemeCss: string[] = [];
if (theme !== "default" && THEMES_EXTENDING_DEFAULT.has(theme)) {
if (!fs.existsSync(defaultThemePath)) {
throw new Error(`Missing default theme CSS: ${defaultThemePath}`);
}
layeredThemeCss.push(fs.readFileSync(defaultThemePath, "utf-8"));
}
layeredThemeCss.push(fs.readFileSync(themePath, "utf-8"));
return {
baseCss: fs.readFileSync(basePath, "utf-8"),
themeCss: fs.readFileSync(themePath, "utf-8"),
themeCss: layeredThemeCss.join("\n"),
};
}
+24 -11
View File
@@ -151,6 +151,9 @@ async function main() {
.map((tag) => tag.trim())
.filter(Boolean);
let succeeded = 0;
const failed = [];
for (const candidate of actionable) {
const version =
candidate.status === "new"
@@ -158,20 +161,30 @@ async function main() {
: bumpSemver(candidate.latestVersion, options.bump);
console.log(`Publishing ${candidate.slug}@${version}`);
const files = await listTextFiles(candidate.folder);
await publishSkill({
registry,
token: config.token,
skill: candidate,
files,
version,
changelog: options.changelog,
tags,
});
try {
const files = await listTextFiles(candidate.folder);
await publishSkill({
registry,
token: config.token,
skill: candidate,
files,
version,
changelog: options.changelog,
tags,
});
succeeded++;
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
console.error(`SKIPPED ${candidate.slug}: ${msg}`);
failed.push(candidate.slug);
}
}
console.log("");
console.log(`Uploaded ${actionable.length} skill(s).`);
console.log(`Uploaded ${succeeded}/${actionable.length} skill(s).`);
if (failed.length > 0) {
console.log(`Failed (${failed.length}): ${failed.join(", ")}`);
}
}
function parseArgs(argv) {
@@ -280,5 +280,5 @@ TEXTURE: Halftone transitions between sides
If watermark enabled in preferences, append:
```
Include a subtle watermark "[content]" positioned at [position] with approximately [opacity*100]% visibility.
Include a subtle watermark "[content]" positioned at [position].
```
+4 -5
View File
@@ -278,7 +278,7 @@ Create storyboard and character definitions using the confirmed style from Step
| Role | Character | Visual Description |
|------|-----------|-------------------|
| Student | 大雄 (Nobita) | Japanese boy, 10yo, round glasses, black hair parted in middle, yellow shirt, navy shorts |
| Mentor | 哆啦A梦 (Doraemon) | Round blue robot cat, big white eyes, red nose, whiskers, white belly with 4D pocket, golden bell, no ears |
| Mentor | 哆啦 A 梦 (Doraemon) | Round blue robot cat, big white eyes, red nose, whiskers, white belly with 4D pocket, golden bell, no ears |
| Challenge | 胖虎 (Gian) | Stocky boy, rough features, small eyes, orange shirt |
| Support | 静香 (Shizuka) | Cute girl, black short hair, pink dress, gentle expression |
@@ -359,8 +359,7 @@ Art: [art style] | Tone: [tone] | Layout: [layout type]
**Watermark Application** (if enabled in preferences):
Add to each prompt:
```
Include a subtle watermark "[content]" positioned at [position]
with approximately [opacity*100]% visibility. The watermark should
Include a subtle watermark "[content]" positioned at [position]. The watermark should
be legible but not distracting from the comic panels and storytelling.
Ensure watermark does not overlap speech bubbles or key action.
```
@@ -452,8 +451,8 @@ When skill does NOT support reference images, create combined prompt files:
## Character Reference (maintain consistency)
[Copy relevant sections from characters/characters.md here]
- 大雄: Japanese boy, round glasses, yellow shirt, navy shorts...
- 哆啦A梦: Round blue robot cat, white belly, red nose, golden bell...
- 大雄Japanese boy, round glasses, yellow shirt, navy shorts...
- 哆啦 A 梦:Round blue robot cat, white belly, red nose, golden bell...
## Page Content
[Original page prompt here]
+4 -5
View File
@@ -162,15 +162,14 @@ if (Test-Path "$HOME/.baoyu-skills/baoyu-cover-image/EXTEND.md") { "user" }
5. **Detect language**: Compare source, user input, EXTEND.md preference
6. **Determine output directory**: Per File Structure rules
**⚠️ People in Reference Images — MUST follow all 3 rules:**
**⚠️ People in Reference Images:**
If reference images contain **people** who should appear in the cover:
1. **`usage: direct`** — MUST set in refs description file. NEVER use `style` or `palette` when people need to appear
2. **Per-character description** — MUST describe each person's distinctive features (hair, glasses, skin tone, clothing) in `refs/ref-NN-{slug}.md`. Vague descriptions like "a man" will fail
3. **`--ref` flag** — MUST pass reference image via `--ref` in Step 4 so the model sees actual faces
- **Model supports `--ref`** (default): Copy image to `refs/`, pass via `--ref` at generation. No description file needed — the model sees the face directly.
- **Model does NOT support `--ref`** (Jimeng, Seedream 3.0): Create `refs/ref-NN-{slug}.md` with per-character description (hair, glasses, skin tone, clothing). Embed as MUST/REQUIRED instructions in prompt text.
See [reference-images.md § Character Analysis](references/workflow/reference-images.md) for description format.
See [reference-images.md](references/workflow/reference-images.md) for full decision table.
### Step 2: Confirm Options ⚠️
@@ -16,17 +16,24 @@ Guide for processing user-provided reference images in cover generation.
**If user provides file path**:
1. Copy to `refs/ref-NN-{slug}.{ext}` (NN = 01, 02, ...)
2. Create description: `refs/ref-NN-{slug}.md`
3. Verify files exist before proceeding
2. **Only** create description file `refs/ref-NN-{slug}.md` when model does NOT support `--ref` (see below)
3. Verify image file exists before proceeding
**Description File Format**:
**When to create description file**:
| Situation | Action |
|-----------|--------|
| Model supports `--ref` (Google, OpenAI, OpenRouter, Replicate, Seedream 4.0+) | Copy image only. **No description file needed.** Pass via `--ref` at generation. |
| Model does NOT support `--ref` (Jimeng, Seedream 3.0) | Copy image + create description file. Embed description in prompt text. |
**Description File Format** (only when needed):
```yaml
---
ref_id: NN
filename: ref-NN-{slug}.{ext}
usage: direct | style | palette
---
[User's description or auto-generated description]
[Character or style description to embed in prompt]
```
| Usage | When to Use |
@@ -0,0 +1,168 @@
import assert from "node:assert/strict";
import test from "node:test";
import type { CliArgs } from "../types.ts";
import {
buildContent,
buildRequestBody,
extractImageFromResponse,
getAspectRatio,
getImageSize,
validateArgs,
} from "./openrouter.ts";
const GEMINI_MODEL = "google/gemini-3.1-flash-image-preview";
const GEMINI_25_MODEL = "google/gemini-2.5-flash-image";
const GPT_5_IMAGE_MODEL = "openai/gpt-5-image";
const OPENROUTER_AUTO_MODEL = "openrouter/auto";
const FLUX_MODEL = "black-forest-labs/flux.2-pro";
function makeArgs(overrides: Partial<CliArgs> = {}): CliArgs {
return {
prompt: null,
promptFiles: [],
imagePath: null,
provider: null,
model: null,
aspectRatio: null,
size: null,
quality: null,
imageSize: null,
referenceImages: [],
n: 1,
batchFile: null,
jobs: null,
json: false,
help: false,
...overrides,
};
}
test("OpenRouter request body uses image_config and string content for text-only prompts", () => {
const args = makeArgs({ aspectRatio: "16:9", quality: "2k" });
const body = buildRequestBody("hello", GEMINI_MODEL, args, []);
assert.deepEqual(body.image_config, {
image_size: "2K",
aspect_ratio: "16:9",
});
assert.deepEqual(body.provider, {
require_parameters: true,
});
assert.deepEqual(body.modalities, ["image", "text"]);
assert.equal(body.stream, false);
assert.equal(body.messages[0].content, "hello");
});
test("OpenRouter request body keeps text+image modalities for current text+image models", () => {
for (const model of [GEMINI_MODEL, GEMINI_25_MODEL, GPT_5_IMAGE_MODEL, OPENROUTER_AUTO_MODEL]) {
const body = buildRequestBody("hello", model, makeArgs({ quality: "2k" }), []);
assert.deepEqual(body.image_config, {
image_size: "2K",
});
assert.deepEqual(body.provider, {
require_parameters: true,
});
assert.deepEqual(body.modalities, ["image", "text"]);
assert.equal(body.messages[0].content, "hello");
}
});
test("OpenRouter request body uses image-only modalities for image-only models under CLI defaults", () => {
const body = buildRequestBody("hello", FLUX_MODEL, makeArgs({ quality: "2k" }), []);
assert.deepEqual(body.image_config, {
image_size: "2K",
});
assert.deepEqual(body.provider, {
require_parameters: true,
});
assert.deepEqual(body.modalities, ["image"]);
assert.equal(body.stream, false);
assert.equal(body.messages[0].content, "hello");
});
test("OpenRouter helper omits image_config when no size or quality is passed", () => {
const body = buildRequestBody("hello", FLUX_MODEL, makeArgs(), []);
assert.equal(body.image_config, undefined);
assert.equal(body.provider, undefined);
assert.deepEqual(body.modalities, ["image"]);
assert.equal(body.stream, false);
assert.equal(body.messages[0].content, "hello");
});
test("OpenRouter request body keeps multimodal array content when references are provided", () => {
const content = buildContent("hello", ["data:image/png;base64,abc"]);
assert.ok(Array.isArray(content));
assert.deepEqual(content[0], { type: "text", text: "hello" });
assert.deepEqual(content[1], {
type: "image_url",
image_url: { url: "data:image/png;base64,abc" },
});
});
test("OpenRouter size and aspect helpers infer supported values", () => {
assert.equal(getImageSize(makeArgs()), null);
assert.equal(getImageSize(makeArgs({ quality: "normal" })), "1K");
assert.equal(getImageSize(makeArgs({ size: "2048x1024" })), "2K");
assert.equal(getAspectRatio(GEMINI_MODEL, makeArgs({ size: "1600x900" })), "16:9");
assert.equal(getAspectRatio(GEMINI_MODEL, makeArgs({ size: "1024x4096" })), "1:4");
assert.equal(getAspectRatio(GEMINI_25_MODEL, makeArgs({ size: "1600x900" })), "16:9");
assert.equal(getAspectRatio(FLUX_MODEL, makeArgs({ size: "1024x4096" })), null);
});
test("OpenRouter validates explicit aspect ratios and inferred size ratios against model support", () => {
assert.doesNotThrow(() =>
validateArgs(GEMINI_MODEL, makeArgs({ aspectRatio: "1:4" })),
);
assert.doesNotThrow(() =>
validateArgs(GEMINI_MODEL, makeArgs({ size: "1024x4096" })),
);
assert.throws(
() => validateArgs(GEMINI_25_MODEL, makeArgs({ aspectRatio: "1:4" })),
/does not support aspect ratio 1:4/,
);
assert.throws(
() => validateArgs(FLUX_MODEL, makeArgs({ aspectRatio: "1:4" })),
/does not support aspect ratio 1:4/,
);
assert.throws(
() => validateArgs(GEMINI_MODEL, makeArgs({ size: "2048x1024" })),
/does not support size 2048x1024 \(aspect ratio 2:1\)/,
);
});
test("OpenRouter response extraction supports inline image data and finish_reason errors", async () => {
const bytes = await extractImageFromResponse({
choices: [
{
message: {
images: [
{
image_url: {
url: `data:image/png;base64,${Buffer.from("hello").toString("base64")}`,
},
},
],
},
},
],
});
assert.equal(Buffer.from(bytes).toString("utf8"), "hello");
await assert.rejects(
() =>
extractImageFromResponse({
choices: [
{
finish_reason: "error",
native_finish_reason: "MALFORMED_FUNCTION_CALL",
message: { content: null },
},
],
}),
/finish_reason=MALFORMED_FUNCTION_CALL/,
);
});
@@ -3,6 +3,19 @@ import { readFile } from "node:fs/promises";
import type { CliArgs } from "../types";
const DEFAULT_MODEL = "google/gemini-3.1-flash-image-preview";
const COMMON_ASPECT_RATIOS = [
"1:1",
"2:3",
"3:2",
"3:4",
"4:3",
"4:5",
"5:4",
"9:16",
"16:9",
"21:9",
];
const GEMINI_EXTENDED_ASPECT_RATIOS = ["1:4", "4:1", "1:8", "8:1"];
type OpenRouterImageEntry = {
image_url?: string | { url?: string | null } | null;
@@ -18,9 +31,11 @@ type OpenRouterMessagePart = {
type OpenRouterResponse = {
choices?: Array<{
finish_reason?: string | null;
native_finish_reason?: string | null;
message?: {
images?: OpenRouterImageEntry[];
content?: string | OpenRouterMessagePart[];
content?: string | OpenRouterMessagePart[] | null;
};
}>;
};
@@ -29,6 +44,36 @@ export function getDefaultModel(): string {
return process.env.OPENROUTER_IMAGE_MODEL || DEFAULT_MODEL;
}
function normalizeModelId(model: string): string {
return model.trim().toLowerCase().split(":")[0]!;
}
function isTextAndImageModel(model: string): boolean {
const normalized = normalizeModelId(model);
if (normalized === "openrouter/auto") {
return true;
}
if (normalized.startsWith("google/gemini-") && normalized.includes("image")) {
return true;
}
if (normalized.startsWith("openai/gpt-") && normalized.includes("image")) {
return true;
}
return false;
}
function getSupportedAspectRatios(model: string): Set<string> {
const normalized = normalizeModelId(model);
if (normalized !== "google/gemini-3.1-flash-image-preview") {
return new Set(COMMON_ASPECT_RATIOS);
}
return new Set([...COMMON_ASPECT_RATIOS, ...GEMINI_EXTENDED_ASPECT_RATIOS]);
}
function getApiKey(): string | null {
return process.env.OPENROUTER_API_KEY || null;
}
@@ -103,17 +148,50 @@ function inferImageSize(size: string | null): "1K" | "2K" | "4K" | null {
return "4K";
}
function getImageSize(args: CliArgs): "1K" | "2K" | "4K" {
export function getImageSize(args: CliArgs): "1K" | "2K" | "4K" | null {
if (args.imageSize) return args.imageSize as "1K" | "2K" | "4K";
const inferredFromSize = inferImageSize(args.size);
if (inferredFromSize) return inferredFromSize;
return args.quality === "normal" ? "1K" : "2K";
if (args.quality === "normal") return "1K";
if (args.quality === "2k") return "2K";
return null;
}
function getAspectRatio(args: CliArgs): string | null {
return args.aspectRatio || inferAspectRatio(args.size);
export function getAspectRatio(model: string, args: CliArgs): string | null {
if (args.aspectRatio) return args.aspectRatio;
const inferred = inferAspectRatio(args.size);
if (!inferred || !getSupportedAspectRatios(model).has(inferred)) {
return null;
}
return inferred;
}
function getModalities(model: string): string[] {
return isTextAndImageModel(model) ? ["image", "text"] : ["image"];
}
export function validateArgs(model: string, args: CliArgs): void {
const requestedAspectRatio = args.aspectRatio || inferAspectRatio(args.size);
if (!requestedAspectRatio) {
return;
}
const supported = getSupportedAspectRatios(model);
if (supported.has(requestedAspectRatio)) {
return;
}
const requestedValue = args.aspectRatio
? `aspect ratio ${requestedAspectRatio}`
: `size ${args.size} (aspect ratio ${requestedAspectRatio})`;
throw new Error(
`OpenRouter model ${model} does not support ${requestedValue}. Supported values: ${Array.from(supported).join(", ")}`
);
}
function getMimeType(filename: string): string {
@@ -129,7 +207,14 @@ async function readImageAsDataUrl(filePath: string): Promise<string> {
return `data:${getMimeType(filePath)};base64,${bytes.toString("base64")}`;
}
function buildContent(prompt: string, referenceImages: string[]): Array<Record<string, unknown>> {
export function buildContent(
prompt: string,
referenceImages: string[],
): string | Array<Record<string, unknown>> {
if (referenceImages.length === 0) {
return prompt;
}
const content: Array<Record<string, unknown>> = [{ type: "text", text: prompt }];
for (const imageUrl of referenceImages) {
@@ -171,8 +256,9 @@ async function downloadImage(value: string): Promise<Uint8Array> {
return Uint8Array.from(Buffer.from(value, "base64"));
}
async function extractImageFromResponse(result: OpenRouterResponse): Promise<Uint8Array> {
const message = result.choices?.[0]?.message;
export async function extractImageFromResponse(result: OpenRouterResponse): Promise<Uint8Array> {
const choice = result.choices?.[0];
const message = choice?.message;
for (const image of message?.images ?? []) {
const imageUrl = extractImageUrl(image);
@@ -194,7 +280,52 @@ async function extractImageFromResponse(result: OpenRouterResponse): Promise<Uin
if (inline) return inline;
}
throw new Error("No image in OpenRouter response");
const finishReason =
choice?.native_finish_reason || choice?.finish_reason || "unknown";
throw new Error(
`No image in OpenRouter response (finish_reason=${finishReason})`,
);
}
export function buildRequestBody(
prompt: string,
model: string,
args: CliArgs,
referenceImages: string[],
): Record<string, unknown> {
validateArgs(model, args);
const imageConfig: Record<string, string> = {};
const imageSize = getImageSize(args);
if (imageSize) {
imageConfig.image_size = imageSize;
}
const aspectRatio = getAspectRatio(model, args);
if (aspectRatio) {
imageConfig.aspect_ratio = aspectRatio;
}
const body: Record<string, unknown> = {
messages: [
{
role: "user",
content: buildContent(prompt, referenceImages),
},
],
modalities: getModalities(model),
stream: false,
};
if (Object.keys(imageConfig).length > 0) {
body.image_config = imageConfig;
body.provider = {
require_parameters: true,
};
}
return body;
}
export async function generateImage(
@@ -212,32 +343,15 @@ export async function generateImage(
referenceImages.push(await readImageAsDataUrl(refPath));
}
const imageGenerationOptions: Record<string, string> = {
size: getImageSize(args),
};
const aspectRatio = getAspectRatio(args);
if (aspectRatio) {
imageGenerationOptions.aspect_ratio = aspectRatio;
}
const body = {
model,
messages: [
{
role: "user",
content: buildContent(prompt, referenceImages),
},
],
modalities: ["image", "text"],
max_tokens: 256,
imageGenerationOptions,
providerPreferences: {
require_parameters: true,
},
...buildRequestBody(prompt, model, args, referenceImages),
};
console.log(`Generating image with OpenRouter (${model})...`, imageGenerationOptions);
console.log(
`Generating image with OpenRouter (${model})...`,
(body.image_config as Record<string, string>),
);
const response = await fetch(`${getBaseUrl()}/chat/completions`, {
method: "POST",
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: baoyu-markdown-to-html
description: Converts Markdown to styled HTML with WeChat-compatible themes. Supports code highlighting, math, PlantUML, footnotes, alerts, infographics, and optional bottom citations for external links. Use when user asks for "markdown to html", "convert md to html", "mdhtml", "微信外链转底部引用", or needs styled HTML output from markdown.
description: Converts Markdown to styled HTML with WeChat-compatible themes. Supports code highlighting, math, PlantUML, footnotes, alerts, infographics, and optional bottom citations for external links. Use when user asks for "markdown to html", "convert md to html", "mdhtml", "微信外链转底部引用", or needs styled HTML output from markdown.
version: 1.56.1
metadata:
openclaw:
@@ -0,0 +1,56 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import process from "node:process";
import test from "node:test";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
const execFileAsync = promisify(execFile);
const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
const SCRIPT_PATH = path.join(SCRIPT_DIR, "main.ts");
async function makeTempDir(prefix: string): Promise<string> {
return fs.mkdtemp(path.join(os.tmpdir(), prefix));
}
test("CLI forwards wrapper title and vendor render options", async () => {
const root = await makeTempDir("baoyu-markdown-to-html-cli-");
const markdownPath = path.join(root, "article.md");
await fs.writeFile(markdownPath, "## Section\n\nParagraph with **bold** text.\n", "utf-8");
const { stdout } = await execFileAsync(
process.execPath,
[
"--import",
"tsx",
SCRIPT_PATH,
markdownPath,
"--theme", "grace",
"--color", "red",
"--font-family", "mono",
"--font-size", "18",
"--keep-title",
"--title", "Overridden",
],
{ cwd: SCRIPT_DIR },
);
const result = JSON.parse(stdout.trim()) as {
htmlPath: string;
title: string;
};
assert.equal(result.title, "Overridden");
const html = await fs.readFile(result.htmlPath, "utf-8");
assert.match(html, /<title>Overridden<\/title>/);
assert.match(html, /<h2[^>]*style="[^"]*background: #A93226/);
assert.match(html, /<strong[^>]*style="[^"]*color: #A93226/);
assert.match(
html,
/<body[^>]*style="[^"]*font-family: Menlo, Monaco, 'Courier New', monospace;[^"]*font-size: 18px/,
);
});
+87 -33
View File
@@ -4,16 +4,22 @@ import path from "node:path";
import process from "node:process";
import {
COLOR_PRESETS,
FONT_FAMILY_MAP,
FONT_SIZE_OPTIONS,
THEME_NAMES,
extractSummaryFromBody,
extractTitleFromMarkdown,
formatTimestamp,
parseArgs,
parseFrontmatter,
renderMarkdownDocument,
replaceMarkdownImagesWithPlaceholders,
resolveContentImages,
serializeFrontmatter,
stripWrappingQuotes,
} from "baoyu-md";
} from "./vendor/baoyu-md/src/index.ts";
import type { CliOptions } from "./vendor/baoyu-md/src/types.ts";
interface ImageInfo {
placeholder: string;
@@ -30,9 +36,13 @@ interface ParsedResult {
contentImages: ImageInfo[];
}
type ConvertMarkdownOptions = Partial<Omit<CliOptions, "inputPath">> & {
title?: string;
};
export async function convertMarkdown(
markdownPath: string,
options?: { title?: string; theme?: string; keepTitle?: boolean; citeStatus?: boolean },
options?: ConvertMarkdownOptions,
): Promise<ParsedResult> {
const baseDir = path.dirname(markdownPath);
const content = fs.readFileSync(markdownPath, "utf-8");
@@ -56,20 +66,32 @@ export async function convertMarkdown(
summary = extractSummaryFromBody(body, 120);
}
const effectiveFrontmatter = options?.title
? { ...frontmatter, title }
: frontmatter;
const { images, markdown: rewrittenBody } = replaceMarkdownImagesWithPlaceholders(
body,
"MDTOHTMLIMGPH_",
);
const rewrittenMarkdown = `${serializeFrontmatter(frontmatter)}${rewrittenBody}`;
const rewrittenMarkdown = `${serializeFrontmatter(effectiveFrontmatter)}${rewrittenBody}`;
console.error(
`[markdown-to-html] Rendering with theme: ${theme ?? "default"}, keepTitle: ${keepTitle}, citeStatus: ${citeStatus}`,
);
const { html } = await renderMarkdownDocument(rewrittenMarkdown, {
codeTheme: options?.codeTheme,
countStatus: options?.countStatus,
citeStatus,
defaultTitle: title,
fontFamily: options?.fontFamily,
fontSize: options?.fontSize,
isMacCodeBlock: options?.isMacCodeBlock,
isShowLineNumber: options?.isShowLineNumber,
keepTitle,
legend: options?.legend,
primaryColor: options?.primaryColor,
theme,
});
@@ -111,18 +133,30 @@ export async function convertMarkdown(
};
}
function printUsage(): never {
function printUsage(exitCode = 0): never {
const colorNames = Object.keys(COLOR_PRESETS).join(", ");
const fontFamilyNames = Object.keys(FONT_FAMILY_MAP).join(", ");
console.log(`Convert Markdown to styled HTML
Usage:
npx -y bun main.ts <markdown_file> [options]
Options:
--title <title> Override title
--theme <name> Theme name (default, grace, simple). Default: default
--cite Convert ordinary external links to bottom citations. Default: off
--keep-title Keep the first heading in content. Default: false (removed)
--help Show this help
--title <title> Override title
--theme <name> Theme name (${THEME_NAMES.join(", ")}). Default: default
--color <name|hex> Primary color: ${colorNames}
--font-family <name> Font: ${fontFamilyNames}, or CSS value
--font-size <N> Font size: ${FONT_SIZE_OPTIONS.join(", ")} (default: 16px)
--code-theme <name> Code highlight theme (default: github)
--mac-code-block Show Mac-style code block header
--no-mac-code-block Hide Mac-style code block header
--line-number Show line numbers in code blocks
--cite Convert ordinary external links to bottom citations. Default: off
--count Show reading time / word count
--legend <value> Image caption: title-alt, alt-title, title, alt, none
--keep-title Keep the first heading in content. Default: false (removed)
--help Show this help
Output:
HTML file saved to same directory as input markdown file.
@@ -142,40 +176,60 @@ Output JSON format:
Example:
npx -y bun main.ts article.md
npx -y bun main.ts article.md --theme grace
npx -y bun main.ts article.md --theme modern --color red
npx -y bun main.ts article.md --cite
`);
process.exit(0);
process.exit(exitCode);
}
function parseArgValue(argv: string[], i: number, flag: string): string | null {
const arg = argv[i]!;
if (arg.includes("=")) {
return arg.slice(flag.length + 1);
}
const next = argv[i + 1];
return next ?? null;
}
function extractTitleArg(argv: string[]): { renderArgs: string[]; title?: string } {
let title: string | undefined;
const renderArgs: string[] = [];
for (let i = 0; i < argv.length; i += 1) {
const arg = argv[i]!;
if (arg === "--title" || arg.startsWith("--title=")) {
const value = parseArgValue(argv, i, "--title");
if (!value) {
console.error("Missing value for --title");
printUsage(1);
}
title = value;
if (!arg.includes("=")) {
i += 1;
}
continue;
}
renderArgs.push(arg);
}
return { renderArgs, title };
}
async function main(): Promise<void> {
const args = process.argv.slice(2);
if (args.length === 0 || args.includes("--help") || args.includes("-h")) {
printUsage();
printUsage(0);
}
let markdownPath: string | undefined;
let title: string | undefined;
let theme: string | undefined;
let citeStatus = false;
let keepTitle = false;
for (let i = 0; i < args.length; i++) {
const arg = args[i]!;
if (arg === "--title" && args[i + 1]) {
title = args[++i];
} else if (arg === "--theme" && args[i + 1]) {
theme = args[++i];
} else if (arg === "--cite") {
citeStatus = true;
} else if (arg === "--keep-title") {
keepTitle = true;
} else if (!arg.startsWith("-")) {
markdownPath = arg;
}
const { renderArgs, title } = extractTitleArg(args);
const options = parseArgs(renderArgs);
if (!options) {
printUsage(1);
}
if (!markdownPath) {
console.error("Error: Markdown file path is required");
const markdownPath = path.resolve(process.cwd(), options.inputPath);
if (!markdownPath.toLowerCase().endsWith(".md")) {
console.error("Input file must end with .md");
process.exit(1);
}
@@ -184,7 +238,7 @@ async function main(): Promise<void> {
process.exit(1);
}
const result = await convertMarkdown(markdownPath, { title, theme, keepTitle, citeStatus });
const result = await convertMarkdown(markdownPath, { ...options, title });
console.log(JSON.stringify(result, null, 2));
}
@@ -9,12 +9,26 @@ import { COLOR_PRESETS, FONT_FAMILY_MAP } from "./constants.ts";
import {
buildMarkdownDocumentMeta,
formatTimestamp,
renderMarkdownDocument,
resolveColorToken,
resolveFontFamilyToken,
resolveMarkdownStyle,
resolveRenderOptions,
} from "./document.ts";
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`);
}
function findInlineStyle(html: string, tagName: string, text: string): string {
const pattern = new RegExp(
`<${tagName}[^>]*style="([^"]*)"[^>]*>${escapeRegExp(text)}</${tagName}>`,
);
const match = html.match(pattern);
assert.ok(match, `Expected inline style for <${tagName}>${text}</${tagName}>`);
return match![1]!;
}
function useCwd(t: TestContext, cwd: string): void {
const previous = process.cwd();
process.chdir(cwd);
@@ -138,3 +152,23 @@ keep_title: true
assert.equal(explicit.fontSize, "18px");
assert.equal(explicit.keepTitle, false);
});
test("renderMarkdownDocument layers default rules into grace theme before CSS inlining", async () => {
const { html } = await renderMarkdownDocument(
`## Section\n\nParagraph with **bold** text.`,
{ keepTitle: true, theme: "grace" },
);
const h2Style = findInlineStyle(html, "h2", "Section");
assert.match(h2Style, /background: #92617E/);
assert.match(h2Style, /box-shadow: 0 4px 6px rgba\(0, 0, 0, 0\.1\)/);
const pMatch = html.match(/<p[^>]*style="([^"]*)"[^>]*>/);
assert.ok(pMatch, "Expected inline style on <p> tag");
assert.match(pMatch![1]!, /color:/);
const strongPattern = /<strong[^>]*style="([^"]*)"[^>]*>bold<\/strong>/;
const strongMatch = html.match(strongPattern);
assert.ok(strongMatch, "Expected inline style for <strong>bold</strong>");
assert.match(strongMatch![1]!, /font-weight:/);
});
@@ -59,6 +59,17 @@ test("normalizeCssText and normalizeInlineCss replace variables and strip declar
assert.doesNotMatch(normalizedHtml, /var\(--md-primary-color\)/);
});
test("normalizeInlineCss removes quoted custom property values without leaving fragments behind", () => {
const normalizedHtml = normalizeInlineCss(
`<html style="--md-font-family: Menlo, Monaco, 'Courier New', monospace; color: var(--md-primary-color)"></html>`,
DEFAULT_STYLE,
);
assert.match(normalizedHtml, /style=" color: #0F4C81"/);
assert.doesNotMatch(normalizedHtml, /Courier New/);
assert.doesNotMatch(normalizedHtml, /--md-font-family/);
});
test("HTML structure helpers hoist nested lists and remove the first heading", () => {
const nestedList = `<ul><li>Parent<ul><li>Child</li></ul></li></ul>`;
assert.equal(
@@ -100,13 +100,13 @@ export function normalizeCssText(cssText: string, style: StyleConfig = DEFAULT_S
.replace(/var\(--md-accent-color\)/g, style.accentColor)
.replace(/var\(--md-container-bg\)/g, style.containerBg)
.replace(/hsl\(var\(--foreground\)\)/g, "#3f3f3f")
.replace(/--md-primary-color:\s*[^;"']+;?/g, "")
.replace(/--md-font-family:\s*[^;"']+;?/g, "")
.replace(/--md-font-size:\s*[^;"']+;?/g, "")
.replace(/--blockquote-background:\s*[^;"']+;?/g, "")
.replace(/--md-accent-color:\s*[^;"']+;?/g, "")
.replace(/--md-container-bg:\s*[^;"']+;?/g, "")
.replace(/--foreground:\s*[^;"']+;?/g, "");
.replace(/--md-primary-color:\s*[^;]+;?/g, "")
.replace(/--md-font-family:\s*[^;]+;?/g, "")
.replace(/--md-font-size:\s*[^;]+;?/g, "")
.replace(/--blockquote-background:\s*[^;]+;?/g, "")
.replace(/--md-accent-color:\s*[^;]+;?/g, "")
.replace(/--md-container-bg:\s*[^;]+;?/g, "")
.replace(/--foreground:\s*[^;]+;?/g, "");
}
export function normalizeInlineCss(html: string, style: StyleConfig = DEFAULT_STYLE): string {
@@ -6,6 +6,7 @@ import type { ThemeName } from "./types.js";
const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
export const THEME_DIR = path.resolve(SCRIPT_DIR, "themes");
const FALLBACK_THEMES: ThemeName[] = ["default", "grace", "simple"];
const THEMES_EXTENDING_DEFAULT = new Set<ThemeName>(["grace", "simple"]);
function stripOutputScope(cssContent: string): string {
let css = cssContent;
@@ -41,6 +42,7 @@ export function loadThemeCss(theme: ThemeName): {
themeCss: string;
} {
const basePath = path.join(THEME_DIR, "base.css");
const defaultThemePath = path.join(THEME_DIR, "default.css");
const themePath = path.join(THEME_DIR, `${theme}.css`);
if (!fs.existsSync(basePath)) {
@@ -51,9 +53,18 @@ export function loadThemeCss(theme: ThemeName): {
throw new Error(`Missing theme CSS for "${theme}": ${themePath}`);
}
const layeredThemeCss: string[] = [];
if (theme !== "default" && THEMES_EXTENDING_DEFAULT.has(theme)) {
if (!fs.existsSync(defaultThemePath)) {
throw new Error(`Missing default theme CSS: ${defaultThemePath}`);
}
layeredThemeCss.push(fs.readFileSync(defaultThemePath, "utf-8"));
}
layeredThemeCss.push(fs.readFileSync(themePath, "utf-8"));
return {
baseCss: fs.readFileSync(basePath, "utf-8"),
themeCss: fs.readFileSync(themePath, "utf-8"),
themeCss: layeredThemeCss.join("\n"),
};
}
@@ -9,12 +9,26 @@ import { COLOR_PRESETS, FONT_FAMILY_MAP } from "./constants.ts";
import {
buildMarkdownDocumentMeta,
formatTimestamp,
renderMarkdownDocument,
resolveColorToken,
resolveFontFamilyToken,
resolveMarkdownStyle,
resolveRenderOptions,
} from "./document.ts";
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`);
}
function findInlineStyle(html: string, tagName: string, text: string): string {
const pattern = new RegExp(
`<${tagName}[^>]*style="([^"]*)"[^>]*>${escapeRegExp(text)}</${tagName}>`,
);
const match = html.match(pattern);
assert.ok(match, `Expected inline style for <${tagName}>${text}</${tagName}>`);
return match![1]!;
}
function useCwd(t: TestContext, cwd: string): void {
const previous = process.cwd();
process.chdir(cwd);
@@ -138,3 +152,23 @@ keep_title: true
assert.equal(explicit.fontSize, "18px");
assert.equal(explicit.keepTitle, false);
});
test("renderMarkdownDocument layers default rules into grace theme before CSS inlining", async () => {
const { html } = await renderMarkdownDocument(
`## Section\n\nParagraph with **bold** text.`,
{ keepTitle: true, theme: "grace" },
);
const h2Style = findInlineStyle(html, "h2", "Section");
assert.match(h2Style, /background: #92617E/);
assert.match(h2Style, /box-shadow: 0 4px 6px rgba\(0, 0, 0, 0\.1\)/);
const pMatch = html.match(/<p[^>]*style="([^"]*)"[^>]*>/);
assert.ok(pMatch, "Expected inline style on <p> tag");
assert.match(pMatch![1]!, /color:/);
const strongPattern = /<strong[^>]*style="([^"]*)"[^>]*>bold<\/strong>/;
const strongMatch = html.match(strongPattern);
assert.ok(strongMatch, "Expected inline style for <strong>bold</strong>");
assert.match(strongMatch![1]!, /font-weight:/);
});
@@ -59,6 +59,17 @@ test("normalizeCssText and normalizeInlineCss replace variables and strip declar
assert.doesNotMatch(normalizedHtml, /var\(--md-primary-color\)/);
});
test("normalizeInlineCss removes quoted custom property values without leaving fragments behind", () => {
const normalizedHtml = normalizeInlineCss(
`<html style="--md-font-family: Menlo, Monaco, 'Courier New', monospace; color: var(--md-primary-color)"></html>`,
DEFAULT_STYLE,
);
assert.match(normalizedHtml, /style=" color: #0F4C81"/);
assert.doesNotMatch(normalizedHtml, /Courier New/);
assert.doesNotMatch(normalizedHtml, /--md-font-family/);
});
test("HTML structure helpers hoist nested lists and remove the first heading", () => {
const nestedList = `<ul><li>Parent<ul><li>Child</li></ul></li></ul>`;
assert.equal(
@@ -100,13 +100,13 @@ export function normalizeCssText(cssText: string, style: StyleConfig = DEFAULT_S
.replace(/var\(--md-accent-color\)/g, style.accentColor)
.replace(/var\(--md-container-bg\)/g, style.containerBg)
.replace(/hsl\(var\(--foreground\)\)/g, "#3f3f3f")
.replace(/--md-primary-color:\s*[^;"']+;?/g, "")
.replace(/--md-font-family:\s*[^;"']+;?/g, "")
.replace(/--md-font-size:\s*[^;"']+;?/g, "")
.replace(/--blockquote-background:\s*[^;"']+;?/g, "")
.replace(/--md-accent-color:\s*[^;"']+;?/g, "")
.replace(/--md-container-bg:\s*[^;"']+;?/g, "")
.replace(/--foreground:\s*[^;"']+;?/g, "");
.replace(/--md-primary-color:\s*[^;]+;?/g, "")
.replace(/--md-font-family:\s*[^;]+;?/g, "")
.replace(/--md-font-size:\s*[^;]+;?/g, "")
.replace(/--blockquote-background:\s*[^;]+;?/g, "")
.replace(/--md-accent-color:\s*[^;]+;?/g, "")
.replace(/--md-container-bg:\s*[^;]+;?/g, "")
.replace(/--foreground:\s*[^;]+;?/g, "");
}
export function normalizeInlineCss(html: string, style: StyleConfig = DEFAULT_STYLE): string {
@@ -6,6 +6,7 @@ import type { ThemeName } from "./types.js";
const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
export const THEME_DIR = path.resolve(SCRIPT_DIR, "themes");
const FALLBACK_THEMES: ThemeName[] = ["default", "grace", "simple"];
const THEMES_EXTENDING_DEFAULT = new Set<ThemeName>(["grace", "simple"]);
function stripOutputScope(cssContent: string): string {
let css = cssContent;
@@ -41,6 +42,7 @@ export function loadThemeCss(theme: ThemeName): {
themeCss: string;
} {
const basePath = path.join(THEME_DIR, "base.css");
const defaultThemePath = path.join(THEME_DIR, "default.css");
const themePath = path.join(THEME_DIR, `${theme}.css`);
if (!fs.existsSync(basePath)) {
@@ -51,9 +53,18 @@ export function loadThemeCss(theme: ThemeName): {
throw new Error(`Missing theme CSS for "${theme}": ${themePath}`);
}
const layeredThemeCss: string[] = [];
if (theme !== "default" && THEMES_EXTENDING_DEFAULT.has(theme)) {
if (!fs.existsSync(defaultThemePath)) {
throw new Error(`Missing default theme CSS: ${defaultThemePath}`);
}
layeredThemeCss.push(fs.readFileSync(defaultThemePath, "utf-8"));
}
layeredThemeCss.push(fs.readFileSync(themePath, "utf-8"));
return {
baseCss: fs.readFileSync(basePath, "utf-8"),
themeCss: fs.readFileSync(themePath, "utf-8"),
themeCss: layeredThemeCss.join("\n"),
};
}
@@ -694,6 +694,10 @@ async function main(): Promise<void> {
}
const creds = loadCredentials(resolved);
for (const skippedSource of creds.skippedSources) {
console.error(`[wechat-api] Skipped incomplete credential source: ${skippedSource}`);
}
console.error(`[wechat-api] Credentials source: ${creds.source}`);
console.error("[wechat-api] Fetching access token...");
const accessToken = await fetchAccessToken(creds.appId, creds.appSecret);
@@ -0,0 +1,139 @@
import assert from "node:assert/strict";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import process from "node:process";
import test, { type TestContext } from "node:test";
import { loadCredentials } from "./wechat-extend-config.ts";
function useCwd(t: TestContext, cwd: string): void {
const previous = process.cwd();
process.chdir(cwd);
t.after(() => {
process.chdir(previous);
});
}
function useHome(t: TestContext, home: string): void {
const previous = process.env.HOME;
process.env.HOME = home;
t.after(() => {
if (previous === undefined) {
delete process.env.HOME;
return;
}
process.env.HOME = previous;
});
}
function useWechatEnv(
t: TestContext,
values: Partial<Record<"WECHAT_APP_ID" | "WECHAT_APP_SECRET", string | undefined>>,
): void {
const previous = {
WECHAT_APP_ID: process.env.WECHAT_APP_ID,
WECHAT_APP_SECRET: process.env.WECHAT_APP_SECRET,
};
if (values.WECHAT_APP_ID === undefined) {
delete process.env.WECHAT_APP_ID;
} else {
process.env.WECHAT_APP_ID = values.WECHAT_APP_ID;
}
if (values.WECHAT_APP_SECRET === undefined) {
delete process.env.WECHAT_APP_SECRET;
} else {
process.env.WECHAT_APP_SECRET = values.WECHAT_APP_SECRET;
}
t.after(() => {
if (previous.WECHAT_APP_ID === undefined) {
delete process.env.WECHAT_APP_ID;
} else {
process.env.WECHAT_APP_ID = previous.WECHAT_APP_ID;
}
if (previous.WECHAT_APP_SECRET === undefined) {
delete process.env.WECHAT_APP_SECRET;
} else {
process.env.WECHAT_APP_SECRET = previous.WECHAT_APP_SECRET;
}
});
}
async function makeTempDir(prefix: string): Promise<string> {
return fs.mkdtemp(path.join(os.tmpdir(), prefix));
}
async function writeEnvFile(root: string, content: string): Promise<void> {
const envPath = path.join(root, ".baoyu-skills", ".env");
await fs.mkdir(path.dirname(envPath), { recursive: true });
await fs.writeFile(envPath, content);
}
test("loadCredentials selects the first complete source without mixing values across sources", async (t) => {
const cwdRoot = await makeTempDir("wechat-creds-cwd-");
const homeRoot = await makeTempDir("wechat-creds-home-");
useCwd(t, cwdRoot);
useHome(t, homeRoot);
useWechatEnv(t, {
WECHAT_APP_ID: undefined,
WECHAT_APP_SECRET: "stale-secret-from-process-env",
});
await writeEnvFile(cwdRoot, "WECHAT_APP_ID=cwd-app-id\nWECHAT_APP_SECRET=cwd-app-secret\n");
await writeEnvFile(homeRoot, "WECHAT_APP_ID=home-app-id\nWECHAT_APP_SECRET=home-app-secret\n");
const credentials = loadCredentials();
assert.equal(credentials.appId, "cwd-app-id");
assert.equal(credentials.appSecret, "cwd-app-secret");
assert.equal(credentials.source, "<cwd>/.baoyu-skills/.env");
assert.deepEqual(credentials.skippedSources, [
"process.env missing WECHAT_APP_ID",
]);
});
test("loadCredentials prefers a complete process.env pair over lower-priority files", async (t) => {
const cwdRoot = await makeTempDir("wechat-creds-cwd-");
const homeRoot = await makeTempDir("wechat-creds-home-");
useCwd(t, cwdRoot);
useHome(t, homeRoot);
useWechatEnv(t, {
WECHAT_APP_ID: "env-app-id",
WECHAT_APP_SECRET: "env-app-secret",
});
await writeEnvFile(cwdRoot, "WECHAT_APP_ID=cwd-app-id\nWECHAT_APP_SECRET=cwd-app-secret\n");
await writeEnvFile(homeRoot, "WECHAT_APP_ID=home-app-id\nWECHAT_APP_SECRET=home-app-secret\n");
const credentials = loadCredentials();
assert.equal(credentials.appId, "env-app-id");
assert.equal(credentials.appSecret, "env-app-secret");
assert.equal(credentials.source, "process.env");
assert.deepEqual(credentials.skippedSources, []);
});
test("loadCredentials reports skipped incomplete sources when no complete pair exists", async (t) => {
const cwdRoot = await makeTempDir("wechat-creds-cwd-");
const homeRoot = await makeTempDir("wechat-creds-home-");
useCwd(t, cwdRoot);
useHome(t, homeRoot);
useWechatEnv(t, {
WECHAT_APP_ID: "env-app-id",
WECHAT_APP_SECRET: undefined,
});
await writeEnvFile(cwdRoot, "WECHAT_APP_SECRET=cwd-app-secret\n");
assert.throws(
() => loadCredentials(),
/Incomplete credential sources skipped:\n- process\.env missing WECHAT_APP_SECRET\n- <cwd>\/\.baoyu-skills\/\.env missing WECHAT_APP_ID/,
);
});
@@ -196,48 +196,116 @@ function aliasToEnvKey(alias: string): string {
return alias.toUpperCase().replace(/-/g, "_");
}
export function loadCredentials(account?: ResolvedAccount): { appId: string; appSecret: string } {
if (account?.app_id && account?.app_secret) {
return { appId: account.app_id, appSecret: account.app_secret };
interface CredentialSource {
name: string;
appIdKey: string;
appSecretKey: string;
appId?: string;
appSecret?: string;
}
export interface LoadedCredentials {
appId: string;
appSecret: string;
source: string;
skippedSources: string[];
}
function normalizeCredentialValue(value?: string): string | undefined {
const trimmed = value?.trim();
return trimmed ? trimmed : undefined;
}
function describeMissingKeys(source: CredentialSource): string {
const missingKeys: string[] = [];
if (!source.appId) missingKeys.push(source.appIdKey);
if (!source.appSecret) missingKeys.push(source.appSecretKey);
return `${source.name} missing ${missingKeys.join(" and ")}`;
}
function buildCredentialSource(
name: string,
values: Record<string, string | undefined>,
appIdKey: string,
appSecretKey: string,
): CredentialSource {
return {
name,
appIdKey,
appSecretKey,
appId: normalizeCredentialValue(values[appIdKey]),
appSecret: normalizeCredentialValue(values[appSecretKey]),
};
}
function resolveCredentialSource(
sources: CredentialSource[],
account?: ResolvedAccount,
): LoadedCredentials {
const skippedSources: string[] = [];
for (const source of sources) {
if (source.appId && source.appSecret) {
return {
appId: source.appId,
appSecret: source.appSecret,
source: source.name,
skippedSources,
};
}
if (source.appId || source.appSecret) {
skippedSources.push(describeMissingKeys(source));
}
}
const hint = account?.alias ? ` (account: ${account.alias})` : "";
const partialHint = skippedSources.length > 0
? `\nIncomplete credential sources skipped:\n- ${skippedSources.join("\n- ")}`
: "";
throw new Error(
`Missing WECHAT_APP_ID or WECHAT_APP_SECRET${hint}.\n` +
"Set via EXTEND.md account config, environment variables, or .baoyu-skills/.env file." +
partialHint
);
}
export function loadCredentials(account?: ResolvedAccount): LoadedCredentials {
const cwdEnvPath = path.join(process.cwd(), ".baoyu-skills", ".env");
const homeEnvPath = path.join(os.homedir(), ".baoyu-skills", ".env");
const cwdEnv = loadEnvFile(cwdEnvPath);
const homeEnv = loadEnvFile(homeEnvPath);
const sources: CredentialSource[] = [];
if (account?.app_id || account?.app_secret) {
sources.push({
name: account.alias ? `EXTEND.md account "${account.alias}"` : "EXTEND.md account config",
appIdKey: "app_id",
appSecretKey: "app_secret",
appId: normalizeCredentialValue(account.app_id),
appSecret: normalizeCredentialValue(account.app_secret),
});
}
const prefix = account?.alias ? `WECHAT_${aliasToEnvKey(account.alias)}_` : "";
let appId = "";
let appSecret = "";
if (prefix) {
appId = process.env[`${prefix}APP_ID`]
|| cwdEnv[`${prefix}APP_ID`]
|| homeEnv[`${prefix}APP_ID`]
|| "";
appSecret = process.env[`${prefix}APP_SECRET`]
|| cwdEnv[`${prefix}APP_SECRET`]
|| homeEnv[`${prefix}APP_SECRET`]
|| "";
}
if (!appId) {
appId = process.env.WECHAT_APP_ID || cwdEnv.WECHAT_APP_ID || homeEnv.WECHAT_APP_ID || "";
}
if (!appSecret) {
appSecret = process.env.WECHAT_APP_SECRET || cwdEnv.WECHAT_APP_SECRET || homeEnv.WECHAT_APP_SECRET || "";
}
if (!appId || !appSecret) {
const hint = account?.alias ? ` (account: ${account.alias})` : "";
throw new Error(
`Missing WECHAT_APP_ID or WECHAT_APP_SECRET${hint}.\n` +
"Set via EXTEND.md account config, environment variables, or .baoyu-skills/.env file."
const prefixedKeyLabel = `${prefix}APP_ID/${prefix}APP_SECRET`;
sources.push(
buildCredentialSource(`process.env (${prefixedKeyLabel})`, process.env, `${prefix}APP_ID`, `${prefix}APP_SECRET`),
buildCredentialSource(`<cwd>/.baoyu-skills/.env (${prefixedKeyLabel})`, cwdEnv, `${prefix}APP_ID`, `${prefix}APP_SECRET`),
buildCredentialSource(`~/.baoyu-skills/.env (${prefixedKeyLabel})`, homeEnv, `${prefix}APP_ID`, `${prefix}APP_SECRET`),
);
}
return { appId, appSecret };
sources.push(
buildCredentialSource("process.env", process.env, "WECHAT_APP_ID", "WECHAT_APP_SECRET"),
buildCredentialSource("<cwd>/.baoyu-skills/.env", cwdEnv, "WECHAT_APP_ID", "WECHAT_APP_SECRET"),
buildCredentialSource("~/.baoyu-skills/.env", homeEnv, "WECHAT_APP_ID", "WECHAT_APP_SECRET"),
);
return resolveCredentialSource(sources, account);
}
export function listAccounts(config: WechatExtendConfig): string[] {
+2
View File
@@ -123,6 +123,8 @@ ${BUN_X} {baseDir}/scripts/weibo-article.ts article.md --cover ./cover.jpg
- Title: 32 characters max (truncated with warning if longer)
- Summary/导语: 44 characters max (auto-regenerated from content if longer)
**Markdown-to-HTML**: Do NOT pass any `--theme` parameter when converting markdown to HTML. Use the default theme (no theme argument).
**Article Workflow**:
1. Opens `https://card.weibo.com/article/v3/editor`
2. Clicks "写文章" button, waits for editor to become editable
@@ -9,12 +9,26 @@ import { COLOR_PRESETS, FONT_FAMILY_MAP } from "./constants.ts";
import {
buildMarkdownDocumentMeta,
formatTimestamp,
renderMarkdownDocument,
resolveColorToken,
resolveFontFamilyToken,
resolveMarkdownStyle,
resolveRenderOptions,
} from "./document.ts";
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`);
}
function findInlineStyle(html: string, tagName: string, text: string): string {
const pattern = new RegExp(
`<${tagName}[^>]*style="([^"]*)"[^>]*>${escapeRegExp(text)}</${tagName}>`,
);
const match = html.match(pattern);
assert.ok(match, `Expected inline style for <${tagName}>${text}</${tagName}>`);
return match![1]!;
}
function useCwd(t: TestContext, cwd: string): void {
const previous = process.cwd();
process.chdir(cwd);
@@ -138,3 +152,23 @@ keep_title: true
assert.equal(explicit.fontSize, "18px");
assert.equal(explicit.keepTitle, false);
});
test("renderMarkdownDocument layers default rules into grace theme before CSS inlining", async () => {
const { html } = await renderMarkdownDocument(
`## Section\n\nParagraph with **bold** text.`,
{ keepTitle: true, theme: "grace" },
);
const h2Style = findInlineStyle(html, "h2", "Section");
assert.match(h2Style, /background: #92617E/);
assert.match(h2Style, /box-shadow: 0 4px 6px rgba\(0, 0, 0, 0\.1\)/);
const pMatch = html.match(/<p[^>]*style="([^"]*)"[^>]*>/);
assert.ok(pMatch, "Expected inline style on <p> tag");
assert.match(pMatch![1]!, /color:/);
const strongPattern = /<strong[^>]*style="([^"]*)"[^>]*>bold<\/strong>/;
const strongMatch = html.match(strongPattern);
assert.ok(strongMatch, "Expected inline style for <strong>bold</strong>");
assert.match(strongMatch![1]!, /font-weight:/);
});
@@ -59,6 +59,17 @@ test("normalizeCssText and normalizeInlineCss replace variables and strip declar
assert.doesNotMatch(normalizedHtml, /var\(--md-primary-color\)/);
});
test("normalizeInlineCss removes quoted custom property values without leaving fragments behind", () => {
const normalizedHtml = normalizeInlineCss(
`<html style="--md-font-family: Menlo, Monaco, 'Courier New', monospace; color: var(--md-primary-color)"></html>`,
DEFAULT_STYLE,
);
assert.match(normalizedHtml, /style=" color: #0F4C81"/);
assert.doesNotMatch(normalizedHtml, /Courier New/);
assert.doesNotMatch(normalizedHtml, /--md-font-family/);
});
test("HTML structure helpers hoist nested lists and remove the first heading", () => {
const nestedList = `<ul><li>Parent<ul><li>Child</li></ul></li></ul>`;
assert.equal(
@@ -100,13 +100,13 @@ export function normalizeCssText(cssText: string, style: StyleConfig = DEFAULT_S
.replace(/var\(--md-accent-color\)/g, style.accentColor)
.replace(/var\(--md-container-bg\)/g, style.containerBg)
.replace(/hsl\(var\(--foreground\)\)/g, "#3f3f3f")
.replace(/--md-primary-color:\s*[^;"']+;?/g, "")
.replace(/--md-font-family:\s*[^;"']+;?/g, "")
.replace(/--md-font-size:\s*[^;"']+;?/g, "")
.replace(/--blockquote-background:\s*[^;"']+;?/g, "")
.replace(/--md-accent-color:\s*[^;"']+;?/g, "")
.replace(/--md-container-bg:\s*[^;"']+;?/g, "")
.replace(/--foreground:\s*[^;"']+;?/g, "");
.replace(/--md-primary-color:\s*[^;]+;?/g, "")
.replace(/--md-font-family:\s*[^;]+;?/g, "")
.replace(/--md-font-size:\s*[^;]+;?/g, "")
.replace(/--blockquote-background:\s*[^;]+;?/g, "")
.replace(/--md-accent-color:\s*[^;]+;?/g, "")
.replace(/--md-container-bg:\s*[^;]+;?/g, "")
.replace(/--foreground:\s*[^;]+;?/g, "");
}
export function normalizeInlineCss(html: string, style: StyleConfig = DEFAULT_STYLE): string {
@@ -6,6 +6,7 @@ import type { ThemeName } from "./types.js";
const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
export const THEME_DIR = path.resolve(SCRIPT_DIR, "themes");
const FALLBACK_THEMES: ThemeName[] = ["default", "grace", "simple"];
const THEMES_EXTENDING_DEFAULT = new Set<ThemeName>(["grace", "simple"]);
function stripOutputScope(cssContent: string): string {
let css = cssContent;
@@ -41,6 +42,7 @@ export function loadThemeCss(theme: ThemeName): {
themeCss: string;
} {
const basePath = path.join(THEME_DIR, "base.css");
const defaultThemePath = path.join(THEME_DIR, "default.css");
const themePath = path.join(THEME_DIR, `${theme}.css`);
if (!fs.existsSync(basePath)) {
@@ -51,9 +53,18 @@ export function loadThemeCss(theme: ThemeName): {
throw new Error(`Missing theme CSS for "${theme}": ${themePath}`);
}
const layeredThemeCss: string[] = [];
if (theme !== "default" && THEMES_EXTENDING_DEFAULT.has(theme)) {
if (!fs.existsSync(defaultThemePath)) {
throw new Error(`Missing default theme CSS: ${defaultThemePath}`);
}
layeredThemeCss.push(fs.readFileSync(defaultThemePath, "utf-8"));
}
layeredThemeCss.push(fs.readFileSync(themePath, "utf-8"));
return {
baseCss: fs.readFileSync(basePath, "utf-8"),
themeCss: fs.readFileSync(themePath, "utf-8"),
themeCss: layeredThemeCss.join("\n"),
};
}
+14 -6
View File
@@ -1,7 +1,7 @@
---
name: baoyu-url-to-markdown
description: Fetch any URL and convert to markdown using Chrome CDP. Saves the rendered HTML snapshot alongside the markdown, uses an upgraded Defuddle pipeline with better web-component handling and YouTube transcript extraction, and automatically falls back to the pre-Defuddle HTML-to-Markdown pipeline when needed. If local browser capture fails entirely, it can fall back to the hosted defuddle.md API. Supports two modes - auto-capture on page load, or wait for user signal (for pages requiring login). Use when user wants to save a webpage as markdown.
version: 1.58.1
version: 1.59.0
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-url-to-markdown
@@ -30,6 +30,9 @@ Fetches any URL via Chrome CDP, saves the rendered HTML snapshot, and converts i
|--------|---------|
| `scripts/main.ts` | CLI entry point for URL fetching |
| `scripts/html-to-markdown.ts` | Markdown conversion entry point and converter selection |
| `scripts/parsers/index.ts` | Unified parser entry: dispatches URL-specific rules before generic converters |
| `scripts/parsers/types.ts` | Unified parser interface shared by all rule files |
| `scripts/parsers/rules/*.ts` | One file per URL rule, for example X status and X article |
| `scripts/defuddle-converter.ts` | Defuddle-based conversion |
| `scripts/legacy-converter.ts` | Pre-Defuddle legacy extraction and markdown conversion |
| `scripts/markdown-conversion-shared.ts` | Shared metadata parsing and markdown document helpers |
@@ -115,10 +118,13 @@ Full reference: [references/config/first-time-setup.md](references/config/first-
## Features
- Chrome CDP for full JavaScript rendering
- URL-specific parser layer for sites that need custom HTML rules before generic extraction
- Two capture modes: auto or wait-for-user
- Save rendered HTML as a sibling `-captured.html` file
- Clean markdown output with metadata
- Upgraded Defuddle-first markdown conversion with automatic fallback to the pre-Defuddle extractor from git history
- X/Twitter pages can use HTML-specific parsing for Tweets and Articles, which improves title/body/media extraction on `x.com` / `twitter.com`
- `archive.ph` / related archive mirrors can restore the original URL from `input[name=q]` and prefer `#CONTENT` before falling back to the page body
- Materializes shadow DOM content before conversion so web-component pages survive serialization better
- YouTube pages can include transcript/caption text in the markdown when YouTube exposes a caption track
- If local browser capture fails completely, can fall back to `defuddle.md/<url>` and still save markdown
@@ -201,14 +207,16 @@ When `--download-media` is enabled:
Conversion order:
1. Try Defuddle first
2. For rich pages such as YouTube, prefer Defuddle's extractor-specific output (including transcripts when available) instead of replacing it with the legacy pipeline
3. If Defuddle throws, cannot load, returns obviously incomplete markdown, or captures lower-quality content than the legacy pipeline, automatically fall back to the pre-Defuddle extractor
4. If the entire local browser capture flow fails before markdown can be produced, try the hosted `https://defuddle.md/<url>` API and save its markdown output directly
5. The legacy fallback path uses the older Readability/selector/Next.js-data based HTML-to-Markdown implementation recovered from git history
1. Try the URL-specific parser layer first when a site rule matches
2. If no specialized parser matches, try Defuddle
3. For rich pages such as YouTube, prefer Defuddle's extractor-specific output (including transcripts when available) instead of replacing it with the legacy pipeline
4. If Defuddle throws, cannot load, returns obviously incomplete markdown, or captures lower-quality content than the legacy pipeline, automatically fall back to the pre-Defuddle extractor
5. If the entire local browser capture flow fails before markdown can be produced, try the hosted `https://defuddle.md/<url>` API and save its markdown output directly
6. The legacy fallback path uses the older Readability/selector/Next.js-data based HTML-to-Markdown implementation recovered from git history
CLI output will show:
- `Converter: parser:...` when a URL-specific parser succeeded
- `Converter: defuddle` when Defuddle succeeds
- `Converter: legacy:...` plus `Fallback used: ...` when fallback was needed
- `Converter: defuddle-api` when local browser capture failed and the hosted API was used instead
@@ -12,6 +12,7 @@ import {
scoreMarkdownQuality,
shouldCompareWithLegacy,
} from "./legacy-converter.js";
import { tryUrlRuleParsers } from "./parsers/index.js";
export type { ConversionResult, PageMetadata };
export { createMarkdownDocument, formatMetadataYaml };
@@ -105,6 +106,11 @@ export async function extractContent(html: string, url: string): Promise<Convers
const capturedAt = new Date().toISOString();
const baseMetadata = extractMetadataFromHtml(html, url, capturedAt);
const specializedResult = tryUrlRuleParsers(html, url, baseMetadata);
if (specializedResult) {
return specializedResult;
}
const defuddleResult = await tryDefuddleConversion(html, url, baseMetadata);
if (defuddleResult.ok) {
if (shouldPreferDefuddle(defuddleResult.result)) {
@@ -521,14 +521,18 @@ turndown.addRule("collapseFigure", {
turndown.addRule("dropInvisibleAnchors", {
filter(node) {
return node.nodeName === "A" && !(node as Element).textContent?.trim();
return (
node.nodeName === "A" &&
!(node as Element).textContent?.trim() &&
!(node as Element).querySelector("img, video, picture, source")
);
},
replacement() {
return "";
},
});
function convertHtmlToMarkdown(html: string): string {
export function convertHtmlFragmentToMarkdown(html: string): string {
if (!html || !html.trim()) return "";
try {
@@ -609,7 +613,7 @@ export function shouldCompareWithLegacy(markdown: string): boolean {
export function convertWithLegacyExtractor(html: string, baseMetadata: PageMetadata): ConversionResult {
const extracted = extractFromHtml(html);
let markdown = extracted?.html ? convertHtmlToMarkdown(extracted.html) : "";
let markdown = extracted?.html ? convertHtmlFragmentToMarkdown(extracted.html) : "";
if (!markdown.trim()) {
markdown = extracted?.textContent?.trim() || fallbackPlainText(html);
}
+70 -16
View File
@@ -52,15 +52,71 @@ function parseArgs(argv: string[]): Args {
return args;
}
function generateSlug(title: string, url: string): string {
const text = title || new URL(url).pathname.replace(/\//g, "-");
return text
const SLUG_STOP_WORDS = new Set([
"the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
"have", "has", "had", "do", "does", "did", "will", "would", "shall",
"should", "may", "might", "must", "can", "could", "to", "of", "in",
"for", "on", "with", "at", "by", "from", "as", "into", "through",
"during", "before", "after", "above", "below", "between", "out",
"off", "over", "under", "again", "further", "then", "once", "here",
"there", "when", "where", "why", "how", "all", "both", "each",
"few", "more", "most", "other", "some", "such", "no", "nor", "not",
"only", "own", "same", "so", "than", "too", "very", "just", "but",
"and", "or", "if", "this", "that", "these", "those", "it", "its",
"http", "https", "www", "com", "org", "net", "post", "article",
]);
function extractSlugFromContent(content: string): string | null {
const body = content.replace(/^---\n[\s\S]*?\n---\n?/, "").slice(0, 1000);
const words = body
.replace(/[^\w\s-]/g, " ")
.split(/\s+/)
.filter((w) => /^[a-zA-Z]/.test(w) && w.length >= 2 && !SLUG_STOP_WORDS.has(w.toLowerCase()))
.map((w) => w.toLowerCase());
const unique: string[] = [];
const seen = new Set<string>();
for (const w of words) {
if (!seen.has(w)) {
seen.add(w);
unique.push(w);
if (unique.length >= 6) break;
}
}
return unique.length >= 2 ? unique.join("-").slice(0, 50) : null;
}
function generateSlug(title: string, url: string, content?: string): string {
const asciiWords = title
.replace(/[^\w\s]/g, " ")
.split(/\s+/)
.filter((w) => /[a-zA-Z]/.test(w) && w.length >= 2 && !SLUG_STOP_WORDS.has(w.toLowerCase()))
.map((w) => w.toLowerCase());
if (asciiWords.length >= 2) {
return asciiWords.slice(0, 6).join("-").slice(0, 50);
}
if (content) {
const contentSlug = extractSlugFromContent(content);
if (contentSlug) return contentSlug;
}
const GENERIC_PATH_SEGMENTS = new Set(["status", "article", "post", "posts", "p", "blog", "news", "articles"]);
const parsed = new URL(url);
const pathSlug = parsed.pathname
.split("/")
.filter((s) => s.length > 0 && !/^\d{10,}$/.test(s) && !GENERIC_PATH_SEGMENTS.has(s.toLowerCase()))
.join("-")
.toLowerCase()
.replace(/[^\w\s-]/g, "")
.replace(/\s+/g, "-")
.replace(/[^\w-]/g, "-")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "")
.slice(0, 50) || "page";
.slice(0, 40);
const prefix = asciiWords.slice(0, 2).join("-");
const combined = prefix ? `${prefix}-${pathSlug}` : pathSlug;
return combined.slice(0, 50) || "page";
}
function formatTimestamp(): string {
@@ -124,18 +180,18 @@ async function fetchDefuddleApiMarkdown(targetUrl: string): Promise<{ markdown:
};
}
async function generateOutputPath(url: string, title: string, outputDir?: string): Promise<string> {
async function generateOutputPath(url: string, title: string, outputDir?: string, content?: string): Promise<string> {
const domain = new URL(url).hostname.replace(/^www\./, "");
const slug = generateSlug(title, url);
const slug = generateSlug(title, url, content);
const dataDir = outputDir ? path.resolve(outputDir) : resolveUrlToMarkdownDataDir();
const basePath = path.join(dataDir, domain, `${slug}.md`);
const basePath = path.join(dataDir, domain, slug, `${slug}.md`);
if (!(await fileExists(basePath))) {
return basePath;
}
const timestampSlug = `${slug}-${formatTimestamp()}`;
return path.join(dataDir, domain, `${timestampSlug}.md`);
return path.join(dataDir, domain, timestampSlug, `${timestampSlug}.md`);
}
async function waitForUserSignal(): Promise<void> {
@@ -249,13 +305,12 @@ async function main(): Promise<void> {
try {
const result = await captureUrl(args);
outputPath = args.output || await generateOutputPath(args.url, result.metadata.title, args.outputDir);
document = createMarkdownDocument(result);
outputPath = args.output || await generateOutputPath(args.url, result.metadata.title, args.outputDir, document);
const outputDir = path.dirname(outputPath);
htmlSnapshotPath = deriveHtmlSnapshotPath(outputPath);
await mkdir(outputDir, { recursive: true });
await writeFile(htmlSnapshotPath, result.rawHtml, "utf-8");
document = createMarkdownDocument(result);
conversionMethod = result.conversionMethod;
fallbackReason = result.fallbackReason;
} catch (error) {
@@ -265,10 +320,9 @@ async function main(): Promise<void> {
try {
const remoteResult = await fetchDefuddleApiMarkdown(args.url);
outputPath = args.output || await generateOutputPath(args.url, remoteResult.title, args.outputDir);
await mkdir(path.dirname(outputPath), { recursive: true });
document = remoteResult.markdown;
outputPath = args.output || await generateOutputPath(args.url, remoteResult.title, args.outputDir, document);
await mkdir(path.dirname(outputPath), { recursive: true });
conversionMethod = "defuddle-api";
fallbackReason = `Local browser capture failed: ${primaryError}`;
} catch (remoteError) {
@@ -300,6 +300,24 @@ export function createMarkdownDocument(result: ConversionResult): string {
const escapedTitle = result.metadata.title.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const titleRegex = new RegExp(`^#\\s+${escapedTitle}\\s*(\\n|$)`, "i");
const hasTitle = titleRegex.test(result.markdown.trimStart());
const title = result.metadata.title && !hasTitle ? `\n\n# ${result.metadata.title}\n\n` : "\n\n";
const firstMeaningfulLine = result.markdown
.replace(/\r\n/g, "\n")
.split("\n")
.map((line) => line.trim())
.find((line) => line && !/^!?\[[^\]]*\]\([^)]+\)$/.test(line))
?.replace(/^>\s*/, "")
?.replace(/^#+\s+/, "")
?.trim();
const comparableTitle = result.metadata.title.toLowerCase().replace(/(?:\.{3}|…)\s*$/, "");
const comparableFirstLine = firstMeaningfulLine?.toLowerCase() ?? "";
const titleRepeatsContent =
comparableTitle !== "" &&
comparableFirstLine !== "" &&
(comparableFirstLine === comparableTitle ||
comparableFirstLine.startsWith(comparableTitle) ||
comparableTitle.startsWith(comparableFirstLine));
const title = result.metadata.title && !hasTitle && !titleRepeatsContent
? `\n\n# ${result.metadata.title}\n\n`
: "\n\n";
return yaml + title + result.markdown;
}
@@ -0,0 +1,206 @@
import assert from "node:assert/strict";
import test from "node:test";
import {
createMarkdownDocument,
extractMetadataFromHtml,
} from "../markdown-conversion-shared.js";
import { tryUrlRuleParsers } from "./index.js";
const CAPTURED_AT = "2026-03-22T06:00:00.000Z";
const ARTICLE_HTML = `<!doctype html>
<html lang="zh-CN">
<body>
<div data-testid="twitterArticleReadView">
<a href="/dotey/article/2035141635713941927/media/1">
<div data-testid="tweetPhoto">
<img src="https://pbs.twimg.com/media/article-cover.jpg" alt="Image">
</div>
</a>
<div data-testid="twitter-article-title">Karpathy"写代码"</div>
<div data-testid="User-Name">
<a href="/dotey"> Verified account</a>
<a href="/dotey">@dotey</a>
<time datetime="2026-03-20T23:49:11.000Z">Mar 20</time>
</div>
<div data-testid="twitterArticleRichTextView">
<p>Andrej Karpathy 2024 12 </p>
<a href="/dotey/article/2035141635713941927/media/2">
<div>
<div>
<div data-testid="tweetPhoto">
<img src="https://pbs.twimg.com/media/article-inline.jpg" alt="Image">
</div>
</div>
</div>
</a>
<h2></h2>
<ul>
<li> GPU Token </li>
</ul>
<blockquote>
<p></p>
</blockquote>
</div>
</div>
</body>
</html>`;
const STATUS_HTML = `<!doctype html>
<html lang="en">
<body>
<article data-testid="tweet">
<div data-testid="User-Name">
<a href="/dotey"> Verified account</a>
<a href="/dotey">@dotey</a>
<time datetime="2026-03-22T05:33:00.000Z">Mar 22</time>
</div>
<div data-testid="tweetText">
<span> Codex </span>
</div>
<div data-testid="tweetPhoto">
<img src="https://pbs.twimg.com/media/tweet-main.jpg" alt="Image">
</div>
<div data-testid="User-Name">
<a href="/mattshumer_">Matt Shumer Verified account</a>
<a href="/mattshumer_">@mattshumer_</a>
<time datetime="2026-03-17T00:00:00.000Z">Mar 17</time>
</div>
<div data-testid="tweetText">
<span>Add this to your Codex custom instructions for a way better experience.</span>
</div>
</article>
</body>
</html>`;
const ARCHIVE_HTML = `<!doctype html>
<html>
<head>
<title>archive.ph</title>
</head>
<body>
<form>
<input
type="text"
name="q"
value="https://www.newscientist.com/article/2520204-major-leap-towards-reanimation-after-death-as-mammals-brain-preserved/"
>
</form>
<div id="HEADER">
Archive shell text that should be ignored when CONTENT exists.
</div>
<div id="CONTENT">
<h1>Major leap towards reanimation after death as mammal brain preserved</h1>
<p>
Researchers say the preserved structure and activity markers suggest a significant step
forward in keeping delicate brain tissue viable after clinical death.
</p>
<p>
The archive wrapper should not take precedence over the actual article body when the
CONTENT container is available for parsing.
</p>
<img src="https://cdn.example.com/brain.jpg" alt="Brain tissue">
</div>
</body>
</html>`;
const ARCHIVE_FALLBACK_HTML = `<!doctype html>
<html>
<head>
<title>archive.ph</title>
</head>
<body>
<input type="text" name="q" value="https://example.com/fallback-story">
<main>
<h1>Fallback body parsing still works</h1>
<p>
When CONTENT is absent, the parser should fall back to the body content instead of
returning null or keeping the archive wrapper as the final URL.
</p>
<p>
This ensures archived pages with slightly different layouts still produce usable markdown.
</p>
</main>
</body>
</html>`;
function parse(html: string, url: string) {
const baseMetadata = extractMetadataFromHtml(html, url, CAPTURED_AT);
return tryUrlRuleParsers(html, url, baseMetadata);
}
test("parses archive.ph pages from CONTENT and restores the original URL", () => {
const result = parse(ARCHIVE_HTML, "https://archive.ph/SMcX5");
assert.ok(result);
assert.equal(result.conversionMethod, "parser:archive-ph");
assert.equal(
result.metadata.url,
"https://www.newscientist.com/article/2520204-major-leap-towards-reanimation-after-death-as-mammals-brain-preserved/"
);
assert.equal(
result.metadata.title,
"Major leap towards reanimation after death as mammal brain preserved"
);
assert.equal(result.metadata.coverImage, "https://cdn.example.com/brain.jpg");
assert.ok(result.markdown.includes("Researchers say the preserved structure"));
assert.ok(result.markdown.includes("![Brain tissue](https://cdn.example.com/brain.jpg)"));
assert.ok(!result.markdown.includes("Archive shell text that should be ignored"));
});
test("falls back to body when archive.ph CONTENT is missing", () => {
const result = parse(ARCHIVE_FALLBACK_HTML, "https://archive.ph/fallback");
assert.ok(result);
assert.equal(result.conversionMethod, "parser:archive-ph");
assert.equal(result.metadata.url, "https://example.com/fallback-story");
assert.equal(result.metadata.title, "Fallback body parsing still works");
assert.ok(result.markdown.includes("When CONTENT is absent"));
});
test("parses X article pages from HTML", () => {
const result = parse(
ARTICLE_HTML,
"https://x.com/dotey/article/2035141635713941927"
);
assert.ok(result);
assert.equal(result.conversionMethod, "parser:x-article");
assert.equal(result.metadata.title, "Karpathy\"写代码\"已经不是对的动词了");
assert.equal(result.metadata.author, "宝玉 (@dotey)");
assert.equal(result.metadata.coverImage, "https://pbs.twimg.com/media/article-cover.jpg");
assert.equal(result.metadata.published, "2026-03-20T23:49:11.000Z");
assert.equal(result.metadata.language, "zh");
assert.ok(result.markdown.includes("## 要点速览"));
assert.ok(
result.markdown.includes(
"[![](https://pbs.twimg.com/media/article-inline.jpg)](/dotey/article/2035141635713941927/media/2)"
)
);
assert.ok(result.markdown.includes("写代码已经不是对的动词了。"));
const document = createMarkdownDocument(result);
assert.ok(document.includes("# Karpathy\"写代码\"已经不是对的动词了"));
});
test("parses X status pages from HTML without duplicating the title heading", () => {
const result = parse(
STATUS_HTML,
"https://x.com/dotey/status/2035590649081196710"
);
assert.ok(result);
assert.equal(result.conversionMethod, "parser:x-status");
assert.equal(result.metadata.author, "宝玉 (@dotey)");
assert.equal(result.metadata.coverImage, "https://pbs.twimg.com/media/tweet-main.jpg");
assert.equal(result.metadata.language, "zh");
assert.ok(result.markdown.includes("转译:把下面这段加到你的 Codex 自定义指令里"));
assert.ok(result.markdown.includes("> Quote from Matt Shumer (@mattshumer_)"));
assert.ok(result.markdown.includes("!["));
const document = createMarkdownDocument(result);
assert.ok(
!document.includes("\n\n# 转译:把下面这段加到你的 Codex 自定义指令里,体验会好太多:\n\n")
);
});
@@ -0,0 +1,47 @@
import {
isMarkdownUsable,
normalizeMarkdown,
parseDocument,
type ConversionResult,
type PageMetadata,
} from "../markdown-conversion-shared.js";
import { URL_RULE_PARSERS } from "./rules/index.js";
import type { UrlRuleParserContext } from "./types.js";
export type { UrlRuleParser, UrlRuleParserContext } from "./types.js";
export function tryUrlRuleParsers(
html: string,
url: string,
baseMetadata: PageMetadata
): ConversionResult | null {
const document = parseDocument(html);
const context: UrlRuleParserContext = {
html,
url,
document,
baseMetadata,
};
for (const parser of URL_RULE_PARSERS) {
if (!parser.supports(context)) continue;
try {
const result = parser.parse(context);
if (!result) continue;
const markdown = normalizeMarkdown(result.markdown);
if (!isMarkdownUsable(markdown, html)) continue;
return {
...result,
markdown,
};
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.warn(`[url-to-markdown] parser ${parser.id} failed: ${message}`);
}
}
return null;
}
@@ -0,0 +1,97 @@
import { convertHtmlFragmentToMarkdown } from "../../legacy-converter.js";
import {
normalizeMarkdown,
pickString,
type ConversionResult,
} from "../../markdown-conversion-shared.js";
import type { UrlRuleParser, UrlRuleParserContext } from "../types.js";
const ARCHIVE_HOSTS = new Set([
"archive.ph",
"archive.is",
"archive.today",
"archive.md",
"archive.vn",
"archive.li",
"archive.fo",
]);
function isArchiveHost(url: string): boolean {
try {
return ARCHIVE_HOSTS.has(new URL(url).hostname.toLowerCase());
} catch {
return false;
}
}
function readOriginalUrl(document: Document): string | undefined {
const value = document.querySelector("input[name='q']")?.getAttribute("value")?.trim();
if (!value) return undefined;
try {
return new URL(value).href;
} catch {
return undefined;
}
}
function summarize(text: string, maxLength: number): string | undefined {
const normalized = text.replace(/\s+/g, " ").trim();
if (!normalized) return undefined;
if (normalized.length <= maxLength) return normalized;
return `${normalized.slice(0, Math.max(0, maxLength - 1)).trimEnd()}`;
}
function pickContentRoot(document: Document): Element | null {
return (
document.querySelector("#CONTENT") ??
document.querySelector("#content") ??
document.body
);
}
function pickContentTitle(root: Element, fallbackTitle: string): string {
const contentTitle = pickString(
root.querySelector("h1")?.textContent,
root.querySelector("[itemprop='headline']")?.textContent,
root.querySelector("article h2")?.textContent
);
if (contentTitle) return contentTitle;
if (fallbackTitle && !/^archive\./i.test(fallbackTitle.trim())) return fallbackTitle;
return "";
}
function parseArchivePage(context: UrlRuleParserContext): ConversionResult | null {
const root = pickContentRoot(context.document);
if (!root) return null;
const markdown = normalizeMarkdown(convertHtmlFragmentToMarkdown(root.innerHTML));
if (!markdown) return null;
const originalUrl = readOriginalUrl(context.document) ?? context.baseMetadata.url;
const bodyText = root.textContent?.replace(/\s+/g, " ").trim() ?? "";
const published = root.querySelector("time[datetime]")?.getAttribute("datetime") ?? undefined;
const coverImage = root.querySelector("img[src]")?.getAttribute("src") ?? undefined;
return {
metadata: {
...context.baseMetadata,
url: originalUrl,
title: pickContentTitle(root, context.baseMetadata.title),
description: summarize(bodyText, 220) ?? context.baseMetadata.description,
published: pickString(published, context.baseMetadata.published) ?? undefined,
coverImage: pickString(coverImage, context.baseMetadata.coverImage) ?? undefined,
},
markdown,
rawHtml: context.html,
conversionMethod: "parser:archive-ph",
};
}
export const archivePhRuleParser: UrlRuleParser = {
id: "archive-ph",
supports(context) {
return isArchiveHost(context.url);
},
parse: parseArchivePage,
};
@@ -0,0 +1,10 @@
import { archivePhRuleParser } from "./archive-ph.js";
import { xArticleRuleParser } from "./x-article.js";
import { xStatusRuleParser } from "./x-status.js";
import type { UrlRuleParser } from "../types.js";
export const URL_RULE_PARSERS: UrlRuleParser[] = [
archivePhRuleParser,
xArticleRuleParser,
xStatusRuleParser,
];
@@ -0,0 +1,137 @@
import {
normalizeMarkdown,
pickString,
type ConversionResult,
} from "../../markdown-conversion-shared.js";
import type { UrlRuleParser, UrlRuleParserContext } from "../types.js";
import {
cleanText,
collectMediaMarkdown,
convertXRichTextElementToMarkdown,
extractPublishedForCurrentUrl,
inferLanguage,
isXArticlePath,
isXHost,
normalizeXMarkdown,
parseUrl,
pickFirstValidLinkText,
sanitizeCoverImage,
summarizeText,
} from "./x-shared.js";
function collectArticleMarkdown(root: Element): { markdown: string; mediaUrls: string[] } {
const parts: string[] = [];
const seenMedia = new Set<string>();
const mediaUrls: string[] = [];
function pushPart(value: string): void {
const normalized = normalizeMarkdown(value);
if (!normalized) return;
parts.push(normalized);
}
function walk(node: Element): void {
const testId = node.getAttribute("data-testid");
if (testId === "twitterArticleRichTextView" || testId === "longformRichTextComponent") {
const bodyMedia = collectMediaMarkdown(node, seenMedia);
mediaUrls.push(...bodyMedia.urls.filter((url) => !mediaUrls.includes(url)));
pushPart(convertXRichTextElementToMarkdown(node));
return;
}
if (testId === "tweetPhoto") {
const media = collectMediaMarkdown(node, seenMedia);
mediaUrls.push(...media.urls.filter((url) => !mediaUrls.includes(url)));
for (const line of media.lines) pushPart(line);
return;
}
if (
testId === "twitter-article-title" ||
testId === "User-Name" ||
testId === "Tweet-User-Avatar" ||
testId === "reply" ||
testId === "retweet" ||
testId === "like" ||
testId === "bookmark" ||
testId === "caret" ||
testId === "app-text-transition-container"
) {
return;
}
if (node.tagName === "TIME" || node.tagName === "BUTTON") {
return;
}
for (const child of Array.from(node.children)) {
walk(child);
}
}
for (const child of Array.from(root.children)) {
walk(child);
}
return {
markdown: normalizeXMarkdown(parts.join("\n\n")),
mediaUrls,
};
}
function parseXArticle(context: UrlRuleParserContext): ConversionResult | null {
const articleRoot = context.document.querySelector("[data-testid='twitterArticleReadView']") as Element | null;
if (!articleRoot) return null;
const title = cleanText(
context.document.querySelector("[data-testid='twitter-article-title']")?.textContent
);
const identity = pickFirstValidLinkText(
context.document.querySelector("[data-testid='User-Name']")
);
const published = extractPublishedForCurrentUrl(articleRoot, context.url);
const { markdown, mediaUrls } = collectArticleMarkdown(articleRoot);
if (!markdown) return null;
const bodyText = cleanText(
context.document.querySelector("[data-testid='twitterArticleRichTextView']")?.textContent ??
context.document.querySelector("[data-testid='longformRichTextComponent']")?.textContent
);
return {
metadata: {
...context.baseMetadata,
title: pickString(title, context.baseMetadata.title) ?? "",
description: summarizeText(bodyText, 220) ?? context.baseMetadata.description,
author: pickString(identity.author, context.baseMetadata.author) ?? undefined,
published: pickString(published, context.baseMetadata.published) ?? undefined,
coverImage: sanitizeCoverImage(mediaUrls[0], context.baseMetadata.coverImage),
language: inferLanguage(bodyText, context.baseMetadata.language),
},
markdown,
rawHtml: context.html,
conversionMethod: "parser:x-article",
};
}
export const xArticleRuleParser: UrlRuleParser = {
id: "x-article",
supports(context) {
const parsed = parseUrl(context.url);
if (!parsed || !isXHost(parsed.hostname)) {
return false;
}
return (
isXArticlePath(parsed.pathname) ||
Boolean(
context.document.querySelector("[data-testid='twitterArticleReadView']") ||
context.document.querySelector("[data-testid='twitterArticleRichTextView']")
)
);
},
parse(context) {
return parseXArticle(context);
},
};
@@ -0,0 +1,249 @@
import { convertHtmlFragmentToMarkdown } from "../../legacy-converter.js";
import { normalizeMarkdown } from "../../markdown-conversion-shared.js";
export const DEFAULT_X_OG_IMAGE = "https://abs.twimg.com/rweb/ssr/default/v2/og/image.png";
export type MediaResult = {
lines: string[];
urls: string[];
};
export function isXHost(hostname: string): boolean {
const normalized = hostname.toLowerCase();
return (
normalized === "x.com" ||
normalized === "twitter.com" ||
normalized.endsWith(".x.com") ||
normalized.endsWith(".twitter.com")
);
}
export function parseUrl(input: string): URL | null {
try {
return new URL(input);
} catch {
return null;
}
}
export function isXStatusPath(pathname: string): boolean {
return /^\/[^/]+\/status(?:es)?\/\d+$/i.test(pathname) || /^\/i\/web\/status\/\d+$/i.test(pathname);
}
export function isXArticlePath(pathname: string): boolean {
return /^\/[^/]+\/article\/\d+$/i.test(pathname) || /^\/(?:i\/)?article\/\d+$/i.test(pathname);
}
export function cleanText(value: string | null | undefined): string {
return (value ?? "").replace(/\s+/g, " ").trim();
}
export function cleanUserLabel(value: string | null | undefined): string {
return cleanText(value).replace(/\bVerified account\b/gi, "").replace(/\s{2,}/g, " ").trim();
}
export function escapeMarkdownAlt(text: string): string {
return text.replace(/[\[\]]/g, "\\$&");
}
export function normalizeAlt(text: string | null | undefined): string {
const cleaned = cleanText(text);
if (!cleaned || /^(image|photo)$/i.test(cleaned)) return "";
return escapeMarkdownAlt(cleaned);
}
export function summarizeText(text: string, maxLength: number): string | undefined {
const normalized = cleanText(text);
if (!normalized) return undefined;
return normalized.length > maxLength
? `${normalized.slice(0, maxLength - 3)}...`
: normalized;
}
export function buildTweetTitle(text: string, fallback: string): string {
return summarizeText(text, 80) ?? fallback;
}
export function normalizeXMarkdown(markdown: string): string {
return normalizeMarkdown(markdown.replace(/^(#{1,6})\s*\n+([^\n])/gm, "$1 $2"));
}
export function inferLanguage(text: string, fallback?: string): string | undefined {
const normalized = cleanText(text);
if (!normalized) return fallback;
const han = (normalized.match(/\p{Script=Han}/gu) || []).length;
const hiragana = (normalized.match(/\p{Script=Hiragana}/gu) || []).length;
const katakana = (normalized.match(/\p{Script=Katakana}/gu) || []).length;
const hangul = (normalized.match(/\p{Script=Hangul}/gu) || []).length;
if (hangul >= 8) return "ko";
if (hiragana + katakana >= 8) return "ja";
if (han >= 16) return "zh";
return fallback;
}
export function buildQuoteMarkdown(markdown: string, author?: string): string {
const normalized = normalizeMarkdown(markdown);
if (!normalized) return "";
const lines = normalized.split("\n");
const prefixed = lines.map((line) => (line ? `> ${line}` : ">")).join("\n");
const header = author ? `> Quote from ${author}` : "> Quote";
return `${header}\n${prefixed}`;
}
export function pickFirstValidLinkText(userNameEl: Element | null | undefined): {
name?: string;
username?: string;
author?: string;
} {
if (!userNameEl) return {};
const linkTexts = Array.from(userNameEl.querySelectorAll("a[href]"))
.map((link) => cleanUserLabel(link.textContent))
.filter(Boolean);
let username = linkTexts.find((text) => text.startsWith("@"));
let name = linkTexts.find((text) => !text.startsWith("@") && !/^(promote|more)$/i.test(text));
if (!username || !name) {
const text = cleanUserLabel(userNameEl.textContent);
const fallbackMatch = text.match(/^(.*?)\s*(@[A-Za-z0-9_]+)(?:\s*·.*)?$/);
if (fallbackMatch) {
name = name ?? cleanText(fallbackMatch[1]);
username = username ?? cleanText(fallbackMatch[2]);
}
}
const author = name && username ? `${name} (${username})` : username ?? name;
return { name, username, author };
}
export function extractPublishedForCurrentUrl(root: ParentNode, url: string): string | undefined {
const parsed = parseUrl(url);
if (!parsed) return undefined;
const currentPath = parsed.pathname.toLowerCase();
for (const timeElement of root.querySelectorAll("a[href] time[datetime]")) {
const href = timeElement.closest("a")?.getAttribute("href");
const hrefUrl = href ? parseUrl(href.startsWith("http") ? href : `${parsed.origin}${href}`) : null;
if (hrefUrl?.pathname.toLowerCase() === currentPath) {
return timeElement.getAttribute("datetime") ?? undefined;
}
}
return root.querySelector("time[datetime]")?.getAttribute("datetime") ?? undefined;
}
export function collectMediaMarkdown(root: ParentNode, seen: Set<string>): MediaResult {
const lines: string[] = [];
const urls: string[] = [];
const rootElement = root as Element & {
getAttribute?: (name: string) => string | null;
};
const photoNodes = [
...(typeof rootElement.getAttribute === "function" &&
rootElement.getAttribute("data-testid") === "tweetPhoto"
? [rootElement]
: []),
...Array.from(root.querySelectorAll("[data-testid='tweetPhoto']")),
];
for (const node of photoNodes) {
const img = node.querySelector("img");
const imageUrl = img?.getAttribute("src");
if (imageUrl && !seen.has(imageUrl)) {
seen.add(imageUrl);
urls.push(imageUrl);
lines.push(`![${normalizeAlt(img?.getAttribute("alt"))}](${imageUrl})`);
}
const video = node.querySelector("video");
const posterUrl = video?.getAttribute("poster");
if (posterUrl && !seen.has(posterUrl)) {
seen.add(posterUrl);
urls.push(posterUrl);
lines.push(`![video](${posterUrl})`);
}
const videoUrl = video?.getAttribute("src") ?? video?.querySelector("source")?.getAttribute("src");
if (videoUrl && !seen.has(videoUrl)) {
seen.add(videoUrl);
urls.push(videoUrl);
lines.push(`[video](${videoUrl})`);
}
}
return { lines, urls };
}
export function materializeTweetPhotoNodes(root: Element): void {
for (const photo of Array.from(root.querySelectorAll("[data-testid='tweetPhoto']"))) {
const document = photo.ownerDocument;
const container = document.createElement("span");
const img = photo.querySelector("img");
const imageUrl = img?.getAttribute("src");
if (imageUrl) {
const image = document.createElement("img");
image.setAttribute("src", imageUrl);
const alt = normalizeAlt(img?.getAttribute("alt"));
if (alt) {
image.setAttribute("alt", alt);
}
container.appendChild(image);
}
const video = photo.querySelector("video");
const posterUrl = video?.getAttribute("poster");
if (posterUrl) {
const poster = document.createElement("img");
poster.setAttribute("src", posterUrl);
poster.setAttribute("alt", "video");
container.appendChild(poster);
}
const videoUrl = video?.getAttribute("src") ?? video?.querySelector("source")?.getAttribute("src");
if (videoUrl) {
if (container.childNodes.length > 0) {
container.appendChild(document.createTextNode(" "));
}
const link = document.createElement("a");
link.setAttribute("href", videoUrl);
link.textContent = "video";
container.appendChild(link);
}
if (container.childNodes.length === 0) {
photo.remove();
continue;
}
photo.replaceWith(container);
}
}
function collapseLinkedMediaContainers(root: Element): void {
for (const anchor of Array.from(root.querySelectorAll("a[href]"))) {
const images = Array.from(anchor.querySelectorAll("img"));
if (images.length !== 1) continue;
if (cleanText(anchor.textContent)) continue;
const image = images[0].cloneNode(true);
anchor.replaceChildren(image);
}
}
export function convertXRichTextElementToMarkdown(node: Element): string {
const clone = node.cloneNode(true) as Element;
materializeTweetPhotoNodes(clone);
collapseLinkedMediaContainers(clone);
return normalizeXMarkdown(convertHtmlFragmentToMarkdown(clone.innerHTML));
}
export function sanitizeCoverImage(primary?: string, fallback?: string): string | undefined {
if (primary) return primary;
if (!fallback || fallback === DEFAULT_X_OG_IMAGE) return undefined;
return fallback;
}
@@ -0,0 +1,82 @@
import type { ConversionResult } from "../../markdown-conversion-shared.js";
import type { UrlRuleParser, UrlRuleParserContext } from "../types.js";
import {
buildQuoteMarkdown,
buildTweetTitle,
cleanText,
collectMediaMarkdown,
convertXRichTextElementToMarkdown,
extractPublishedForCurrentUrl,
inferLanguage,
isXHost,
isXStatusPath,
normalizeXMarkdown,
parseUrl,
pickFirstValidLinkText,
sanitizeCoverImage,
summarizeText,
} from "./x-shared.js";
function parseXStatus(context: UrlRuleParserContext): ConversionResult | null {
const article = context.document.querySelector("article[data-testid='tweet'], article") as Element | null;
if (!article) return null;
const tweetTextElements = Array.from(article.querySelectorAll("[data-testid='tweetText']")) as Element[];
if (tweetTextElements.length === 0) return null;
const userNameElements = Array.from(article.querySelectorAll("[data-testid='User-Name']")) as Element[];
const mainTextElement = tweetTextElements[0];
const mainIdentity = pickFirstValidLinkText(userNameElements[0]);
const published = extractPublishedForCurrentUrl(article, context.url);
const mainMarkdown = normalizeXMarkdown(convertXRichTextElementToMarkdown(mainTextElement));
if (!mainMarkdown) return null;
const parts = [mainMarkdown];
const quotedTextElements = tweetTextElements.slice(1);
const quotedUserNameElements = userNameElements.slice(1);
quotedTextElements.forEach((element, index) => {
const quoteMarkdown = normalizeXMarkdown(convertXRichTextElementToMarkdown(element));
if (!quoteMarkdown) return;
const quoteIdentity = pickFirstValidLinkText(quotedUserNameElements[index]);
parts.push(buildQuoteMarkdown(quoteMarkdown, quoteIdentity.author));
});
const media = collectMediaMarkdown(article, new Set<string>());
if (media.lines.length > 0) {
parts.push(media.lines.join("\n\n"));
}
const mainText = cleanText(mainTextElement.textContent);
const markdown = normalizeXMarkdown(parts.join("\n\n"));
return {
metadata: {
...context.baseMetadata,
title: buildTweetTitle(mainText, context.baseMetadata.title),
description: summarizeText(mainText, 220) ?? context.baseMetadata.description,
author: mainIdentity.author ?? context.baseMetadata.author,
published: published ?? context.baseMetadata.published,
coverImage: sanitizeCoverImage(media.urls[0], context.baseMetadata.coverImage),
language: inferLanguage(mainText, context.baseMetadata.language),
},
markdown,
rawHtml: context.html,
conversionMethod: "parser:x-status",
};
}
export const xStatusRuleParser: UrlRuleParser = {
id: "x-status",
supports(context) {
const parsed = parseUrl(context.url);
if (!parsed || !isXHost(parsed.hostname)) {
return false;
}
return isXStatusPath(parsed.pathname) && Boolean(context.document.querySelector("[data-testid='tweetText']"));
},
parse(context): ConversionResult | null {
return parseXStatus(context);
},
};
@@ -0,0 +1,14 @@
import type { ConversionResult, PageMetadata } from "../markdown-conversion-shared.js";
export interface UrlRuleParserContext {
html: string;
url: string;
document: Document;
baseMetadata: PageMetadata;
}
export interface UrlRuleParser {
id: string;
supports(context: UrlRuleParserContext): boolean;
parse(context: UrlRuleParserContext): ConversionResult | null;
}
@@ -151,8 +151,7 @@ From outline entry:
```markdown
## Watermark
Include a subtle watermark "{content}" positioned at {position}
with approximately {opacity*100}% visibility. The watermark should
Include a subtle watermark "{content}" positioned at {position}. The watermark should
be legible but not distracting from the main content.
```
@@ -295,19 +294,19 @@ Create a Xiaohongshu (Little Red Book) style infographic following these guideli
## Content
**Position**: Content (Page 3 of 6)
**Core Message**: ChatGPT使用技巧
**Core Message**: ChatGPT 使用技巧
**Text Content**:
- Title: 「ChatGPT」
- Subtitle: 最强AI助手
- Subtitle: 最强 AI 助手
- Points:
- 写文案:给出框架,秒出初稿
- 改文章:润色、翻译、总结
- 编程:写代码、找bug
- 编程:写代码、找 bug
- 学习:解释概念、出题练习
**Visual Concept**:
ChatGPT logo居中,四周放射状展示功能点
ChatGPT logo 居中,四周放射状展示功能点
深色科技背景,霓虹绿点缀
---
+177
View File
@@ -0,0 +1,177 @@
---
name: baoyu-youtube-transcript
description: Downloads YouTube video transcripts/subtitles and cover images by URL or video ID. Supports multiple languages, translation, chapters, and speaker identification. Caches raw data for fast re-formatting. Use when user asks to "get YouTube transcript", "download subtitles", "get captions", "YouTube字幕", "YouTube封面", "视频封面", "video thumbnail", "video cover image", or provides a YouTube URL and wants the transcript/subtitle text or cover image extracted.
version: 1.1.0
metadata:
openclaw:
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-youtube-transcript
requires:
anyBins:
- bun
- npx
---
# YouTube Transcript
Downloads transcripts (subtitles/captions) from YouTube videos. Works with both manually created and auto-generated transcripts. No API key or browser required — uses YouTube's InnerTube API directly.
Fetches video metadata and cover image on first run, caches raw data for fast re-formatting.
## Script Directory
Scripts in `scripts/` subdirectory. `{baseDir}` = this SKILL.md's directory path. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun. Replace `{baseDir}` and `${BUN_X}` with actual values.
| Script | Purpose |
|--------|---------|
| `scripts/main.ts` | Transcript download CLI |
## Usage
```bash
# Default: markdown with timestamps (English)
${BUN_X} {baseDir}/scripts/main.ts <youtube-url-or-id>
# Specify languages (priority order)
${BUN_X} {baseDir}/scripts/main.ts <url> --languages zh,en,ja
# Without timestamps
${BUN_X} {baseDir}/scripts/main.ts <url> --no-timestamps
# With chapter segmentation
${BUN_X} {baseDir}/scripts/main.ts <url> --chapters
# With speaker identification (requires AI post-processing)
${BUN_X} {baseDir}/scripts/main.ts <url> --speakers
# SRT subtitle file
${BUN_X} {baseDir}/scripts/main.ts <url> --format srt
# Translate transcript
${BUN_X} {baseDir}/scripts/main.ts <url> --translate zh-Hans
# List available transcripts
${BUN_X} {baseDir}/scripts/main.ts <url> --list
# Force re-fetch (ignore cache)
${BUN_X} {baseDir}/scripts/main.ts <url> --refresh
```
## Options
| Option | Description | Default |
|--------|-------------|---------|
| `<url-or-id>` | YouTube URL or video ID (multiple allowed) | Required |
| `--languages <codes>` | Language codes, comma-separated, in priority order | `en` |
| `--format <fmt>` | Output format: `text`, `srt` | `text` |
| `--translate <code>` | Translate to specified language code | |
| `--list` | List available transcripts instead of fetching | |
| `--timestamps` | Include `[HH:MM:SS → HH:MM:SS]` timestamps per paragraph | on |
| `--no-timestamps` | Disable timestamps | |
| `--chapters` | Chapter segmentation from video description | |
| `--speakers` | Raw transcript with metadata for speaker identification | |
| `--exclude-generated` | Skip auto-generated transcripts | |
| `--exclude-manually-created` | Skip manually created transcripts | |
| `--refresh` | Force re-fetch, ignore cached data | |
| `-o, --output <path>` | Save to specific file path | auto-generated |
| `--output-dir <dir>` | Base output directory | `youtube-transcript` |
## Input Formats
Accepts any of these as video input:
- Full URL: `https://www.youtube.com/watch?v=dQw4w9WgXcQ`
- Short URL: `https://youtu.be/dQw4w9WgXcQ`
- Embed URL: `https://www.youtube.com/embed/dQw4w9WgXcQ`
- Shorts URL: `https://www.youtube.com/shorts/dQw4w9WgXcQ`
- Video ID: `dQw4w9WgXcQ`
## Output Formats
| Format | Extension | Description |
|--------|-----------|-------------|
| `text` | `.md` | Markdown with frontmatter (incl. `description`), title heading, summary, optional TOC/cover/timestamps/chapters/speakers |
| `srt` | `.srt` | SubRip subtitle format for video players |
## Output Directory
```
youtube-transcript/
├── .index.json # Video ID → directory path mapping (for cache lookup)
└── {channel-slug}/{title-full-slug}/
├── meta.json # Video metadata (title, channel, description, duration, chapters, etc.)
├── transcript-raw.json # Raw transcript snippets from YouTube API (cached)
├── transcript-sentences.json # Sentence-segmented transcript (split by punctuation, merged across snippets)
├── imgs/
│ └── cover.jpg # Video thumbnail
├── transcript.md # Markdown transcript (generated from sentences)
└── transcript.srt # SRT subtitle (generated from raw snippets, if --format srt)
```
- `{channel-slug}`: Channel name in kebab-case
- `{title-full-slug}`: Full video title in kebab-case
The `--list` mode outputs to stdout only (no file saved).
## Caching
On first fetch, the script saves:
- `meta.json` — video metadata, chapters, cover image path, language info
- `transcript-raw.json` — raw transcript snippets from YouTube API (`{ text, start, duration }[]`)
- `transcript-sentences.json` — sentence-segmented transcript (`{ text, start: "HH:mm:ss", end: "HH:mm:ss" }[]`), split by sentence-ending punctuation (`.?!…。?!` etc.), timestamps proportionally allocated by character length, CJK-aware text merging
- `imgs/cover.jpg` — video thumbnail
Subsequent runs for the same video use cached data (no network calls). Use `--refresh` to force re-fetch. If a different language is requested, the cache is automatically refreshed.
SRT output (`--format srt`) is generated from `transcript-raw.json`. Text/markdown output uses `transcript-sentences.json` for natural sentence boundaries.
## Workflow
When user provides a YouTube URL and wants the transcript:
1. Run with `--list` first if the user hasn't specified a language, to show available options
2. **Always single-quote the URL** when running the script — zsh treats `?` as a glob wildcard, so an unquoted YouTube URL causes "no matches found": use `'https://www.youtube.com/watch?v=ID'`
3. Default: run with `--chapters --speakers` for the richest output (chapters + speaker identification)
3. The script auto-saves cached data + output file and prints the file path
4. For `--speakers` mode: after the script saves the raw file, follow the speaker identification workflow below to post-process with speaker labels
When user only wants a cover image or metadata, running the script with any option will also cache `meta.json` and `imgs/cover.jpg`.
When re-formatting the same video (e.g., first text then SRT), the cached data is reused — no re-fetch needed.
## Chapter & Speaker Workflow
### Chapters (`--chapters`)
The script parses chapter timestamps from the video description (e.g., `0:00 Introduction`), segments the transcript by chapter boundaries, groups snippets into readable paragraphs, and saves as `.md` with a Table of Contents. No further processing needed.
If no chapter timestamps exist in the description, the transcript is output as grouped paragraphs without chapter headings.
### Speaker Identification (`--speakers`)
Speaker identification requires AI processing. The script outputs a raw `.md` file containing:
- YAML frontmatter with video metadata (title, channel, date, cover, description, language)
- Video description (for speaker name extraction)
- Chapter list from description (if available)
- Raw transcript in SRT format (pre-computed start/end timestamps, token-efficient)
After the script saves the raw file, spawn a sub-agent (use a cheaper model like Sonnet for cost efficiency) to process speaker identification:
1. Read the saved `.md` file
2. Read the prompt template at `{baseDir}/prompts/speaker-transcript.md`
3. Process the raw transcript following the prompt:
- Identify speakers using video metadata (title → guest, channel → host, description → names)
- Detect speaker turns from conversation flow, question-answer patterns, and contextual cues
- Segment into chapters (use description chapters if available, else create from topic shifts)
- Format with `**Speaker Name:**` labels, paragraph grouping (2-4 sentences), and `[HH:MM:SS → HH:MM:SS]` timestamps
4. Overwrite the `.md` file with the processed transcript (keep the YAML frontmatter)
When `--speakers` is used, `--chapters` is implied — the processed output always includes chapter segmentation.
## Error Cases
| Error | Meaning |
|-------|---------|
| Transcripts disabled | Video has no captions at all |
| No transcript found | Requested language not available |
| Video unavailable | Video deleted, private, or region-locked |
| IP blocked | Too many requests, try again later |
| Age restricted | Video requires login for age verification |
@@ -0,0 +1,118 @@
# Speaker & Chapter Transcript Processing
You are an expert transcript specialist. Process the raw transcript file (with YAML frontmatter metadata and SRT-formatted transcript) into a structured, verbatim transcript with speaker identification and chapter segmentation.
## Output Structure
Produce a single cohesive markdown file containing:
1. YAML frontmatter (keep the original frontmatter from the raw file, which includes `description`)
2. `# Title` heading (from frontmatter title)
3. Description/summary paragraph (from frontmatter `description`)
4. Table of Contents
5. Cover image (if `cover` exists in frontmatter): `![cover](imgs/cover.jpg)` — right after the ToC
6. Full chapter-segmented transcript with speaker labels
Use the same language as the transcription for the title and ToC.
## Rules
### Transcription Fidelity
- Preserve every spoken word exactly, including filler words (`um`, `uh`, `like`) and stutters
- **NEVER translate.** If the audio mixes languages (e.g., "这个 feature 很酷"), replicate that mix exactly
### Speaker Identification
- **Priority 1: Use metadata.** Analyze the video's title, channel name, and description to identify speakers
- **Priority 2: Use transcript content.** Look for introductions, how speakers address each other, contextual cues
- **Fallback:** Use consistent generic labels (`**Speaker 1:**`, `**Host:**`, etc.)
- **Consistency:** If a speaker's name is revealed later, update ALL previous labels for that speaker
### Chapter Generation
- If the raw file contains a `# Chapters` section, use those as the primary basis for segmenting
- Otherwise, create chapters based on significant topic shifts in the conversation
### Input Format
- The `# Transcript` section contains SRT-formatted subtitles with pre-computed start/end timestamps
- Each SRT block has: sequence number, `HH:MM:SS,mmm --> HH:MM:SS,mmm` timestamp line, and text
- Use the SRT timestamps directly — no need to calculate paragraph start/end times, just merge adjacent blocks
### Formatting
**Timestamps:** Use `[HH:MM:SS → HH:MM:SS]` format (start → end) at the end of each paragraph. No milliseconds.
**Table of Contents:**
```
## Table of Contents
* [HH:MM:SS] Chapter Title
```
**Chapters:**
```
## Chapter Title [HH:MM:SS]
```
Two blank lines between chapters.
**Dialogue Paragraphs:**
- First paragraph of a speaker's turn starts with `**Speaker Name:** `
- Split long monologues into 2-4 sentence paragraphs separated by blank lines
- Subsequent paragraphs from the SAME speaker do NOT repeat the speaker label
- Every paragraph ends with exactly ONE timestamp range `[HH:MM:SS → HH:MM:SS]`
Correct example:
```
**Jane Doe:** The study focuses on long-term effects of dietary changes. We tracked two groups over five years. [00:00:15 → 00:00:21]
The first group followed the new regimen, while the second group maintained a traditional diet. [00:00:21 → 00:00:28]
**Host:** Fascinating. And what did you find? [00:00:28 → 00:00:31]
```
Wrong (multiple timestamps in one paragraph):
```
**Host:** Welcome back. [00:00:01] Today we have a guest. [00:00:02]
```
**Non-Speech Audio:** On its own line: `[Laughter] [HH:MM:SS]`
## Example Output
```markdown
---
title: "Example Interview"
channel: "The Show"
date: 2024-04-15
url: "https://www.youtube.com/watch?v=xxx"
cover: imgs/cover.jpg
description: "Jane Doe discusses her groundbreaking five-year study on the long-term effects of dietary changes."
language: en
---
# Example Interview
Jane Doe discusses her groundbreaking five-year study on the long-term effects of dietary changes.
## Table of Contents
* [00:00:00] Introduction and Welcome
* [00:00:12] Overview of the New Research
![cover](imgs/cover.jpg)
## Introduction and Welcome [00:00:00]
**Host:** Welcome back to the show. Today, we have a, uh, very special guest, Jane Doe. [00:00:00 → 00:00:03]
**Jane Doe:** Thank you for having me. I'm excited to be here and discuss the findings. [00:00:03 → 00:00:07]
## Overview of the New Research [00:00:12]
**Host:** So, Jane, before we get into the nitty-gritty, could you, you know, give us a brief overview for our audience? [00:00:12 → 00:00:16]
**Jane Doe:** Of course. The study focuses on the long-term effects of specific dietary changes. It's a bit complicated but essentially we tracked two large groups over a five-year period. [00:00:16 → 00:00:23]
The first group followed the new regimen, while the second group, our control, maintained a traditional diet. This allowed us to isolate variables effectively. [00:00:23 → 00:00:30]
[Laughter] [00:00:30]
**Host:** Fascinating. And what did you find? [00:00:31 → 00:00:33]
```
@@ -0,0 +1,857 @@
#!/usr/bin/env bun
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "fs";
import { dirname, join, resolve } from "path";
type Format = "text" | "srt";
interface Options {
videoIds: string[];
languages: string[];
format: Format;
translate: string;
list: boolean;
excludeGenerated: boolean;
excludeManual: boolean;
output: string;
outputDir: string;
timestamps: boolean;
chapters: boolean;
speakers: boolean;
refresh: boolean;
}
interface Snippet {
text: string;
start: number;
duration: number;
}
interface Sentence {
text: string;
start: string;
end: string;
}
interface TranscriptInfo {
language: string;
languageCode: string;
isGenerated: boolean;
isTranslatable: boolean;
baseUrl: string;
translationLanguages: { language: string; languageCode: string }[];
}
interface Chapter {
title: string;
start: number;
end: number;
}
interface VideoMeta {
videoId: string;
title: string;
channel: string;
channelId: string;
description: string;
duration: number;
publishDate: string;
url: string;
coverImage: string;
thumbnailUrl: string;
language: { code: string; name: string; isGenerated: boolean };
chapters: Chapter[];
}
interface VideoResult {
videoId: string;
title?: string;
filePath?: string;
content?: string;
error?: string;
}
const WATCH_URL = "https://www.youtube.com/watch?v=";
const INNERTUBE_URL = "https://www.youtube.com/youtubei/v1/player";
const INNERTUBE_CTX = { client: { clientName: "ANDROID", clientVersion: "20.10.38" } };
function extractVideoId(input: string): string {
input = input.replace(/\\/g, "").trim();
const patterns = [
/(?:youtube\.com\/watch\?.*v=|youtu\.be\/|youtube\.com\/embed\/|youtube\.com\/v\/|youtube\.com\/shorts\/)([a-zA-Z0-9_-]{11})/,
/^([a-zA-Z0-9_-]{11})$/,
];
for (const p of patterns) {
const m = input.match(p);
if (m) return m[1];
}
return input;
}
function slugify(s: string): string {
return s
.toLowerCase()
.replace(/[^\w\s-]/g, "")
.replace(/\s+/g, "-")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "") || "untitled";
}
function htmlUnescape(s: string): string {
return s
.replace(/&amp;/g, "&")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&#39;/g, "'")
.replace(/&#x27;/g, "'")
.replace(/&#x2F;/g, "/")
.replace(/&apos;/g, "'")
.replace(/&#(\d+);/g, (_, n) => String.fromCharCode(parseInt(n)))
.replace(/&#x([0-9a-fA-F]+);/g, (_, n) => String.fromCharCode(parseInt(n, 16)));
}
function stripTags(s: string): string {
return s.replace(/<[^>]*>/g, "");
}
function parseTranscriptXml(xml: string): Snippet[] {
const snippets: Snippet[] = [];
const re = /<text\s+start="([^"]*)"(?:\s+dur="([^"]*)")?[^>]*>([\s\S]*?)<\/text>/g;
let m: RegExpExecArray | null;
while ((m = re.exec(xml)) !== null) {
const raw = m[3];
if (!raw) continue;
snippets.push({
text: htmlUnescape(stripTags(raw)),
start: parseFloat(m[1]),
duration: parseFloat(m[2] || "0"),
});
}
return snippets;
}
// --- YouTube API ---
async function fetchHtml(videoId: string): Promise<string> {
const r = await fetch(WATCH_URL + videoId, {
headers: { "Accept-Language": "en-US", "User-Agent": "Mozilla/5.0" },
});
if (!r.ok) throw new Error(`HTTP ${r.status} fetching video page`);
let html = await r.text();
if (html.includes('action="https://consent.youtube.com/s"')) {
const cv = html.match(/name="v" value="(.*?)"/);
if (!cv) throw new Error("Failed to create consent cookie");
const r2 = await fetch(WATCH_URL + videoId, {
headers: {
"Accept-Language": "en-US",
"User-Agent": "Mozilla/5.0",
Cookie: `CONSENT=YES+${cv[1]}`,
},
});
if (!r2.ok) throw new Error(`HTTP ${r2.status} fetching video page (consent)`);
html = await r2.text();
}
return html;
}
function extractApiKey(html: string, videoId: string): string {
const m = html.match(/"INNERTUBE_API_KEY":\s*"([a-zA-Z0-9_-]+)"/);
if (!m) {
if (html.includes('class="g-recaptcha"')) throw new Error(`IP blocked for ${videoId} (reCAPTCHA)`);
throw new Error(`Cannot extract API key for ${videoId}`);
}
return m[1];
}
async function fetchInnertubeData(videoId: string, apiKey: string): Promise<any> {
const r = await fetch(`${INNERTUBE_URL}?key=${apiKey}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ context: INNERTUBE_CTX, videoId }),
});
if (r.status === 429) throw new Error(`IP blocked for ${videoId} (429)`);
if (!r.ok) throw new Error(`HTTP ${r.status} from InnerTube API`);
return r.json();
}
function assertPlayability(data: any, videoId: string) {
const ps = data?.playabilityStatus;
if (!ps) return;
const status = ps.status;
if (status === "OK" || !status) return;
const reason = ps.reason || "";
if (status === "LOGIN_REQUIRED") {
if (reason.includes("bot")) throw new Error(`Request blocked for ${videoId}: bot detected`);
if (reason.includes("inappropriate")) throw new Error(`Age restricted: ${videoId}`);
}
if (status === "ERROR" && reason.includes("unavailable")) {
if (videoId.startsWith("http")) throw new Error(`Invalid video ID: pass the ID, not the URL`);
throw new Error(`Video unavailable: ${videoId}`);
}
const subreasons = ps.errorScreen?.playerErrorMessageRenderer?.subreason?.runs?.map((r: any) => r.text).join("") || "";
throw new Error(`Video unplayable (${videoId}): ${reason} ${subreasons}`.trim());
}
function extractCaptionsJson(data: any, videoId: string): any {
assertPlayability(data, videoId);
const cj = data?.captions?.playerCaptionsTracklistRenderer;
if (!cj || !cj.captionTracks) throw new Error(`Transcripts disabled for ${videoId}`);
return cj;
}
function buildTranscriptList(captionsJson: any): TranscriptInfo[] {
const tlLangs = (captionsJson.translationLanguages || []).map((tl: any) => ({
language: tl.languageName?.runs?.[0]?.text || tl.languageName?.simpleText || "",
languageCode: tl.languageCode,
}));
return (captionsJson.captionTracks || []).map((t: any) => ({
language: t.name?.runs?.[0]?.text || t.name?.simpleText || "",
languageCode: t.languageCode,
isGenerated: t.kind === "asr",
isTranslatable: !!t.isTranslatable,
baseUrl: (t.baseUrl || "").replace(/&fmt=srv3/g, ""),
translationLanguages: t.isTranslatable ? tlLangs : [],
}));
}
function findTranscript(
transcripts: TranscriptInfo[],
languages: string[],
excludeGenerated: boolean,
excludeManual: boolean
): TranscriptInfo {
let filtered = transcripts;
if (excludeGenerated) filtered = filtered.filter((t) => !t.isGenerated);
if (excludeManual) filtered = filtered.filter((t) => t.isGenerated);
for (const lang of languages) {
const found = filtered.find((t) => t.languageCode === lang);
if (found) return found;
}
const available = filtered.map((t) => `${t.languageCode} ("${t.language}")`).join(", ");
throw new Error(`No transcript found for languages [${languages.join(", ")}]. Available: ${available || "none"}`);
}
async function fetchTranscriptSnippets(info: TranscriptInfo, translateTo?: string): Promise<{ snippets: Snippet[]; language: string; languageCode: string }> {
let url = info.baseUrl;
let lang = info.language;
let langCode = info.languageCode;
if (translateTo) {
if (!info.isTranslatable) throw new Error(`Transcript ${info.languageCode} is not translatable`);
const tl = info.translationLanguages.find((t) => t.languageCode === translateTo);
if (!tl) throw new Error(`Translation language ${translateTo} not available`);
url += `&tlang=${translateTo}`;
lang = tl.language;
langCode = translateTo;
}
const r = await fetch(url, { headers: { "Accept-Language": "en-US" } });
if (!r.ok) throw new Error(`HTTP ${r.status} fetching transcript`);
return { snippets: parseTranscriptXml(await r.text()), language: lang, languageCode: langCode };
}
// --- Metadata & chapters ---
function parseChapters(description: string, duration: number = 0): Chapter[] {
const raw: { title: string; start: number }[] = [];
for (const line of description.split("\n")) {
const m = line.trim().match(/^(?:(\d{1,2}):)?(\d{1,2}):(\d{2})\s+(.+)$/);
if (m) {
const h = m[1] ? parseInt(m[1]) : 0;
raw.push({ title: m[4].trim(), start: h * 3600 + parseInt(m[2]) * 60 + parseInt(m[3]) });
}
}
if (raw.length < 2) return [];
return raw.map((ch, i) => ({
title: ch.title,
start: ch.start,
end: i < raw.length - 1 ? raw[i + 1].start : Math.max(duration, ch.start),
}));
}
function getThumbnailUrls(videoId: string, data: any): string[] {
const urls = [
`https://i.ytimg.com/vi/${videoId}/maxresdefault.jpg`,
`https://i.ytimg.com/vi/${videoId}/hqdefault.jpg`,
];
const thumbnails = data?.videoDetails?.thumbnail?.thumbnails ||
data?.microformat?.playerMicroformatRenderer?.thumbnail?.thumbnails || [];
if (thumbnails.length) {
const sorted = [...thumbnails].sort((a: any, b: any) => (b.width || 0) - (a.width || 0));
for (const t of sorted) if (t.url && !urls.includes(t.url)) urls.push(t.url);
}
return urls;
}
function buildVideoMeta(data: any, videoId: string, langInfo: { code: string; name: string; isGenerated: boolean }, chapters: Chapter[]): VideoMeta {
const vd = data?.videoDetails || {};
const mf = data?.microformat?.playerMicroformatRenderer || {};
return {
videoId,
title: vd.title || mf.title?.simpleText || "",
channel: vd.author || mf.ownerChannelName || "",
channelId: vd.channelId || mf.externalChannelId || "",
description: vd.shortDescription || mf.description?.simpleText || "",
duration: parseInt(vd.lengthSeconds || "0"),
publishDate: mf.publishDate || mf.uploadDate || "",
url: `https://www.youtube.com/watch?v=${videoId}`,
coverImage: "",
thumbnailUrl: getThumbnailUrls(videoId, data)[0],
language: langInfo,
chapters,
};
}
async function downloadCoverImage(urls: string[], outputPath: string): Promise<boolean> {
for (const u of urls) {
try {
const r = await fetch(u);
if (r.ok) {
writeFileSync(outputPath, Buffer.from(await r.arrayBuffer()));
return true;
}
} catch {}
}
return false;
}
function parseSrt(srt: string): Snippet[] {
const blocks = srt.trim().split(/\n\n+/);
const snippets: Snippet[] = [];
for (const block of blocks) {
const lines = block.split("\n");
if (lines.length < 3) continue;
const m = lines[1].match(/(\d{2}):(\d{2}):(\d{2}),(\d{3})\s*-->\s*(\d{2}):(\d{2}):(\d{2}),(\d{3})/);
if (!m) continue;
const start = parseInt(m[1]) * 3600 + parseInt(m[2]) * 60 + parseInt(m[3]) + parseInt(m[4]) / 1000;
const end = parseInt(m[5]) * 3600 + parseInt(m[6]) * 60 + parseInt(m[7]) + parseInt(m[8]) / 1000;
snippets.push({ text: lines.slice(2).join(" "), start, duration: end - start });
}
return snippets;
}
// --- Timestamp formatting ---
function ts(t: number): string {
const h = Math.floor(t / 3600);
const m = Math.floor((t % 3600) / 60);
const s = Math.floor(t % 60);
return `${String(h).padStart(2, "0")}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`;
}
function tsMs(t: number, sep: string): string {
const h = Math.floor(t / 3600);
const m = Math.floor((t % 3600) / 60);
const s = Math.floor(t % 60);
const ms = Math.round((t - Math.floor(t)) * 1000);
return `${String(h).padStart(2, "0")}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}${sep}${String(ms).padStart(3, "0")}`;
}
// --- Paragraph grouping ---
interface Paragraph {
text: string;
start: number;
end: number;
}
function groupIntoParagraphs(snippets: Snippet[]): Paragraph[] {
if (!snippets.length) return [];
const paras: Paragraph[] = [];
let buf: Snippet[] = [];
for (let i = 0; i < snippets.length; i++) {
buf.push(snippets[i]);
const last = i === snippets.length - 1;
const gap = !last && snippets[i + 1].start - (snippets[i].start + snippets[i].duration) > 1.5;
if (last || gap || buf.length >= 8) {
const lastS = buf[buf.length - 1];
paras.push({ text: buf.map(s => s.text).join(" "), start: buf[0].start, end: lastS.start + lastS.duration });
buf = [];
}
}
return paras;
}
// --- Sentence segmentation ---
const SENTENCE_END_RE = /[.?!…。?!⁈⁇‼‽.]/;
function isCJK(ch: string): boolean {
const code = ch.charCodeAt(0);
return (code >= 0x4E00 && code <= 0x9FFF) ||
(code >= 0x3040 && code <= 0x309F) ||
(code >= 0x30A0 && code <= 0x30FF) ||
(code >= 0xAC00 && code <= 0xD7AF) ||
(code >= 0x3400 && code <= 0x4DBF) ||
(code >= 0xF900 && code <= 0xFAFF);
}
function splitSnippetAtPunctuation(s: Snippet): { text: string; start: number; end: number }[] {
const { text, start, duration } = s;
const end = start + duration;
if (!text.length) return [];
const splitPoints: number[] = [];
for (let i = 0; i < text.length; i++) {
if (SENTENCE_END_RE.test(text[i])) {
while (i + 1 < text.length && SENTENCE_END_RE.test(text[i + 1])) i++;
if (i < text.length - 1) splitPoints.push(i);
}
}
if (!splitPoints.length) return [{ text, start, end }];
const parts: { text: string; start: number; end: number }[] = [];
let prev = 0;
for (const pos of splitPoints) {
const partText = text.slice(prev, pos + 1).trim();
if (partText) {
parts.push({
text: partText,
start: start + (prev / text.length) * duration,
end: start + ((pos + 1) / text.length) * duration,
});
}
prev = pos + 1;
}
const remaining = text.slice(prev).trim();
if (remaining) {
parts.push({ text: remaining, start: start + (prev / text.length) * duration, end });
}
return parts;
}
function mergeTexts(texts: string[]): string {
if (!texts.length) return "";
let result = texts[0];
for (let i = 1; i < texts.length; i++) {
const next = texts[i];
if (!next) continue;
const lastChar = result[result.length - 1];
const firstChar = next[0];
if (isCJK(lastChar) || isCJK(firstChar)) {
result += next;
} else {
result = result.trimEnd() + " " + next.trimStart();
}
}
return result.replace(/ {2,}/g, " ");
}
function segmentIntoSentences(snippets: Snippet[]): Sentence[] {
const parts: { text: string; start: number; end: number }[] = [];
for (const s of snippets) parts.push(...splitSnippetAtPunctuation(s));
const sentences: Sentence[] = [];
let buf: { text: string; start: number; end: number }[] = [];
for (const part of parts) {
buf.push(part);
if (SENTENCE_END_RE.test(part.text[part.text.length - 1])) {
sentences.push({
text: mergeTexts(buf.map(b => b.text)),
start: ts(buf[0].start),
end: ts(buf[buf.length - 1].end),
});
buf = [];
}
}
if (buf.length) {
sentences.push({
text: mergeTexts(buf.map(b => b.text)),
start: ts(buf[0].start),
end: ts(buf[buf.length - 1].end),
});
}
return sentences;
}
function parseTs(t: string): number {
const [h, m, s] = t.split(":").map(Number);
return h * 3600 + m * 60 + s;
}
function groupSentenceParas(sentences: Sentence[]): Paragraph[] {
if (!sentences.length) return [];
const paras: Paragraph[] = [];
let buf: Sentence[] = [];
for (let i = 0; i < sentences.length; i++) {
buf.push(sentences[i]);
const last = i === sentences.length - 1;
const gap = !last && parseTs(sentences[i + 1].start) - parseTs(sentences[i].end) > 2;
if (last || gap || buf.length >= 5) {
paras.push({
text: mergeTexts(buf.map(s => s.text)),
start: parseTs(buf[0].start),
end: parseTs(buf[buf.length - 1].end),
});
buf = [];
}
}
return paras;
}
// --- Format functions ---
function formatSrt(snippets: Snippet[]): string {
return snippets
.map((s, i) => {
const end = i < snippets.length - 1 && snippets[i + 1].start < s.start + s.duration
? snippets[i + 1].start
: s.start + s.duration;
return `${i + 1}\n${tsMs(s.start, ",")} --> ${tsMs(end, ",")}\n${s.text}`;
})
.join("\n\n") + "\n";
}
function yamlEscape(s: string): string {
if (/[:"'{}\[\]#&*!|>%@`\n]/.test(s) || s.trim() !== s) return `"${s.replace(/\\/g, "\\\\").replace(/"/g, '\\"')}"`;
return s;
}
function extractSummary(description: string): string {
if (!description) return "";
const firstPara = description.split(/\n\s*\n/)[0].trim();
const lines = firstPara.split("\n").filter(l => !/^\s*(https?:\/\/|#|@|\d+:\d+)/.test(l) && l.trim());
return lines.join(" ").slice(0, 300).trim();
}
function formatMarkdown(sentences: Sentence[], meta: VideoMeta, opts: { timestamps: boolean; chapters: boolean; speakers: boolean }, snippets?: Snippet[]): string {
const summary = extractSummary(meta.description);
let md = "---\n";
md += `title: ${yamlEscape(meta.title)}\n`;
md += `channel: ${yamlEscape(meta.channel)}\n`;
if (meta.publishDate) md += `date: ${meta.publishDate}\n`;
md += `url: ${yamlEscape(meta.url)}\n`;
if (meta.coverImage) md += `cover: ${meta.coverImage}\n`;
if (summary) md += `description: ${yamlEscape(summary)}\n`;
if (meta.language) md += `language: ${meta.language.code}\n`;
md += "---\n\n";
if (opts.speakers) {
md += `# ${meta.title}\n\n`;
if (summary) md += `${summary}\n\n`;
if (meta.description) md += "# Description\n\n" + meta.description.trim() + "\n\n";
if (meta.chapters.length) {
md += "# Chapters\n\n";
for (const ch of meta.chapters) md += `* [${ts(ch.start)}] ${ch.title}\n`;
md += "\n";
}
md += "# Transcript\n\n";
md += snippets ? formatSrt(snippets) : "";
return md;
}
md += `# ${meta.title}\n\n`;
if (summary) md += `${summary}\n\n`;
const chapters = opts.chapters ? meta.chapters : [];
if (chapters.length) {
md += "## Table of Contents\n\n";
for (const ch of chapters) md += opts.timestamps ? `* [${ts(ch.start)}] ${ch.title}\n` : `* ${ch.title}\n`;
md += "\n";
if (meta.coverImage) md += `\n![cover](${meta.coverImage})\n`;
md += "\n";
for (let i = 0; i < chapters.length; i++) {
const nextStart = i < chapters.length - 1 ? chapters[i + 1].start : Infinity;
const chSentences = sentences.filter(s => parseTs(s.start) >= chapters[i].start && parseTs(s.start) < nextStart);
const paras = groupSentenceParas(chSentences);
md += opts.timestamps
? `## [${ts(chapters[i].start)}] ${chapters[i].title}\n\n`
: `## ${chapters[i].title}\n\n`;
for (const p of paras) md += opts.timestamps ? `${p.text} [${ts(p.start)}${ts(p.end)}]\n\n` : `${p.text}\n\n`;
md += "\n";
}
} else {
const paras = groupSentenceParas(sentences);
for (const p of paras) md += opts.timestamps ? `${p.text} [${ts(p.start)}${ts(p.end)}]\n\n` : `${p.text}\n\n`;
}
return md.trimEnd() + "\n";
}
function formatListOutput(videoId: string, title: string, transcripts: TranscriptInfo[]): string {
const manual = transcripts.filter((t) => !t.isGenerated);
const generated = transcripts.filter((t) => t.isGenerated);
const tlLangs = transcripts.find((t) => t.translationLanguages.length > 0)?.translationLanguages || [];
const fmtList = (list: TranscriptInfo[]) =>
list.length ? list.map((t) => ` - ${t.languageCode} ("${t.language}")${t.isTranslatable ? " [TRANSLATABLE]" : ""}`).join("\n") : "None";
const fmtTl = tlLangs.length
? tlLangs.map((t) => ` - ${t.languageCode} ("${t.language}")`).join("\n")
: "None";
return `Transcripts for ${videoId}${title ? ` (${title})` : ""}:\n\n(MANUALLY CREATED)\n${fmtList(manual)}\n\n(GENERATED)\n${fmtList(generated)}\n\n(TRANSLATION LANGUAGES)\n${fmtTl}`;
}
// --- File helpers ---
function ensureDir(p: string) {
const dir = dirname(p);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
}
function resolveBaseDir(outputDir: string): string {
return resolve(outputDir || "youtube-transcript");
}
function loadIndex(baseDir: string): Record<string, string> {
try { return JSON.parse(readFileSync(join(baseDir, ".index.json"), "utf-8")); } catch { return {}; }
}
function saveIndex(baseDir: string, index: Record<string, string>) {
const p = join(baseDir, ".index.json");
ensureDir(p);
writeFileSync(p, JSON.stringify(index, null, 2));
}
function lookupVideoDir(videoId: string, baseDir: string): string | null {
const rel = loadIndex(baseDir)[videoId];
if (rel) {
const dir = resolve(baseDir, rel);
if (existsSync(dir)) return dir;
}
return null;
}
function registerVideoDir(videoId: string, channelSlug: string, titleSlug: string, baseDir: string): string {
const rel = join(channelSlug, titleSlug);
const index = loadIndex(baseDir);
index[videoId] = rel;
saveIndex(baseDir, index);
return resolve(baseDir, rel);
}
function hasCachedData(videoDir: string): boolean {
return existsSync(join(videoDir, "meta.json")) && existsSync(join(videoDir, "transcript-raw.json"));
}
function loadMeta(videoDir: string): VideoMeta {
return JSON.parse(readFileSync(join(videoDir, "meta.json"), "utf-8"));
}
function loadSnippets(videoDir: string): Snippet[] {
return JSON.parse(readFileSync(join(videoDir, "transcript-raw.json"), "utf-8"));
}
function loadSentences(videoDir: string): Sentence[] {
return JSON.parse(readFileSync(join(videoDir, "transcript-sentences.json"), "utf-8"));
}
// --- Main processing ---
async function fetchAndCache(videoId: string, baseDir: string, opts: Options): Promise<{ meta: VideoMeta; snippets: Snippet[]; sentences: Sentence[]; videoDir: string }> {
const html = await fetchHtml(videoId);
const apiKey = extractApiKey(html, videoId);
const data = await fetchInnertubeData(videoId, apiKey);
const captionsJson = extractCaptionsJson(data, videoId);
const transcripts = buildTranscriptList(captionsJson);
const info = findTranscript(transcripts, opts.languages, opts.excludeGenerated, opts.excludeManual);
const result = await fetchTranscriptSnippets(info, opts.translate || undefined);
const description = data?.videoDetails?.shortDescription || "";
const duration = parseInt(data?.videoDetails?.lengthSeconds || "0");
const chapters = parseChapters(description, duration);
const langInfo = { code: result.languageCode, name: result.language, isGenerated: info.isGenerated };
const meta = buildVideoMeta(data, videoId, langInfo, chapters);
const videoDir = registerVideoDir(videoId, slugify(meta.channel), slugify(meta.title), baseDir);
ensureDir(join(videoDir, "meta.json"));
writeFileSync(join(videoDir, "transcript-raw.json"), JSON.stringify(result.snippets, null, 2));
const sentences = segmentIntoSentences(result.snippets);
writeFileSync(join(videoDir, "transcript-sentences.json"), JSON.stringify(sentences, null, 2));
const imgPath = join(videoDir, "imgs", "cover.jpg");
ensureDir(imgPath);
const downloaded = await downloadCoverImage(getThumbnailUrls(videoId, data), imgPath);
meta.coverImage = downloaded ? "imgs/cover.jpg" : "";
writeFileSync(join(videoDir, "meta.json"), JSON.stringify(meta, null, 2));
return { meta, snippets: result.snippets, sentences, videoDir };
}
async function processVideo(videoId: string, opts: Options): Promise<VideoResult> {
const baseDir = resolveBaseDir(opts.outputDir);
// --list: always fetch fresh
if (opts.list) {
const html = await fetchHtml(videoId);
const apiKey = extractApiKey(html, videoId);
const data = await fetchInnertubeData(videoId, apiKey);
const title = data?.videoDetails?.title || "";
const captionsJson = extractCaptionsJson(data, videoId);
const transcripts = buildTranscriptList(captionsJson);
return { videoId, title, content: formatListOutput(videoId, title, transcripts) };
}
let videoDir = lookupVideoDir(videoId, baseDir);
let meta: VideoMeta;
let snippets: Snippet[];
let sentences: Sentence[];
let needsFetch = opts.refresh || !videoDir || !hasCachedData(videoDir);
if (!needsFetch && videoDir) {
meta = loadMeta(videoDir);
snippets = loadSnippets(videoDir);
sentences = loadSentences(videoDir);
const wantLangs = opts.translate ? [opts.translate] : opts.languages;
if (!wantLangs.includes(meta.language.code)) needsFetch = true;
// Backfill chapter end times for caches created before this field existed
if (!needsFetch && meta.chapters.length > 0 && meta.chapters.some((ch: any) => ch.end === undefined)) {
for (let i = 0; i < meta.chapters.length; i++) {
meta.chapters[i].end = i < meta.chapters.length - 1
? meta.chapters[i + 1].start
: Math.max(meta.duration, meta.chapters[i].start);
}
try { writeFileSync(join(videoDir, "meta.json"), JSON.stringify(meta, null, 2)); } catch {}
}
}
if (needsFetch) {
const result = await fetchAndCache(videoId, baseDir, opts);
meta = result.meta;
snippets = result.snippets;
sentences = result.sentences;
videoDir = result.videoDir;
} else {
meta = meta!;
snippets = snippets!;
sentences = sentences!;
}
let content: string;
let ext: string;
if (opts.format === "srt") {
content = formatSrt(snippets);
ext = "srt";
} else {
content = formatMarkdown(sentences, meta, {
timestamps: opts.timestamps,
chapters: opts.chapters,
speakers: opts.speakers,
}, snippets);
ext = "md";
}
const filePath = opts.output ? resolve(opts.output) : join(videoDir!, `transcript.${ext}`);
ensureDir(filePath);
writeFileSync(filePath, content);
return { videoId, title: meta.title, filePath };
}
// --- CLI ---
function printHelp() {
console.log(`Usage: bun main.ts <video-url-or-id> [options]
Options:
--languages <codes> Language codes, comma-separated (default: en)
--format <fmt> Output format: text, srt (default: text)
--translate <code> Translate to language code
--list List available transcripts
--timestamps Include timestamps (default: on)
--no-timestamps Disable timestamps
--chapters Chapter segmentation from description
--speakers Raw transcript with metadata for speaker identification
--exclude-generated Skip auto-generated transcripts
--exclude-manually-created Skip manually created transcripts
--refresh Force re-fetch (ignore cache)
-o, --output <path> Save to specific file path
--output-dir <dir> Base output directory (default: youtube-transcript)
-h, --help Show help`);
}
function parseArgs(argv: string[]): Options | null {
const opts: Options = {
videoIds: [],
languages: ["en"],
format: "text",
translate: "",
list: false,
excludeGenerated: false,
excludeManual: false,
output: "",
outputDir: "",
timestamps: true,
chapters: false,
speakers: false,
refresh: false,
};
for (let i = 0; i < argv.length; i++) {
const arg = argv[i];
if (arg === "-h" || arg === "--help") {
printHelp();
process.exit(0);
} else if (arg === "--languages") {
const v = argv[++i];
if (v) opts.languages = v.split(",").map((s) => s.trim());
} else if (arg === "--format") {
const v = argv[++i]?.toLowerCase();
if (v === "text" || v === "srt") opts.format = v;
else {
console.error(`Invalid format: ${v}. Use: text, srt`);
return null;
}
} else if (arg === "--translate") {
opts.translate = argv[++i] || "";
} else if (arg === "--list" || arg === "--list-transcripts") {
opts.list = true;
} else if (arg === "--timestamps" || arg === "-t") {
opts.timestamps = true;
} else if (arg === "--no-timestamps") {
opts.timestamps = false;
} else if (arg === "--chapters") {
opts.chapters = true;
} else if (arg === "--speakers") {
opts.speakers = true;
} else if (arg === "--exclude-generated") {
opts.excludeGenerated = true;
} else if (arg === "--exclude-manually-created") {
opts.excludeManual = true;
} else if (arg === "--refresh") {
opts.refresh = true;
} else if (arg === "-o" || arg === "--output") {
opts.output = argv[++i] || "";
} else if (arg === "--output-dir") {
opts.outputDir = argv[++i] || "";
} else if (!arg.startsWith("-")) {
opts.videoIds.push(extractVideoId(arg));
}
}
if (opts.videoIds.length === 0) {
console.error("Error: At least one video URL or ID required");
printHelp();
return null;
}
return opts;
}
async function main() {
const opts = parseArgs(process.argv.slice(2));
if (!opts) process.exit(1);
if (opts.excludeGenerated && opts.excludeManual) {
console.error("Error: Cannot exclude both generated and manually created transcripts");
process.exit(1);
}
for (const videoId of opts.videoIds) {
try {
const r = await processVideo(videoId, opts);
if (r.error) console.error(`Error (${r.videoId}): ${r.error}`);
else if (r.filePath) console.log(r.filePath);
else if (r.content) console.log(r.content);
} catch (e) {
console.error(`Error (${videoId}): ${(e as Error).message}`);
}
}
}
main();