Compare commits

..

9 Commits

Author SHA1 Message Date
Jim Liu 宝玉 69cd89184e chore: release v1.54.0 2026-03-06 23:40:53 -06:00
Jim Liu 宝玉 5771e6d8c5 feat(baoyu-translate): add external glossary files and frontmatter transformation
Support glossary_files in EXTEND.md for loading glossaries from external
markdown tables or YAML files. Add frontmatter transformation rules to
rename source metadata fields with source prefix during translation.
2026-03-06 23:38:27 -06:00
Jim Liu 宝玉 970f5e741a feat(baoyu-post-to-wechat): default bottom citations for markdown input
Enable bottom citations by default for markdown articles. Add --no-cite
flag to all posting scripts (wechat-api, wechat-article, md-to-wechat)
to optionally disable. WeChat mp links stay inline.
2026-03-06 23:38:23 -06:00
Jim Liu 宝玉 8cea550627 feat(baoyu-markdown-to-html): add --cite option for bottom citations
Add optional --cite flag to convert ordinary external links to numbered
bottom citations. WeChat links stay inline. Also translate theme and
color descriptions from Chinese to English for consistency.
2026-03-06 23:38:19 -06:00
Jim Liu 宝玉 82029160d1 feat(baoyu-format-markdown): improve title and summary generation quality
Add style-differentiated title candidates (subversive, solution, suspense,
concrete number), prohibited patterns, hook-first principles, and stronger
summary guidelines with concrete detail requirements.
2026-03-06 23:38:15 -06:00
Jim Liu 宝玉 46af919634 chore: release v1.53.0 2026-03-06 21:20:22 -06:00
Jim Liu 宝玉 5560db595a feat(baoyu-url-to-markdown): add HTML snapshot saving and Defuddle fallback pipeline
- Save rendered HTML as sibling -captured.html file alongside markdown
- Defuddle-first conversion with automatic fallback to legacy Readability/selector extractor
- Add rawHtml, conversionMethod, fallbackReason to ConversionResult
- Log converter method and fallback reason in CLI output
2026-03-06 21:18:21 -06:00
Jim Liu 宝玉 3b031c7768 chore: release v1.52.0 2026-03-06 20:54:11 -06:00
Jim Liu 宝玉 39c7e86a8d feat(baoyu-post-to-weibo): add video support and improve upload reliability
- Add --video flag for video uploads (max 18 files total)
- Switch from clipboard paste to DOM.setFileInputFiles for uploads
- Add Chrome health check with auto-restart for unresponsive instances
- Add navigation check to ensure Weibo home page before posting
2026-03-06 20:27:18 -06:00
23 changed files with 1510 additions and 153 deletions
+1 -1
View File
@@ -6,7 +6,7 @@
},
"metadata": {
"description": "Skills shared by Baoyu for improving daily work efficiency",
"version": "1.51.2"
"version": "1.54.0"
},
"plugins": [
{
+25
View File
@@ -2,6 +2,31 @@
English | [中文](./CHANGELOG.zh.md)
## 1.54.0 - 2026-03-06
### Features
- `baoyu-format-markdown`: improve title and summary generation with style-differentiated candidates, prohibited patterns, and hook-first principles
- `baoyu-markdown-to-html`: add `--cite` option to convert ordinary external links to numbered bottom citations
- `baoyu-post-to-wechat`: enable bottom citations by default for markdown input, add `--no-cite` flag to disable
- `baoyu-translate`: support external glossary files via `glossary_files` in EXTEND.md (markdown table or YAML)
- `baoyu-translate`: add frontmatter transformation rules to rename source metadata fields with `source` prefix
## 1.53.0 - 2026-03-06
### Features
- `baoyu-url-to-markdown`: save rendered HTML snapshot as `-captured.html` alongside markdown output
- `baoyu-url-to-markdown`: Defuddle-first markdown conversion with automatic fallback to legacy Readability/selector extractor
## 1.52.0 - 2026-03-06
### Features
- `baoyu-post-to-weibo`: add video upload support via `--video` flag (max 18 files total)
- `baoyu-post-to-weibo`: switch from clipboard paste to `DOM.setFileInputFiles` for more reliable uploads
### Fixes
- `baoyu-post-to-weibo`: add Chrome health check with auto-restart for unresponsive instances
- `baoyu-post-to-weibo`: add navigation check to ensure Weibo home page before posting
## 1.51.2 - 2026-03-06
### Fixes
+25
View File
@@ -2,6 +2,31 @@
[English](./CHANGELOG.md) | 中文
## 1.54.0 - 2026-03-06
### 新功能
- `baoyu-format-markdown`:优化标题和摘要生成,支持多风格候选(颠覆型、方案型、悬念型、数字型),新增禁用模式和钩子优先原则
- `baoyu-markdown-to-html`:新增 `--cite` 选项,将普通外链转换为底部编号引用
- `baoyu-post-to-wechat`:Markdown 输入默认启用底部引用,新增 `--no-cite` 标志可关闭
- `baoyu-translate`EXTEND.md 支持 `glossary_files` 加载外部术语表文件(Markdown 表格或 YAML 格式)
- `baoyu-translate`:新增 frontmatter 转换规则,翻译时将源文章元数据字段添加 `source` 前缀
## 1.53.0 - 2026-03-06
### 新功能
- `baoyu-url-to-markdown`:将渲染后的 HTML 快照保存为 `-captured.html`,与 Markdown 文件并列输出
- `baoyu-url-to-markdown`:优先使用 Defuddle 转换,失败时自动回退到旧版 Readability/选择器提取器
## 1.52.0 - 2026-03-06
### 新功能
- `baoyu-post-to-weibo`:新增 `--video` 视频上传支持(图片+视频最多 18 个文件)
- `baoyu-post-to-weibo`:上传方式从剪贴板粘贴改为 `DOM.setFileInputFiles`,提升上传可靠性
### 修复
- `baoyu-post-to-weibo`:新增 Chrome 健康检查,无响应时自动重启
- `baoyu-post-to-weibo`:发布前检查页面是否在微博首页,避免在错误页面操作
## 1.51.2 - 2026-03-06
### 修复
+6 -3
View File
@@ -560,9 +560,9 @@ To obtain credentials:
#### baoyu-post-to-weibo
Post content to Weibo (微博). Supports regular posts with text and images, and headline articles (头条文章) with Markdown input. Uses real Chrome with CDP to bypass anti-automation.
Post content to Weibo (微博). Supports regular posts with text, images, and videos, and headline articles (头条文章) with Markdown input. Uses real Chrome with CDP to bypass anti-automation.
**Regular Posts** - Text + up to 9 images:
**Regular Posts** - Text + images/videos (max 18 files):
```bash
# Post with text
@@ -570,6 +570,9 @@ Post content to Weibo (微博). Supports regular posts with text and images, and
# Post with images
/baoyu-post-to-weibo "Check this out" --image photo.png
# Post with video
/baoyu-post-to-weibo "Watch this" --video clip.mp4
```
**Headline Articles (头条文章)** - Long-form Markdown:
@@ -674,7 +677,7 @@ Utility tools for content processing.
#### baoyu-url-to-markdown
Fetch any URL via Chrome CDP and convert to clean markdown. Supports two capture modes for different scenarios.
Fetch any URL via Chrome CDP and convert to clean markdown. Saves rendered HTML snapshot alongside the markdown, and automatically falls back to a legacy extractor when Defuddle fails.
```bash
# Auto mode (default) - capture when page loads
+6 -3
View File
@@ -560,9 +560,9 @@ WECHAT_APP_SECRET=你的AppSecret
#### baoyu-post-to-weibo
发布内容到微博。支持带图文本发布和头条文章(长篇 Markdown)。使用真实 Chrome + CDP 绕过反自动化检测。
发布内容到微博。支持文字、图片、视频发布和头条文章(长篇 Markdown)。使用真实 Chrome + CDP 绕过反自动化检测。
**普通微博** - 文字 + 最多 9 张图片:
**普通微博** - 文字 + 图片/视频(最多 18 个文件)
```bash
# 发布文字
@@ -570,6 +570,9 @@ WECHAT_APP_SECRET=你的AppSecret
# 发布带图片
/baoyu-post-to-weibo "看看这个" --image photo.png
# 发布带视频
/baoyu-post-to-weibo "看这个" --video clip.mp4
```
**头条文章** - 长篇 Markdown 文章:
@@ -674,7 +677,7 @@ AI 驱动的生成后端。
#### baoyu-url-to-markdown
通过 Chrome CDP 抓取任意 URL 并转换为干净的 Markdown。支持两种抓取模式,适应不同场景
通过 Chrome CDP 抓取任意 URL 并转换为 Markdown。同时保存渲染后的 HTML 快照,Defuddle 失败时自动回退到旧版提取器
```bash
# 自动模式(默认)- 页面加载后立即抓取
+52 -18
View File
@@ -168,29 +168,55 @@ Check for YAML frontmatter (`---` block). Create if missing.
**Title Generation:**
Generate 3 candidate titles with different angles/styles. Present to user for selection:
Whether or not a title already exists, always run the title optimization flow (unless `auto_select_title` is set).
**Preparation** — read the full text and extract:
- Core argument (one sentence: "what is this article about?")
- Most impactful opinion or conclusion
- Reader pain point or curiosity trigger
- Most memorable metaphor or golden quote
**Generate 3-4 style-differentiated candidates:**
| Style | Characteristics | Example |
|-------|----------------|---------|
| Subversive | Deny common practice, create conflict | "All de-AI-flavor prompts are wrong" |
| Solution | Give the answer directly, promise value | "One recipe to make AI write in your voice" |
| Suspense | Reveal half, spark curiosity | "It took me six months to find how to remove AI flavor" |
| Concrete number | Use numbers for credibility | "150 lines of docs taught AI my writing style" |
Present to user:
```
Pick a title:
1. [Title A] — [angle/style note]
2. [Title B] — [angle/style note]
3. [Title C] — [angle/style note]
1. [Title A] — (recommended)
2. [Title B] — [style note]
3. [Title C] — [style note]
Enter number, or type a custom title:
```
Title principles:
- Engaging, sparks reading interest
- Captures core message or most compelling angle
- Accurate, avoids clickbait
- Vary angles: e.g. story-driven, conclusion-driven, question-driven
Put the strongest hook first and mark it (recommended).
If frontmatter already has `title`, skip selection and use it. If first line is H1, extract to frontmatter as default title but still offer alternatives.
**Title principles:**
- **Hook in first 5 chars**: create information gap or cognitive conflict
- **Specific > abstract**: "150 lines" beats "a document"
- **Negation > affirmation**: "you're doing it wrong" beats "the right way"
- **Conversational**: like chatting with a friend, not a paper title
- **Max ~25 chars**: longer titles get truncated in feeds
- **Accurate, not clickbait**: the article must deliver what the title promises
**Prohibited patterns:**
- "浅谈 XX"、"关于 XX 的思考"、"XX 的探索与实践"
- "震惊!"、"万字长文"、"建议收藏"
- Pure questions without direction: "AI 写作的未来在哪里?"
If first line is H1, extract to frontmatter and remove from body. If frontmatter already has `title`, include it as context but still generate fresh candidates.
**Summary Generation:**
Generate 3 candidate summaries with different focuses. Present to user for selection:
Generate 3 candidate summaries with different angles. Present to user:
```
Pick a summary:
@@ -202,11 +228,17 @@ Pick a summary:
Enter number, or type a custom summary:
```
Summary principles:
**Summary principles:**
- 80-150 characters, precise and information-rich
- Convey article's core value, not just topic
- Vary focuses: e.g. problem-driven, result-driven, insight-driven
- Avoid generic descriptions like "本文介绍了..."
- Convey **core value** to the reader, not just the topic
- Vary angles: problem-driven, result-driven, insight-driven
- **Hook the reader**: make them want to read the full article
- Use concrete details (numbers, outcomes, specific methods) over vague descriptions
**Prohibited patterns:**
- "本文介绍了..."、"本文探讨了..."
- Pure topic description without value proposition
- Repeating the title in different words
If frontmatter already has `summary`, skip selection and use it.
@@ -240,9 +272,11 @@ Apply formatting guided by the Step 2 analysis. The goal is making the content s
**Formatting principles — what TO do:**
- Preserve the author's voice, tone, and every word
- Use bold sparingly for genuinely important points
- **Bold key conclusions and core takeaways** — the sentences a reader would highlight
- Extract parallel items from prose into lists only when the structure is clearly there
- Add headings where the topic genuinely shifts
- Add headings where the topic genuinely shifts — prefer vivid, specific headings over generic ones (e.g., "3 天搞定 vs 传统方案" over "方案对比")
- Use tables for comparisons or structured data buried in prose
- Use blockquotes for golden quotes, memorable statements, or important warnings
- Fix obvious typos (based on Step 2 findings)
### Step 5: Save Formatted File
@@ -319,7 +353,7 @@ Display a report summarizing all changes made:
- Tables created: X
- Code markers added: X
- Blockquotes added: X
- Typos fixed: X [list each: "原文" → "修正"]
- Typos fixed: X [list each: "original" → "corrected"]
**Typography Script:**
- CJK spacing: [applied/skipped]
+39 -24
View File
@@ -1,6 +1,6 @@
---
name: baoyu-markdown-to-html
description: Converts Markdown to styled HTML with WeChat-compatible themes. Supports code highlighting, math, PlantUML, footnotes, alerts, and infographics. Use when user asks for "markdown to html", "convert md to html", "md转html", or needs styled HTML output from markdown.
description: Converts Markdown to styled HTML with WeChat-compatible themes. Supports code highlighting, math, PlantUML, footnotes, alerts, infographics, and optional bottom citations for external links. Use when user asks for "markdown to html", "convert md to html", "md转html", "微信外链转底部引用", or needs styled HTML output from markdown.
---
# Markdown to HTML Converter
@@ -98,15 +98,26 @@ if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-wechat/EXTEND.md") { Select-Str
| Theme | Description |
|-------|-------------|
| `default` (Recommended) | 经典主题 - 传统排版,标题居中带底边,二级标题白字彩底 |
| `grace` | 优雅主题 - 文字阴影,圆角卡片,精致引用块 |
| `simple` | 简洁主题 - 现代极简风,不对称圆角,清爽留白 |
| `modern` | 现代主题 - 大圆角、药丸形标题、宽松行距(搭配 `--color red` 还原传统红金风格) |
| `default` (Recommended) | Classic - traditional layout, centered title with bottom border, H2 with white text on colored background |
| `grace` | Elegant - text shadow, rounded cards, refined blockquotes |
| `simple` | Minimal - modern minimalist, asymmetric rounded corners, clean whitespace |
| `modern` | Modern - large radius, pill-shaped titles, relaxed line height (pair with `--color red` for traditional red-gold style) |
### Step 1.5: Determine Citation Mode
**Default**: Off. Do not ask by default.
**Enable only if the user explicitly asks** for "微信外链转底部引用", "底部引用", "文末引用", or passes `--cite`.
**Behavior when enabled**:
- Ordinary external links are rendered with numbered superscripts and collected under a final `引用链接` section.
- `https://mp.weixin.qq.com/...` links stay as direct links and are not moved to the bottom.
- Bare links where link text equals URL stay inline.
### Step 2: Convert
```bash
${BUN_X} ${SKILL_DIR}/scripts/main.ts <markdown_file> --theme <theme>
${BUN_X} ${SKILL_DIR}/scripts/main.ts <markdown_file> --theme <theme> [--cite]
```
### Step 3: Report Result
@@ -128,6 +139,7 @@ ${BUN_X} ${SKILL_DIR}/scripts/main.ts <markdown_file> [options]
| `--font-family <name>` | Font: sans, serif, serif-cjk, mono, or CSS value | theme default |
| `--font-size <N>` | Font size: 14px, 15px, 16px, 17px, 18px | 16px |
| `--title <title>` | Override title from frontmatter | |
| `--cite` | Convert external links to bottom citations, append `引用链接` section | false (off) |
| `--keep-title` | Keep the first heading in content | false (removed) |
| `--help` | Show help | |
@@ -135,19 +147,19 @@ ${BUN_X} ${SKILL_DIR}/scripts/main.ts <markdown_file> [options]
| Name | Hex | Label |
|------|-----|-------|
| blue | #0F4C81 | 经典蓝 |
| green | #009874 | 翡翠绿 |
| vermilion | #FA5151 | 活力橘 |
| yellow | #FECE00 | 柠檬黄 |
| purple | #92617E | 薰衣紫 |
| sky | #55C9EA | 天空蓝 |
| rose | #B76E79 | 玫瑰金 |
| olive | #556B2F | 橄榄绿 |
| black | #333333 | 石墨黑 |
| gray | #A9A9A9 | 雾烟灰 |
| pink | #FFB7C5 | 樱花粉 |
| red | #A93226 | 中国红 |
| orange | #D97757 | 暖橘 (modern default) |
| blue | #0F4C81 | Classic Blue |
| green | #009874 | Emerald Green |
| vermilion | #FA5151 | Vibrant Vermilion |
| yellow | #FECE00 | Lemon Yellow |
| purple | #92617E | Lavender Purple |
| sky | #55C9EA | Sky Blue |
| rose | #B76E79 | Rose Gold |
| olive | #556B2F | Olive Green |
| black | #333333 | Graphite Black |
| gray | #A9A9A9 | Smoke Gray |
| pink | #FFB7C5 | Sakura Pink |
| red | #A93226 | China Red |
| orange | #D97757 | Warm Orange (modern default) |
**Examples:**
@@ -161,6 +173,9 @@ ${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --theme grace
# Theme with custom color
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --theme modern --color red
# Enable bottom citations for ordinary external links
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --cite
# Keep the first heading in content
${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --keep-title
@@ -200,10 +215,10 @@ ${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --title "My Article"
| Theme | Description |
|-------|-------------|
| `default` | 经典主题 - 传统排版,标题居中带底边,二级标题白字彩底 |
| `grace` | 优雅主题 - 文字阴影,圆角卡片,精致引用块 (by @brzhang) |
| `simple` | 简洁主题 - 现代极简风,不对称圆角,清爽留白 (by @okooo5km) |
| `modern` | 现代主题 - 大圆角、药丸形标题、宽松行距(搭配 `--color red` 还原传统红金风格) |
| `default` | Classic - traditional layout, centered title with bottom border, H2 with white text on colored background |
| `grace` | Elegant - text shadow, rounded cards, refined blockquotes (by @brzhang) |
| `simple` | Minimal - modern minimalist, asymmetric rounded corners, clean whitespace (by @okooo5km) |
| `modern` | Modern - large radius, pill-shaped titles, relaxed line height (pair with `--color red` for traditional red-gold style) |
## Supported Markdown Features
@@ -215,7 +230,7 @@ ${BUN_X} ${SKILL_DIR}/scripts/main.ts article.md --title "My Article"
| Inline code | `` `code` `` |
| Tables | GitHub-flavored markdown tables |
| Images | `![alt](src)` |
| Links | `[text](url)` with footnote references |
| Links | `[text](url)`; add `--cite` to move ordinary external links into bottom references |
| Blockquotes | `> quote` |
| Lists | `-` unordered, `1.` ordered |
| Alerts | `> [!NOTE]`, `> [!WARNING]`, etc. |
+13 -4
View File
@@ -122,11 +122,15 @@ function parseFrontmatter(content: string): { frontmatter: Record<string, string
return { frontmatter, body: match[2]! };
}
export async function convertMarkdown(markdownPath: string, options?: { title?: string; theme?: string; keepTitle?: boolean }): Promise<ParsedResult> {
export async function convertMarkdown(
markdownPath: string,
options?: { title?: string; theme?: string; keepTitle?: boolean; citeStatus?: boolean }
): Promise<ParsedResult> {
const baseDir = path.dirname(markdownPath);
const content = fs.readFileSync(markdownPath, 'utf-8');
const theme = options?.theme ?? 'default';
const keepTitle = options?.keepTitle ?? false;
const citeStatus = options?.citeStatus ?? false;
const { frontmatter, body } = parseFrontmatter(content);
@@ -191,7 +195,7 @@ export async function convertMarkdown(markdownPath: string, options?: { title?:
const modifiedMarkdown = `---\n${Object.entries(frontmatter).map(([k, v]) => `${k}: ${v}`).join('\n')}\n---\n${modifiedBody}`;
console.error(`[markdown-to-html] Rendering with theme: ${theme}, keepTitle: ${keepTitle}`);
console.error(`[markdown-to-html] Rendering with theme: ${theme}, keepTitle: ${keepTitle}, citeStatus: ${citeStatus}`);
const themeDefaults = THEME_STYLE_DEFAULTS[theme] ?? {};
const style: StyleConfig = { ...DEFAULT_STYLE, ...themeDefaults };
@@ -199,7 +203,7 @@ export async function convertMarkdown(markdownPath: string, options?: { title?:
const css = normalizeThemeCss(buildCss(baseCss, themeCss, style));
const codeThemeCss = loadCodeThemeCss('github');
const renderer = initRenderer({});
const renderer = initRenderer({ citeStatus });
const { html: baseHtml, readingTime } = renderMarkdown(modifiedMarkdown, renderer);
let htmlContent = postProcessHtml(baseHtml, readingTime, renderer);
if (!keepTitle) htmlContent = removeFirstHeading(htmlContent);
@@ -263,6 +267,7 @@ Usage:
Options:
--title <title> Override title
--theme <name> Theme name (default, grace, simple). Default: default
--cite Convert ordinary external links to bottom citations. Default: off
--keep-title Keep the first heading in content. Default: false (removed)
--help Show this help
@@ -284,6 +289,7 @@ Output JSON format:
Example:
npx -y bun main.ts article.md
npx -y bun main.ts article.md --theme grace
npx -y bun main.ts article.md --cite
`);
process.exit(0);
}
@@ -297,6 +303,7 @@ async function main(): Promise<void> {
let markdownPath: string | undefined;
let title: string | undefined;
let theme: string | undefined;
let citeStatus = false;
let keepTitle = false;
for (let i = 0; i < args.length; i++) {
@@ -305,6 +312,8 @@ async function main(): Promise<void> {
title = args[++i];
} else if (arg === '--theme' && args[i + 1]) {
theme = args[++i];
} else if (arg === '--cite') {
citeStatus = true;
} else if (arg === '--keep-title') {
keepTitle = true;
} else if (!arg.startsWith('-')) {
@@ -322,7 +331,7 @@ async function main(): Promise<void> {
process.exit(1);
}
const result = await convertMarkdown(markdownPath, { title, theme, keepTitle });
const result = await convertMarkdown(markdownPath, { title, theme, keepTitle, citeStatus });
console.log(JSON.stringify(result, null, 2));
}
+8 -3
View File
@@ -1,6 +1,6 @@
---
name: baoyu-post-to-wechat
description: Posts content to WeChat Official Account (微信公众号) via API or Chrome CDP. Supports article posting (文章) with HTML, markdown, or plain text input, and image-text posting (贴图, formerly 图文) with multiple images. Use when user mentions "发布公众号", "post to wechat", "微信公众号", or "贴图/图文/文章".
description: Posts content to WeChat Official Account (微信公众号) via API or Chrome CDP. Supports article posting (文章) with HTML, markdown, or plain text input, and image-text posting (贴图, formerly 图文) with multiple images. Markdown article workflows default to converting ordinary external links into bottom citations for WeChat-friendly output. Use when user mentions "发布公众号", "post to wechat", "微信公众号", or "贴图/图文/文章".
---
# Post to WeChat Official Account
@@ -252,10 +252,15 @@ WECHAT_APP_SECRET=<user_input>
**CRITICAL**: Publishing scripts handle markdown conversion internally. Do NOT pre-convert markdown to HTML — pass the original markdown file directly. This ensures the API method renders images as `<img>` tags (for API upload) while the browser method uses placeholders (for paste-and-replace workflow).
**Markdown citation default**:
- For markdown input, ordinary external links are converted to bottom citations by default.
- Use `--no-cite` only if the user explicitly wants to keep ordinary external links inline.
- Existing HTML input is left as-is; no extra citation conversion is applied.
**API method** (accepts `.md` or `.html`):
```bash
${BUN_X} ${SKILL_DIR}/scripts/wechat-api.ts <file> --theme <theme> [--color <color>] [--title <title>] [--summary <summary>] [--author <author>] [--cover <cover_path>]
${BUN_X} ${SKILL_DIR}/scripts/wechat-api.ts <file> --theme <theme> [--color <color>] [--title <title>] [--summary <summary>] [--author <author>] [--cover <cover_path>] [--no-cite]
```
**CRITICAL**: Always include `--theme` parameter. Never omit it, even if using `default`. Only include `--color` if explicitly set by user or EXTEND.md.
@@ -274,7 +279,7 @@ If script parameters do not expose the two comment fields, still ensure final AP
**Browser method** (accepts `--markdown` or `--html`):
```bash
${BUN_X} ${SKILL_DIR}/scripts/wechat-article.ts --markdown <markdown_file> --theme <theme> [--color <color>]
${BUN_X} ${SKILL_DIR}/scripts/wechat-article.ts --markdown <markdown_file> --theme <theme> [--color <color>] [--no-cite]
${BUN_X} ${SKILL_DIR}/scripts/wechat-article.ts --html <html_file>
```
@@ -11,6 +11,9 @@ ${BUN_X} ./scripts/wechat-article.ts --markdown article.md
# With theme
${BUN_X} ./scripts/wechat-article.ts --markdown article.md --theme grace
# Disable bottom citations for ordinary external links
${BUN_X} ./scripts/wechat-article.ts --markdown article.md --no-cite
# With explicit options
${BUN_X} ./scripts/wechat-article.ts --markdown article.md --author "作者名" --summary "摘要"
```
@@ -21,6 +24,7 @@ ${BUN_X} ./scripts/wechat-article.ts --markdown article.md --author "作者名"
|-----------|-------------|
| `--markdown <path>` | Markdown file to convert and post |
| `--theme <name>` | Theme: default, grace, simple, modern |
| `--no-cite` | Keep ordinary external links inline instead of converting them to bottom citations |
| `--title <text>` | Override title (auto-extracted from markdown) |
| `--author <name>` | Author name |
| `--summary <text>` | Article summary |
@@ -51,6 +55,8 @@ Regular paragraph with **bold** and *italic*.
[Link text](https://example.com)
```
Markdown mode converts ordinary external links into bottom citations by default for WeChat-friendly output. Use `--no-cite` to disable that behavior.
## Image Handling
1. **Parse**: Images in markdown are replaced with `WECHATIMGPH_N`
@@ -112,10 +112,14 @@ function parseFrontmatter(content: string): { frontmatter: Record<string, string
return { frontmatter, body: match[2]! };
}
export async function convertMarkdown(markdownPath: string, options?: { title?: string; theme?: string; color?: string }): Promise<ParsedResult> {
export async function convertMarkdown(
markdownPath: string,
options?: { title?: string; theme?: string; color?: string; citeStatus?: boolean }
): Promise<ParsedResult> {
const baseDir = path.dirname(markdownPath);
const content = fs.readFileSync(markdownPath, 'utf-8');
const theme = options?.theme ?? 'default';
const citeStatus = options?.citeStatus ?? true;
const { frontmatter, body } = parseFrontmatter(content);
@@ -185,8 +189,9 @@ export async function convertMarkdown(markdownPath: string, options?: { title?:
const renderArgs = ['-y', 'bun', renderScript, tempMdPath, '--theme', theme];
if (options?.color) renderArgs.push('--color', options.color);
if (citeStatus) renderArgs.push('--cite');
console.error(`[md-to-wechat] Rendering markdown with theme: ${theme}${options?.color ? `, color: ${options.color}` : ''}`);
console.error(`[md-to-wechat] Rendering markdown with theme: ${theme}${options?.color ? `, color: ${options.color}` : ''}, citeStatus: ${citeStatus}`);
const result = spawnSync('npx', renderArgs, {
stdio: ['inherit', 'pipe', 'pipe'],
@@ -232,6 +237,7 @@ Options:
--title <title> Override title
--theme <name> Theme name (default, grace, simple, modern)
--color <name|hex> Primary color (blue, green, vermilion, etc. or hex)
--no-cite Disable bottom citations for ordinary external links
--help Show this help
Output JSON format:
@@ -251,6 +257,7 @@ Example:
npx -y bun md-to-wechat.ts article.md
npx -y bun md-to-wechat.ts article.md --theme grace
npx -y bun md-to-wechat.ts article.md --theme modern --color blue
npx -y bun md-to-wechat.ts article.md --no-cite
`);
process.exit(0);
}
@@ -265,6 +272,7 @@ async function main(): Promise<void> {
let title: string | undefined;
let theme: string | undefined;
let color: string | undefined;
let citeStatus = true;
for (let i = 0; i < args.length; i++) {
const arg = args[i]!;
@@ -274,6 +282,10 @@ async function main(): Promise<void> {
theme = args[++i];
} else if (arg === '--color' && args[i + 1]) {
color = args[++i];
} else if (arg === '--cite') {
citeStatus = true;
} else if (arg === '--no-cite') {
citeStatus = false;
} else if (!arg.startsWith('-')) {
markdownPath = arg;
}
@@ -289,7 +301,7 @@ async function main(): Promise<void> {
process.exit(1);
}
const result = await convertMarkdown(markdownPath, { title, theme, color });
const result = await convertMarkdown(markdownPath, { title, theme, color, citeStatus });
console.log(JSON.stringify(result, null, 2));
}
@@ -307,7 +307,12 @@ function parseFrontmatter(content: string): { frontmatter: Record<string, string
return { frontmatter, body: match[2]! };
}
function renderMarkdownToHtml(markdownPath: string, theme: string = "default", color?: string): string {
function renderMarkdownToHtml(
markdownPath: string,
theme: string = "default",
color?: string,
citeStatus: boolean = true
): string {
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const renderScript = path.join(__dirname, "md", "render.ts");
@@ -315,8 +320,9 @@ function renderMarkdownToHtml(markdownPath: string, theme: string = "default", c
const renderArgs = ["-y", "bun", renderScript, markdownPath, "--theme", theme];
if (color) renderArgs.push("--color", color);
if (citeStatus) renderArgs.push("--cite");
console.error(`[wechat-api] Rendering markdown with theme: ${theme}${color ? `, color: ${color}` : ""}`);
console.error(`[wechat-api] Rendering markdown with theme: ${theme}${color ? `, color: ${color}` : ""}, citeStatus: ${citeStatus}`);
const result = spawnSync("npx", renderArgs, {
stdio: ["inherit", "pipe", "pipe"],
cwd: baseDir,
@@ -362,6 +368,7 @@ Options:
--theme <name> Theme name for markdown (default, grace, simple, modern). Default: default
--color <name|hex> Primary color (blue, green, vermilion, etc. or hex)
--cover <path> Cover image path (local or URL)
--no-cite Disable bottom citations for ordinary external links in markdown mode
--dry-run Parse and render only, don't publish
--help Show this help
@@ -390,6 +397,7 @@ Example:
npx -y bun wechat-api.ts article.html --title "My Article"
npx -y bun wechat-api.ts images/ --type newspic --title "Photo Album"
npx -y bun wechat-api.ts article.md --dry-run
npx -y bun wechat-api.ts article.md --no-cite
`);
process.exit(0);
}
@@ -404,6 +412,7 @@ interface CliArgs {
theme: string;
color?: string;
cover?: string;
citeStatus: boolean;
dryRun: boolean;
}
@@ -417,6 +426,7 @@ function parseArgs(argv: string[]): CliArgs {
isHtml: false,
articleType: "news",
theme: "default",
citeStatus: true,
dryRun: false,
};
@@ -439,6 +449,10 @@ function parseArgs(argv: string[]): CliArgs {
args.color = argv[++i];
} else if (arg === "--cover" && argv[i + 1]) {
args.cover = argv[++i];
} else if (arg === "--cite") {
args.citeStatus = true;
} else if (arg === "--no-cite") {
args.citeStatus = false;
} else if (arg === "--dry-run") {
args.dryRun = true;
} else if (arg.startsWith("--") && argv[i + 1] && !argv[i + 1]!.startsWith("-")) {
@@ -513,8 +527,8 @@ async function main(): Promise<void> {
if (!author) author = frontmatter.author || "";
if (!digest) digest = frontmatter.digest || frontmatter.summary || frontmatter.description || "";
console.error(`[wechat-api] Theme: ${args.theme}${args.color ? `, color: ${args.color}` : ""}`);
htmlPath = renderMarkdownToHtml(filePath, args.theme, args.color);
console.error(`[wechat-api] Theme: ${args.theme}${args.color ? `, color: ${args.color}` : ""}, citeStatus: ${args.citeStatus}`);
htmlPath = renderMarkdownToHtml(filePath, args.theme, args.color, args.citeStatus);
console.error(`[wechat-api] HTML generated: ${htmlPath}`);
htmlContent = extractHtmlContent(htmlPath);
}
@@ -20,6 +20,7 @@ interface ArticleOptions {
markdownFile?: string;
theme?: string;
color?: string;
citeStatus?: boolean;
author?: string;
summary?: string;
images?: string[];
@@ -182,13 +183,19 @@ async function pasteFromClipboardInEditor(session: ChromeSession): Promise<void>
await sleep(1000);
}
async function parseMarkdownWithPlaceholders(markdownPath: string, theme?: string, color?: string): Promise<{ title: string; author: string; summary: string; htmlPath: string; contentImages: ImageInfo[] }> {
async function parseMarkdownWithPlaceholders(
markdownPath: string,
theme?: string,
color?: string,
citeStatus: boolean = true
): Promise<{ title: string; author: string; summary: string; htmlPath: string; contentImages: ImageInfo[] }> {
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const mdToWechatScript = path.join(__dirname, 'md-to-wechat.ts');
const args = ['-y', 'bun', mdToWechatScript, markdownPath];
if (theme) args.push('--theme', theme);
if (color) args.push('--color', color);
if (!citeStatus) args.push('--no-cite');
const result = spawnSync('npx', args, { stdio: ['inherit', 'pipe', 'pipe'] });
if (result.status !== 0) {
@@ -383,7 +390,7 @@ async function removeExtraEmptyLineAfterImage(session: ChromeSession): Promise<b
}
export async function postArticle(options: ArticleOptions): Promise<void> {
const { title, content, htmlFile, markdownFile, theme, color, author, summary, images = [], submit = false, profileDir, cdpPort } = options;
const { title, content, htmlFile, markdownFile, theme, color, citeStatus = true, author, summary, images = [], submit = false, profileDir, cdpPort } = options;
let { contentImages = [] } = options;
let effectiveTitle = title || '';
let effectiveAuthor = author || '';
@@ -392,7 +399,7 @@ export async function postArticle(options: ArticleOptions): Promise<void> {
if (markdownFile) {
console.log(`[wechat] Parsing markdown: ${markdownFile}`);
const parsed = await parseMarkdownWithPlaceholders(markdownFile, theme, color);
const parsed = await parseMarkdownWithPlaceholders(markdownFile, theme, color, citeStatus);
effectiveTitle = effectiveTitle || parsed.title;
effectiveAuthor = effectiveAuthor || parsed.author;
effectiveSummary = effectiveSummary || parsed.summary;
@@ -677,6 +684,7 @@ Options:
--markdown <path> Markdown file to convert and post (recommended)
--theme <name> Theme for markdown (default, grace, simple, modern)
--color <name|hex> Primary color (blue, green, vermilion, etc. or hex)
--no-cite Disable bottom citations for ordinary external links in markdown mode
--author <name> Author name
--summary <text> Article summary
--image <path> Content image, can repeat (only with --content)
@@ -687,13 +695,15 @@ Options:
Examples:
npx -y bun wechat-article.ts --markdown article.md
npx -y bun wechat-article.ts --markdown article.md --theme grace --submit
npx -y bun wechat-article.ts --markdown article.md --no-cite
npx -y bun wechat-article.ts --title "标题" --content "内容" --image img.png
npx -y bun wechat-article.ts --title "标题" --html article.html --submit
Markdown mode:
Images in markdown are converted to placeholders. After pasting HTML,
each placeholder is selected, scrolled into view, deleted, and replaced
with the actual image via paste.
with the actual image via paste. Ordinary external links are converted to
bottom citations by default.
`);
process.exit(0);
}
@@ -709,6 +719,7 @@ async function main(): Promise<void> {
let markdownFile: string | undefined;
let theme: string | undefined;
let color: string | undefined;
let citeStatus = true;
let author: string | undefined;
let summary: string | undefined;
let submit = false;
@@ -723,6 +734,8 @@ async function main(): Promise<void> {
else if (arg === '--markdown' && args[i + 1]) markdownFile = args[++i];
else if (arg === '--theme' && args[i + 1]) theme = args[++i];
else if (arg === '--color' && args[i + 1]) color = args[++i];
else if (arg === '--cite') citeStatus = true;
else if (arg === '--no-cite') citeStatus = false;
else if (arg === '--author' && args[i + 1]) author = args[++i];
else if (arg === '--summary' && args[i + 1]) summary = args[++i];
else if (arg === '--image' && args[i + 1]) images.push(args[++i]!);
@@ -734,7 +747,7 @@ async function main(): Promise<void> {
if (!markdownFile && !htmlFile && !title) { console.error('Error: --title is required (or use --markdown/--html)'); process.exit(1); }
if (!markdownFile && !htmlFile && !content) { console.error('Error: --content, --html, or --markdown is required'); process.exit(1); }
await postArticle({ title: title || '', content, htmlFile, markdownFile, theme, color, author, summary, images, submit, profileDir, cdpPort });
await postArticle({ title: title || '', content, htmlFile, markdownFile, theme, color, citeStatus, author, summary, images, submit, profileDir, cdpPort });
}
await main().then(() => {
+6 -4
View File
@@ -1,11 +1,11 @@
---
name: baoyu-post-to-weibo
description: Posts content to Weibo (微博). Supports regular posts with text and images, and headline articles (头条文章) with Markdown input via Chrome CDP. Use when user asks to "post to Weibo", "发微博", "发布微博", "publish to Weibo", "share on Weibo", "写微博", or "微博头条文章".
description: Posts content to Weibo (微博). Supports regular posts with text, images, and videos, and headline articles (头条文章) with Markdown input via Chrome CDP. Use when user asks to "post to Weibo", "发微博", "发布微博", "publish to Weibo", "share on Weibo", "写微博", or "微博头条文章".
---
# Post to Weibo
Posts text, images, and long-form articles to Weibo via real Chrome browser (bypasses anti-bot detection).
Posts text, images, videos, and long-form articles to Weibo via real Chrome browser (bypasses anti-bot detection).
## Script Directory
@@ -69,17 +69,19 @@ if (Test-Path "$HOME/.baoyu-skills/baoyu-post-to-weibo/EXTEND.md") { "user" }
## Regular Posts
Text + up to 9 images. Posted on Weibo homepage.
Text + images/videos (max 18 files total). Posted on Weibo homepage.
```bash
${BUN_X} ${SKILL_DIR}/scripts/weibo-post.ts "Hello Weibo!" --image ./photo.png
${BUN_X} ${SKILL_DIR}/scripts/weibo-post.ts "Watch this" --video ./clip.mp4
```
**Parameters**:
| Parameter | Description |
|-----------|-------------|
| `<text>` | Post content (positional) |
| `--image <path>` | Image file (repeatable, max 9) |
| `--image <path>` | Image file (repeatable) |
| `--video <path>` | Video file (repeatable) |
| `--profile <dir>` | Custom Chrome profile |
**Note**: Script opens browser with content filled in. User reviews and publishes manually.
@@ -1,47 +1,48 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs';
import { mkdir } from 'node:fs/promises';
import path from 'node:path';
import process from 'node:process';
import {
CdpConnection,
copyImageToClipboard,
findChromeExecutable,
findExistingChromeDebugPort,
getDefaultProfileDir,
getFreePort,
pasteFromClipboard,
killChromeByProfile,
sleep,
waitForChromeDebugPort,
} from './weibo-utils.js';
const WEIBO_HOME_URL = 'https://weibo.com/';
const MAX_FILES = 18;
interface WeiboPostOptions {
text?: string;
images?: string[];
videos?: string[];
timeoutMs?: number;
profileDir?: string;
chromePath?: string;
}
export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
const { text, images = [], timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const { text, images = [], videos = [], timeoutMs = 120_000, profileDir = getDefaultProfileDir() } = options;
const allFiles = [...images, ...videos];
if (allFiles.length > MAX_FILES) {
throw new Error(`Too many files: ${allFiles.length} (max ${MAX_FILES})`);
}
await mkdir(profileDir, { recursive: true });
const existingPort = findExistingChromeDebugPort(profileDir);
let port: number;
const chromePath = options.chromePath ?? findChromeExecutable();
if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
if (existingPort) {
console.log(`[weibo-post] Found existing Chrome on port ${existingPort}, reusing...`);
port = existingPort;
} else {
const chromePath = options.chromePath ?? findChromeExecutable();
if (!chromePath) throw new Error('Chrome not found. Set WEIBO_BROWSER_CHROME_PATH env var.');
port = await getFreePort();
const launchChrome = async (): Promise<number> => {
const port = await getFreePort();
console.log(`[weibo-post] Launching Chrome (profile: ${profileDir})`);
const chromeArgs = [
`--remote-debugging-port=${port}`,
`--user-data-dir=${profileDir}`,
@@ -51,13 +52,35 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
'--start-maximized',
WEIBO_HOME_URL,
];
if (process.platform === 'darwin') {
const appPath = chromePath.replace(/\/Contents\/MacOS\/Google Chrome$/, '');
spawn('open', ['-na', appPath, '--args', ...chromeArgs], { stdio: 'ignore' });
} else {
spawn(chromePath, chromeArgs, { stdio: 'ignore' });
}
return port;
};
let port: number;
const existingPort = findExistingChromeDebugPort(profileDir);
if (existingPort) {
console.log(`[weibo-post] Found existing Chrome on port ${existingPort}, checking health...`);
try {
const wsUrl = await waitForChromeDebugPort(existingPort, 5_000);
const testCdp = await CdpConnection.connect(wsUrl, 5_000, { defaultTimeoutMs: 5_000 });
await testCdp.send('Target.getTargets');
testCdp.close();
console.log('[weibo-post] Existing Chrome is responsive, reusing.');
port = existingPort;
} catch {
console.log('[weibo-post] Existing Chrome unresponsive, restarting...');
killChromeByProfile(profileDir);
await sleep(2000);
port = await launchChrome();
}
} else {
port = await launchChrome();
}
let cdp: CdpConnection | null = null;
@@ -76,10 +99,23 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
const { sessionId } = await cdp.send<{ sessionId: string }>('Target.attachToTarget', { targetId: pageTarget.targetId, flatten: true });
await cdp.send('Target.activateTarget', { targetId: pageTarget.targetId });
await cdp.send('Page.enable', {}, { sessionId });
await cdp.send('Runtime.enable', {}, { sessionId });
await cdp.send('Input.setIgnoreInputEvents', { ignore: false }, { sessionId });
const currentUrl = await cdp.send<{ result: { value: string } }>('Runtime.evaluate', {
expression: `window.location.href`,
returnByValue: true,
}, { sessionId });
if (!currentUrl.result.value.includes('weibo.com/') || currentUrl.result.value.includes('card.weibo.com')) {
console.log('[weibo-post] Navigating to Weibo home...');
await cdp.send('Page.navigate', { url: WEIBO_HOME_URL }, { sessionId });
await sleep(3000);
}
console.log('[weibo-post] Waiting for Weibo editor...');
await sleep(3000);
@@ -145,56 +181,45 @@ export async function postToWeibo(options: WeiboPostOptions): Promise<void> {
}
}
for (const imagePath of images) {
if (!fs.existsSync(imagePath)) {
console.warn(`[weibo-post] Image not found: ${imagePath}`);
continue;
if (allFiles.length > 0) {
const missing = allFiles.filter((f) => !fs.existsSync(f));
if (missing.length > 0) {
throw new Error(`Files not found: ${missing.join(', ')}`);
}
console.log(`[weibo-post] Pasting image: ${imagePath}`);
const absolutePaths = allFiles.map((f) => path.resolve(f));
console.log(`[weibo-post] Uploading ${absolutePaths.length} file(s) via file input...`);
if (!copyImageToClipboard(imagePath)) {
console.warn(`[weibo-post] Failed to copy image to clipboard: ${imagePath}`);
continue;
}
await cdp.send('DOM.enable', {}, { sessionId });
await sleep(500);
const { root } = await cdp.send<{ root: { nodeId: number } }>('DOM.getDocument', {}, { sessionId });
await cdp.send('Runtime.evaluate', {
expression: `document.querySelector('#homeWrap textarea')?.focus()`,
const { nodeId } = await cdp.send<{ nodeId: number }>('DOM.querySelector', {
nodeId: root.nodeId,
selector: '#homeWrap input[type="file"]',
}, { sessionId });
await sleep(200);
// Count images before paste
const imgCountBefore = await cdp.send<{ result: { value: number } }>('Runtime.evaluate', {
expression: `document.querySelectorAll('#homeWrap img[src^="blob:"], #homeWrap img[src^="data:"]').length`,
if (!nodeId || nodeId === 0) {
throw new Error('File input not found. Make sure the Weibo compose area is visible.');
}
await cdp.send('DOM.setFileInputFiles', {
nodeId,
files: absolutePaths,
}, { sessionId });
console.log('[weibo-post] Files set on input. Waiting for upload...');
await sleep(2000);
const uploadCheck = await cdp.send<{ result: { value: number } }>('Runtime.evaluate', {
expression: `document.querySelectorAll('#homeWrap img[src^="blob:"], #homeWrap img[src^="data:"], #homeWrap video').length`,
returnByValue: true,
}, { sessionId });
console.log('[weibo-post] Pasting from clipboard...');
pasteFromClipboard('Google Chrome', 5, 500);
// Verify image appeared
console.log('[weibo-post] Verifying image upload...');
const expectedImgCount = imgCountBefore.result.value + 1;
let imgUploadOk = false;
const imgWaitStart = Date.now();
while (Date.now() - imgWaitStart < 15_000) {
const r = await cdp!.send<{ result: { value: number } }>('Runtime.evaluate', {
expression: `document.querySelectorAll('#homeWrap img[src^="blob:"], #homeWrap img[src^="data:"]').length`,
returnByValue: true,
}, { sessionId });
if (r.result.value >= expectedImgCount) {
imgUploadOk = true;
break;
}
await sleep(1000);
}
if (imgUploadOk) {
console.log('[weibo-post] Image upload verified');
if (uploadCheck.result.value > 0) {
console.log(`[weibo-post] Upload verified (${uploadCheck.result.value} media item(s) detected)`);
} else {
console.warn('[weibo-post] Image upload not detected after 15s. Check Accessibility permissions.');
console.warn('[weibo-post] Upload may still be in progress. Please verify in browser.');
}
}
@@ -215,14 +240,18 @@ Usage:
npx -y bun weibo-post.ts [options] [text]
Options:
--image <path> Add image (can be repeated, max 9)
--image <path> Add image (can be repeated)
--video <path> Add video (can be repeated)
--profile <dir> Chrome profile directory
--help Show this help
Max ${MAX_FILES} files total (images + videos combined).
Examples:
npx -y bun weibo-post.ts "Hello from CLI!"
npx -y bun weibo-post.ts "Check this out" --image ./screenshot.png
npx -y bun weibo-post.ts "Post it!" --image a.png --image b.png
npx -y bun weibo-post.ts "Watch this" --video ./clip.mp4
`);
process.exit(0);
}
@@ -232,6 +261,7 @@ async function main(): Promise<void> {
if (args.includes('--help') || args.includes('-h')) printUsage();
const images: string[] = [];
const videos: string[] = [];
let profileDir: string | undefined;
const textParts: string[] = [];
@@ -239,6 +269,8 @@ async function main(): Promise<void> {
const arg = args[i]!;
if (arg === '--image' && args[i + 1]) {
images.push(args[++i]!);
} else if (arg === '--video' && args[i + 1]) {
videos.push(args[++i]!);
} else if (arg === '--profile' && args[i + 1]) {
profileDir = args[++i];
} else if (!arg.startsWith('-')) {
@@ -248,12 +280,12 @@ async function main(): Promise<void> {
const text = textParts.join(' ').trim() || undefined;
if (!text && images.length === 0) {
console.error('Error: Provide text or at least one image.');
if (!text && images.length === 0 && videos.length === 0) {
console.error('Error: Provide text or at least one image/video.');
process.exit(1);
}
await postToWeibo({ text, images, profileDir });
await postToWeibo({ text, images, videos, profileDir });
}
await main().catch((err) => {
@@ -53,6 +53,20 @@ export function findExistingChromeDebugPort(profileDir: string): number | null {
return null;
}
export function killChromeByProfile(profileDir: string): void {
try {
const result = spawnSync('ps', ['aux'], { encoding: 'utf-8', timeout: 5000 });
if (result.status !== 0 || !result.stdout) return;
for (const line of result.stdout.split('\n')) {
if (!line.includes(profileDir) || !line.includes('--remote-debugging-port=')) continue;
const pidMatch = line.trim().split(/\s+/)[1];
if (pidMatch) {
try { process.kill(Number(pidMatch), 'SIGTERM'); } catch {}
}
}
} catch {}
}
export function getDefaultProfileDir(): string {
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.WEIBO_BROWSER_PROFILE_DIR?.trim();
if (override) return path.resolve(override);
+3 -2
View File
@@ -41,7 +41,7 @@ if (Test-Path "$HOME/.baoyu-skills/baoyu-translate/EXTEND.md") { "user" }
| Found | Read, parse, apply settings. On first use in session, briefly remind: "Using preferences from [path]. You can edit EXTEND.md to customize glossary, audience, etc." |
| Not found | **MUST** run first-time setup (see below) — do NOT silently use defaults |
**EXTEND.md Supports**: Default target language | Default mode | Target audience | Custom glossaries | Translation style | Chunk settings
**EXTEND.md Supports**: Default target language | Default mode | Target audience | Custom glossaries (inline or file path) | Translation style | Chunk settings
Schema: [references/config/extend-schema.md](references/config/extend-schema.md)
@@ -135,7 +135,7 @@ Custom audience descriptions are also accepted, e.g., `--audience "AI感兴趣
1.2 Load built-in glossary for the language pair if available:
- EN→ZH: [references/glossary-en-zh.md](references/glossary-en-zh.md)
1.3 Merge glossaries: EXTEND.md glossary + built-in glossary + `--glossary` file (CLI overrides all)
1.3 Merge glossaries: EXTEND.md `glossary` (inline) + EXTEND.md `glossary_files` (external files, paths relative to EXTEND.md location) + built-in glossary + `--glossary` file (CLI overrides all)
### Step 2: Materialize Source & Create Output Directory
@@ -201,6 +201,7 @@ Before translating chunks:
- **Natural flow**: Use idiomatic target language word order and sentence patterns; break or restructure sentences freely when the source structure doesn't work naturally in the target language
- **Terminology**: Use standard translations; annotate with original term in parentheses on first occurrence
- **Preserve format**: Keep all markdown formatting (headings, bold, italic, images, links, code blocks)
- **Frontmatter transformation**: If the source has YAML frontmatter, preserve it in the translation with these changes: (1) Rename metadata fields that describe the *source* article — `url``sourceUrl`, `title``sourceTitle`, `description``sourceDescription`, `author``sourceAuthor`, `date``sourceDate`, and any similar origin-metadata fields — by adding a `source` prefix (camelCase). (2) Translate the values of text fields (title, description, etc.) and add them as new top-level fields. (3) Keep other fields (tags, categories, custom fields) as-is, translating their values where appropriate
- **Respect original**: Maintain original meaning and intent; do not add, remove, or editorialize — but sentence structure and imagery may be adapted freely to serve the meaning
- **Translator's notes**: For terms, concepts, or cultural references that target readers may not understand — due to jargon, cultural gaps, or domain-specific knowledge — add a concise explanatory note in parentheses immediately after the term. The note should explain *what it means* in plain language, not just provide the English original. Format: `译文(English original,通俗解释)`. Calibrate annotation depth to the target audience: general readers need more notes than technical readers. Only add notes where genuinely needed; do not over-annotate obvious terms.
@@ -25,6 +25,7 @@ chunk_max_words: 5000
# Custom glossary (merged with built-in glossary)
# CLI --glossary flag overrides these
# Supports inline entries and/or file paths
glossary:
- from: "Reinforcement Learning"
to: "强化学习"
@@ -32,6 +33,14 @@ glossary:
to: "Transformer"
note: "Keep English"
# Load glossary from external file(s)
# Supports absolute path or relative to EXTEND.md location
# File format: markdown table with | from | to | note | columns,
# or YAML list of {from, to, note} entries
glossary_files:
- ./my-glossary.md
- /path/to/shared-glossary.yaml
# Language-pair specific glossaries
glossaries:
en-zh:
@@ -52,7 +61,8 @@ glossaries:
| `style` | string | `storytelling` | Translation style (`storytelling` / `formal` / `technical` / `literal` / `academic` / `business` / `humorous` / `conversational` / `elegant` / custom) |
| `chunk_threshold` | number | `4000` | Word count threshold to trigger chunked translation |
| `chunk_max_words` | number | `5000` | Max words per chunk |
| `glossary` | array | `[]` | Universal glossary entries |
| `glossary` | array | `[]` | Universal glossary entries (inline) |
| `glossary_files` | array | `[]` | External glossary file paths (absolute or relative to EXTEND.md) |
| `glossaries` | object | `{}` | Language-pair specific glossary entries |
## Glossary Entry
@@ -63,11 +73,35 @@ glossaries:
| `to` | yes | Target translation |
| `note` | no | Usage note (e.g., "Keep English", "Only in tech context") |
## Glossary File Format
External glossary files (`glossary_files`) support two formats:
**Markdown table** (`.md`):
```markdown
| from | to | note |
|------|----|------|
| Reinforcement Learning | 强化学习 | |
| Transformer | Transformer | Keep English |
```
**YAML list** (`.yaml` / `.yml`):
```yaml
- from: "Reinforcement Learning"
to: "强化学习"
- from: "Transformer"
to: "Transformer"
note: "Keep English"
```
Paths can be absolute or relative to the EXTEND.md file location.
## Priority
1. CLI `--glossary` file entries
2. EXTEND.md `glossaries[pair]` entries
3. EXTEND.md `glossary` entries
4. Built-in glossary (e.g., `references/glossary-en-zh.md`)
3. EXTEND.md `glossary` entries (inline)
4. EXTEND.md `glossary_files` entries (in listed order, later files override earlier)
5. Built-in glossary (e.g., `references/glossary-en-zh.md`)
Later entries override earlier ones for the same source term.
+30 -4
View File
@@ -1,11 +1,11 @@
---
name: baoyu-url-to-markdown
description: Fetch any URL and convert to markdown using Chrome CDP. Supports two modes - auto-capture on page load, or wait for user signal (for pages requiring login). Use when user wants to save a webpage as markdown.
description: Fetch any URL and convert to markdown using Chrome CDP. Saves the rendered HTML snapshot alongside the markdown, and automatically falls back to the pre-Defuddle HTML-to-Markdown pipeline when Defuddle fails. Supports two modes - auto-capture on page load, or wait for user signal (for pages requiring login). Use when user wants to save a webpage as markdown.
---
# URL to Markdown
Fetches any URL via Chrome CDP and converts HTML to clean markdown.
Fetches any URL via Chrome CDP, saves the rendered HTML snapshot, and converts it to clean markdown.
## Script Directory
@@ -21,6 +21,7 @@ Fetches any URL via Chrome CDP and converts HTML to clean markdown.
| Script | Purpose |
|--------|---------|
| `scripts/main.ts` | CLI entry point for URL fetching |
| `scripts/html-to-markdown.ts` | Defuddle-first conversion with automatic legacy fallback |
## Preferences (EXTEND.md)
@@ -101,7 +102,9 @@ Full reference: [references/config/first-time-setup.md](references/config/first-
- Chrome CDP for full JavaScript rendering
- Two capture modes: auto or wait-for-user
- Save rendered HTML as a sibling `-captured.html` file
- Clean markdown output with metadata
- Defuddle-first markdown conversion with automatic fallback to the pre-Defuddle extractor from git history
- Handles login-required pages via wait mode
- Download images and videos to local directories
@@ -149,13 +152,23 @@ ${BUN_X} ${SKILL_DIR}/scripts/main.ts <url> --download-media
## Output Format
YAML front matter with `url`, `title`, `description`, `author`, `published`, `captured_at` fields, followed by converted markdown content.
Each run saves two files side by side:
- Markdown: YAML front matter with `url`, `title`, `description`, `author`, `published`, optional `coverImage`, and `captured_at`, followed by converted markdown content
- HTML snapshot: `*-captured.html`, containing the rendered page HTML captured from Chrome
The HTML snapshot is saved before any markdown media localization, so it stays a faithful capture of the page DOM used for conversion.
## Output Directory
Default: `url-to-markdown/<domain>/<slug>.md`
With `--output-dir ./posts/`: `./posts/<domain>/<slug>.md`
HTML snapshot path uses the same basename:
- `url-to-markdown/<domain>/<slug>-captured.html`
- `./posts/<domain>/<slug>-captured.html`
- `<slug>`: From page title or URL path (kebab-case, 2-6 words)
- Conflict resolution: Append timestamp `<slug>-YYYYMMDD-HHMMSS.md`
@@ -164,6 +177,19 @@ When `--download-media` is enabled:
- Videos are saved to `videos/` next to the markdown file
- Markdown media links are rewritten to local relative paths
## Conversion Fallback
Conversion order:
1. Try Defuddle first
2. If Defuddle throws, cannot load, returns obviously incomplete markdown, or captures lower-quality content than the legacy pipeline, automatically fall back to the pre-Defuddle extractor
3. The fallback path uses the older Readability/selector/Next.js-data based HTML-to-Markdown implementation recovered from git history
CLI output will show:
- `Converter: defuddle` when Defuddle succeeds
- `Converter: legacy:...` plus `Fallback used: ...` when fallback was needed
## Media Download Workflow
Based on `download_media` setting in EXTEND.md:
@@ -193,7 +219,7 @@ Based on `download_media` setting in EXTEND.md:
| `URL_DATA_DIR` | Custom data directory |
| `URL_CHROME_PROFILE_DIR` | Custom Chrome profile directory |
**Troubleshooting**: Chrome not found → set `URL_CHROME_PATH`. Timeout → increase `--timeout`. Complex pages → try `--wait` mode.
**Troubleshooting**: Chrome not found → set `URL_CHROME_PATH`. Timeout → increase `--timeout`. Complex pages → try `--wait` mode. If markdown quality is poor, inspect the saved `-captured.html` and check whether the run logged a legacy fallback.
## Extension Support
@@ -0,0 +1,191 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "baoyu-url-to-markdown-scripts",
"dependencies": {
"@mozilla/readability": "^0.6.0",
"defuddle": "^0.10.0",
"jsdom": "^24.1.3",
"linkedom": "^0.18.12",
"turndown": "^7.2.2",
"turndown-plugin-gfm": "^1.0.2",
},
},
},
"packages": {
"@asamuzakjp/css-color": ["@asamuzakjp/css-color@3.2.0", "", { "dependencies": { "@csstools/css-calc": "^2.1.3", "@csstools/css-color-parser": "^3.0.9", "@csstools/css-parser-algorithms": "^3.0.4", "@csstools/css-tokenizer": "^3.0.3", "lru-cache": "^10.4.3" } }, "sha512-K1A6z8tS3XsmCMM86xoWdn7Fkdn9m6RSVtocUrJYIwZnFVkng/PvkEoWtOWmP+Scc6saYWHWZYbndEEXxl24jw=="],
"@csstools/color-helpers": ["@csstools/color-helpers@5.1.0", "", {}, "sha512-S11EXWJyy0Mz5SYvRmY8nJYTFFd1LCNV+7cXyAgQtOOuzb4EsgfqDufL+9esx72/eLhsRdGZwaldu/h+E4t4BA=="],
"@csstools/css-calc": ["@csstools/css-calc@2.1.4", "", { "peerDependencies": { "@csstools/css-parser-algorithms": "^3.0.5", "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-3N8oaj+0juUw/1H3YwmDDJXCgTB1gKU6Hc/bB502u9zR0q2vd786XJH9QfrKIEgFlZmhZiq6epXl4rHqhzsIgQ=="],
"@csstools/css-color-parser": ["@csstools/css-color-parser@3.1.0", "", { "dependencies": { "@csstools/color-helpers": "^5.1.0", "@csstools/css-calc": "^2.1.4" }, "peerDependencies": { "@csstools/css-parser-algorithms": "^3.0.5", "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-nbtKwh3a6xNVIp/VRuXV64yTKnb1IjTAEEh3irzS+HkKjAOYLTGNb9pmVNntZ8iVBHcWDA2Dof0QtPgFI1BaTA=="],
"@csstools/css-parser-algorithms": ["@csstools/css-parser-algorithms@3.0.5", "", { "peerDependencies": { "@csstools/css-tokenizer": "^3.0.4" } }, "sha512-DaDeUkXZKjdGhgYaHNJTV9pV7Y9B3b644jCLs9Upc3VeNGg6LWARAT6O+Q+/COo+2gg/bM5rhpMAtf70WqfBdQ=="],
"@csstools/css-tokenizer": ["@csstools/css-tokenizer@3.0.4", "", {}, "sha512-Vd/9EVDiu6PPJt9yAh6roZP6El1xHrdvIVGjyBsHR0RYwNHgL7FJPyIIW4fANJNG6FtyZfvlRPpFI4ZM/lubvw=="],
"@mixmark-io/domino": ["@mixmark-io/domino@2.2.0", "", {}, "sha512-Y28PR25bHXUg88kCV7nivXrP2Nj2RueZ3/l/jdx6J9f8J4nsEGcgX0Qe6lt7Pa+J79+kPiJU3LguR6O/6zrLOw=="],
"@mozilla/readability": ["@mozilla/readability@0.6.0", "", {}, "sha512-juG5VWh4qAivzTAeMzvY9xs9HY5rAcr2E4I7tiSSCokRFi7XIZCAu92ZkSTsIj1OPceCifL3cpfteP3pDT9/QQ=="],
"@xmldom/xmldom": ["@xmldom/xmldom@0.8.11", "", {}, "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw=="],
"agent-base": ["agent-base@7.1.4", "", {}, "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ=="],
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"boolbase": ["boolbase@1.0.0", "", {}, "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="],
"call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="],
"combined-stream": ["combined-stream@1.0.8", "", { "dependencies": { "delayed-stream": "~1.0.0" } }, "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg=="],
"commander": ["commander@12.1.0", "", {}, "sha512-Vw8qHK3bZM9y/P10u3Vib8o/DdkvA2OtPtZvD871QKjy74Wj1WSKFILMPRPSdUSx5RFK1arlJzEtA4PkFgnbuA=="],
"css-select": ["css-select@5.2.2", "", { "dependencies": { "boolbase": "^1.0.0", "css-what": "^6.1.0", "domhandler": "^5.0.2", "domutils": "^3.0.1", "nth-check": "^2.0.1" } }, "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw=="],
"css-what": ["css-what@6.2.2", "", {}, "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA=="],
"cssom": ["cssom@0.5.0", "", {}, "sha512-iKuQcq+NdHqlAcwUY0o/HL69XQrUaQdMjmStJ8JFmUaiiQErlhrmuigkg/CU4E2J0IyUKUrMAgl36TvN67MqTw=="],
"cssstyle": ["cssstyle@4.6.0", "", { "dependencies": { "@asamuzakjp/css-color": "^3.2.0", "rrweb-cssom": "^0.8.0" } }, "sha512-2z+rWdzbbSZv6/rhtvzvqeZQHrBaqgogqt85sqFNbabZOuFbCVFb8kPeEtZjiKkbrm395irpNKiYeFeLiQnFPg=="],
"data-urls": ["data-urls@5.0.0", "", { "dependencies": { "whatwg-mimetype": "^4.0.0", "whatwg-url": "^14.0.0" } }, "sha512-ZYP5VBHshaDAiVZxjbRVcFJpc+4xGgT0bK3vzy1HLN8jTO975HEbuYzZJcHoQEY5K1a0z8YayJkyVETa08eNTg=="],
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
"decimal.js": ["decimal.js@10.6.0", "", {}, "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg=="],
"defuddle": ["defuddle@0.10.0", "", { "dependencies": { "commander": "^12.1.0" }, "optionalDependencies": { "mathml-to-latex": "^1.5.0", "temml": "^0.13.1", "turndown": "^7.2.0" }, "peerDependencies": { "jsdom": "^24.0.0" }, "bin": { "defuddle": "dist/cli.js" } }, "sha512-a43juTtHv6Vs4+sxvahVLM5NxoyDsarO1Ag3UxLORI4Fo/nsNFwzDxuQBvosKVGTIRxCwN/mfnWAzNXmQfieqw=="],
"delayed-stream": ["delayed-stream@1.0.0", "", {}, "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ=="],
"dom-serializer": ["dom-serializer@2.0.0", "", { "dependencies": { "domelementtype": "^2.3.0", "domhandler": "^5.0.2", "entities": "^4.2.0" } }, "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg=="],
"domelementtype": ["domelementtype@2.3.0", "", {}, "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw=="],
"domhandler": ["domhandler@5.0.3", "", { "dependencies": { "domelementtype": "^2.3.0" } }, "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w=="],
"domutils": ["domutils@3.2.2", "", { "dependencies": { "dom-serializer": "^2.0.0", "domelementtype": "^2.3.0", "domhandler": "^5.0.3" } }, "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw=="],
"dunder-proto": ["dunder-proto@1.0.1", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.1", "es-errors": "^1.3.0", "gopd": "^1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="],
"entities": ["entities@6.0.1", "", {}, "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g=="],
"es-define-property": ["es-define-property@1.0.1", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="],
"es-errors": ["es-errors@1.3.0", "", {}, "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw=="],
"es-object-atoms": ["es-object-atoms@1.1.1", "", { "dependencies": { "es-errors": "^1.3.0" } }, "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA=="],
"es-set-tostringtag": ["es-set-tostringtag@2.1.0", "", { "dependencies": { "es-errors": "^1.3.0", "get-intrinsic": "^1.2.6", "has-tostringtag": "^1.0.2", "hasown": "^2.0.2" } }, "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA=="],
"form-data": ["form-data@4.0.5", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.2", "mime-types": "^2.1.12" } }, "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w=="],
"function-bind": ["function-bind@1.1.2", "", {}, "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="],
"get-intrinsic": ["get-intrinsic@1.3.0", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="],
"get-proto": ["get-proto@1.0.1", "", { "dependencies": { "dunder-proto": "^1.0.1", "es-object-atoms": "^1.0.0" } }, "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g=="],
"gopd": ["gopd@1.2.0", "", {}, "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg=="],
"has-symbols": ["has-symbols@1.1.0", "", {}, "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ=="],
"has-tostringtag": ["has-tostringtag@1.0.2", "", { "dependencies": { "has-symbols": "^1.0.3" } }, "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw=="],
"hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"html-encoding-sniffer": ["html-encoding-sniffer@4.0.0", "", { "dependencies": { "whatwg-encoding": "^3.1.1" } }, "sha512-Y22oTqIU4uuPgEemfz7NDJz6OeKf12Lsu+QC+s3BVpda64lTiMYCyGwg5ki4vFxkMwQdeZDl2adZoqUgdFuTgQ=="],
"html-escaper": ["html-escaper@3.0.3", "", {}, "sha512-RuMffC89BOWQoY0WKGpIhn5gX3iI54O6nRA0yC124NYVtzjmFWBIiFd8M0x+ZdX0P9R4lADg1mgP8C7PxGOWuQ=="],
"htmlparser2": ["htmlparser2@10.1.0", "", { "dependencies": { "domelementtype": "^2.3.0", "domhandler": "^5.0.3", "domutils": "^3.2.2", "entities": "^7.0.1" } }, "sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ=="],
"http-proxy-agent": ["http-proxy-agent@7.0.2", "", { "dependencies": { "agent-base": "^7.1.0", "debug": "^4.3.4" } }, "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig=="],
"https-proxy-agent": ["https-proxy-agent@7.0.6", "", { "dependencies": { "agent-base": "^7.1.2", "debug": "4" } }, "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw=="],
"iconv-lite": ["iconv-lite@0.6.3", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw=="],
"is-potential-custom-element-name": ["is-potential-custom-element-name@1.0.1", "", {}, "sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ=="],
"jsdom": ["jsdom@24.1.3", "", { "dependencies": { "cssstyle": "^4.0.1", "data-urls": "^5.0.0", "decimal.js": "^10.4.3", "form-data": "^4.0.0", "html-encoding-sniffer": "^4.0.0", "http-proxy-agent": "^7.0.2", "https-proxy-agent": "^7.0.5", "is-potential-custom-element-name": "^1.0.1", "nwsapi": "^2.2.12", "parse5": "^7.1.2", "rrweb-cssom": "^0.7.1", "saxes": "^6.0.0", "symbol-tree": "^3.2.4", "tough-cookie": "^4.1.4", "w3c-xmlserializer": "^5.0.0", "webidl-conversions": "^7.0.0", "whatwg-encoding": "^3.1.1", "whatwg-mimetype": "^4.0.0", "whatwg-url": "^14.0.0", "ws": "^8.18.0", "xml-name-validator": "^5.0.0" }, "peerDependencies": { "canvas": "^2.11.2" }, "optionalPeers": ["canvas"] }, "sha512-MyL55p3Ut3cXbeBEG7Hcv0mVM8pp8PBNWxRqchZnSfAiES1v1mRnMeFfaHWIPULpwsYfvO+ZmMZz5tGCnjzDUQ=="],
"linkedom": ["linkedom@0.18.12", "", { "dependencies": { "css-select": "^5.1.0", "cssom": "^0.5.0", "html-escaper": "^3.0.3", "htmlparser2": "^10.0.0", "uhyphen": "^0.2.0" }, "peerDependencies": { "canvas": ">= 2" }, "optionalPeers": ["canvas"] }, "sha512-jalJsOwIKuQJSeTvsgzPe9iJzyfVaEJiEXl+25EkKevsULHvMJzpNqwvj1jOESWdmgKDiXObyjOYwlUqG7wo1Q=="],
"lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="],
"math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],
"mathml-to-latex": ["mathml-to-latex@1.5.0", "", { "dependencies": { "@xmldom/xmldom": "^0.8.10" } }, "sha512-rrWn0eEvcEcdMM4xfHcSGIy+i01DX9byOdXTLWg+w1iJ6O6ohP5UXY1dVzNUZLhzfl3EGcRekWLhY7JT5Omaew=="],
"mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
"mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
"nth-check": ["nth-check@2.1.1", "", { "dependencies": { "boolbase": "^1.0.0" } }, "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w=="],
"nwsapi": ["nwsapi@2.2.23", "", {}, "sha512-7wfH4sLbt4M0gCDzGE6vzQBo0bfTKjU7Sfpqy/7gs1qBfYz2vEJH6vXcBKpO3+6Yu1telwd0t9HpyOoLEQQbIQ=="],
"parse5": ["parse5@7.3.0", "", { "dependencies": { "entities": "^6.0.0" } }, "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw=="],
"psl": ["psl@1.15.0", "", { "dependencies": { "punycode": "^2.3.1" } }, "sha512-JZd3gMVBAVQkSs6HdNZo9Sdo0LNcQeMNP3CozBJb3JYC/QUYZTnKxP+f8oWRX4rHP5EurWxqAHTSwUCjlNKa1w=="],
"punycode": ["punycode@2.3.1", "", {}, "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg=="],
"querystringify": ["querystringify@2.2.0", "", {}, "sha512-FIqgj2EUvTa7R50u0rGsyTftzjYmv/a3hO345bZNrqabNqjtgiDMgmo4mkUjd+nzU5oF3dClKqFIPUKybUyqoQ=="],
"requires-port": ["requires-port@1.0.0", "", {}, "sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ=="],
"rrweb-cssom": ["rrweb-cssom@0.7.1", "", {}, "sha512-TrEMa7JGdVm0UThDJSx7ddw5nVm3UJS9o9CCIZ72B1vSyEZoziDqBYP3XIoi/12lKrJR8rE3jeFHMok2F/Mnsg=="],
"safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="],
"saxes": ["saxes@6.0.0", "", { "dependencies": { "xmlchars": "^2.2.0" } }, "sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA=="],
"symbol-tree": ["symbol-tree@3.2.4", "", {}, "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw=="],
"temml": ["temml@0.13.1", "", {}, "sha512-/fL1utq8QUD9YpcLeZHPRnp9Cbzbexq5hZl5uSBhf8mNYiKkcS4eYbLidDB+/nF8C+RHAcBQbKw2bKoS83mz1Q=="],
"tough-cookie": ["tough-cookie@4.1.4", "", { "dependencies": { "psl": "^1.1.33", "punycode": "^2.1.1", "universalify": "^0.2.0", "url-parse": "^1.5.3" } }, "sha512-Loo5UUvLD9ScZ6jh8beX1T6sO1w2/MpCRpEP7V280GKMVUQ0Jzar2U3UJPsrdbziLEMMhu3Ujnq//rhiFuIeag=="],
"tr46": ["tr46@5.1.1", "", { "dependencies": { "punycode": "^2.3.1" } }, "sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw=="],
"turndown": ["turndown@7.2.2", "", { "dependencies": { "@mixmark-io/domino": "^2.2.0" } }, "sha512-1F7db8BiExOKxjSMU2b7if62D/XOyQyZbPKq/nUwopfgnHlqXHqQ0lvfUTeUIr1lZJzOPFn43dODyMSIfvWRKQ=="],
"turndown-plugin-gfm": ["turndown-plugin-gfm@1.0.2", "", {}, "sha512-vwz9tfvF7XN/jE0dGoBei3FXWuvll78ohzCZQuOb+ZjWrs3a0XhQVomJEb2Qh4VHTPNRO4GPZh0V7VRbiWwkRg=="],
"uhyphen": ["uhyphen@0.2.0", "", {}, "sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA=="],
"universalify": ["universalify@0.2.0", "", {}, "sha512-CJ1QgKmNg3CwvAv/kOFmtnEN05f0D/cn9QntgNOQlQF9dgvVTHj3t+8JPdjqawCHk7V/KA+fbUqzZ9XWhcqPUg=="],
"url-parse": ["url-parse@1.5.10", "", { "dependencies": { "querystringify": "^2.1.1", "requires-port": "^1.0.0" } }, "sha512-WypcfiRhfeUP9vvF0j6rw0J3hrWrw6iZv3+22h6iRMJ/8z1Tj6XfLP4DsUix5MhMPnXpiHDoKyoZ/bdCkwBCiQ=="],
"w3c-xmlserializer": ["w3c-xmlserializer@5.0.0", "", { "dependencies": { "xml-name-validator": "^5.0.0" } }, "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA=="],
"webidl-conversions": ["webidl-conversions@7.0.0", "", {}, "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g=="],
"whatwg-encoding": ["whatwg-encoding@3.1.1", "", { "dependencies": { "iconv-lite": "0.6.3" } }, "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ=="],
"whatwg-mimetype": ["whatwg-mimetype@4.0.0", "", {}, "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg=="],
"whatwg-url": ["whatwg-url@14.2.0", "", { "dependencies": { "tr46": "^5.1.0", "webidl-conversions": "^7.0.0" } }, "sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw=="],
"ws": ["ws@8.19.0", "", { "peerDependencies": { "bufferutil": "^4.0.1", "utf-8-validate": ">=5.0.2" }, "optionalPeers": ["bufferutil", "utf-8-validate"] }, "sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg=="],
"xml-name-validator": ["xml-name-validator@5.0.0", "", {}, "sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg=="],
"xmlchars": ["xmlchars@2.2.0", "", {}, "sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw=="],
"cssstyle/rrweb-cssom": ["rrweb-cssom@0.8.0", "", {}, "sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw=="],
"dom-serializer/entities": ["entities@4.5.0", "", {}, "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw=="],
"htmlparser2/entities": ["entities@7.0.1", "", {}, "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA=="],
}
}
@@ -1,5 +1,7 @@
import { JSDOM } from "jsdom";
import { Defuddle } from "defuddle/node";
import { parseHTML } from "linkedom";
import { Readability } from "@mozilla/readability";
import TurndownService from "turndown";
import { gfm } from "turndown-plugin-gfm";
export interface PageMetadata {
url: string;
@@ -14,8 +16,103 @@ export interface PageMetadata {
export interface ConversionResult {
metadata: PageMetadata;
markdown: string;
rawHtml: string;
conversionMethod: string;
fallbackReason?: string;
}
interface ExtractionCandidate {
title: string | null;
byline: string | null;
excerpt: string | null;
published: string | null;
html: string | null;
textContent: string;
method: string;
}
type AnyRecord = Record<string, unknown>;
const MIN_CONTENT_LENGTH = 120;
const GOOD_CONTENT_LENGTH = 900;
const CONTENT_SELECTORS = [
"article",
"main article",
"[role='main'] article",
"[itemprop='articleBody']",
".article-content",
".article-body",
".post-content",
".entry-content",
".story-body",
"main",
"[role='main']",
"#content",
".content",
];
const REMOVE_SELECTORS = [
"script",
"style",
"noscript",
"template",
"iframe",
"svg",
"path",
"nav",
"aside",
"footer",
"header",
"form",
".advertisement",
".ads",
".social-share",
".related-articles",
".comments",
".newsletter",
".cookie-banner",
".cookie-consent",
"[role='navigation']",
"[aria-label*='cookie' i]",
];
const PUBLISHED_TIME_SELECTORS = [
"meta[property='article:published_time']",
"meta[name='pubdate']",
"meta[name='publishdate']",
"meta[name='date']",
"time[datetime]",
];
const ARTICLE_TYPES = new Set([
"Article",
"NewsArticle",
"BlogPosting",
"WebPage",
"ReportageNewsArticle",
]);
const NEXT_DATA_CONTENT_PATHS = [
"props.pageProps.content.body",
"props.pageProps.article.body",
"props.pageProps.article.content",
"props.pageProps.post.body",
"props.pageProps.post.content",
"props.pageProps.data.body",
"props.pageProps.story.body.content",
];
const LOW_QUALITY_MARKERS = [
/Join The Conversation/i,
/One Community\. Many Voices/i,
/Read our community guidelines/i,
/Create a free account to share your thoughts/i,
/Become a Forbes Member/i,
/Subscribe to trusted journalism/i,
/\bComments\b/i,
];
export const absolutizeUrlsScript = String.raw`
(function() {
const baseUrl = document.baseURI || location.href;
@@ -53,21 +150,791 @@ export const absolutizeUrlsScript = String.raw`
})()
`;
export async function extractContent(html: string, url: string): Promise<ConversionResult> {
const dom = new JSDOM(html, { url });
const result = await Defuddle(dom, url, { markdown: true });
function pickString(...values: unknown[]): string | null {
for (const value of values) {
if (typeof value === "string") {
const trimmed = value.trim();
if (trimmed) return trimmed;
}
}
return null;
}
const metadata: PageMetadata = {
function normalizeMarkdown(markdown: string): string {
return markdown
.replace(/\r\n/g, "\n")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
}
function parseDocument(html: string): Document {
const normalized = /<\s*html[\s>]/i.test(html)
? html
: `<!doctype html><html><body>${html}</body></html>`;
return parseHTML(normalized).document as unknown as Document;
}
function sanitizeHtml(html: string): string {
const { document } = parseHTML(`<div id="__root">${html}</div>`);
const root = document.querySelector("#__root");
if (!root) return html;
for (const selector of ["script", "style", "iframe", "noscript", "template", "svg", "path"]) {
for (const el of root.querySelectorAll(selector)) {
el.remove();
}
}
return root.innerHTML;
}
function extractTextFromHtml(html: string): string {
const { document } = parseHTML(`<!doctype html><html><body>${html}</body></html>`);
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
for (const el of document.querySelectorAll(selector)) {
el.remove();
}
}
return document.body?.textContent?.replace(/\s+/g, " ").trim() ?? "";
}
function getMetaContent(document: Document, names: string[]): string | null {
for (const name of names) {
const element =
document.querySelector(`meta[name="${name}"]`) ??
document.querySelector(`meta[property="${name}"]`);
const content = element?.getAttribute("content");
if (content && content.trim()) return content.trim();
}
return null;
}
function flattenJsonLdItems(data: unknown): AnyRecord[] {
if (!data || typeof data !== "object") return [];
if (Array.isArray(data)) return data.flatMap(flattenJsonLdItems);
const item = data as AnyRecord;
if (Array.isArray(item["@graph"])) {
return (item["@graph"] as unknown[]).flatMap(flattenJsonLdItems);
}
return [item];
}
function parseJsonLdScripts(document: Document): AnyRecord[] {
const results: AnyRecord[] = [];
const scripts = document.querySelectorAll("script[type='application/ld+json']");
for (const script of scripts) {
try {
const data = JSON.parse(script.textContent ?? "");
results.push(...flattenJsonLdItems(data));
} catch {
// Ignore malformed blocks.
}
}
return results;
}
function isArticleType(item: AnyRecord): boolean {
const value = Array.isArray(item["@type"]) ? item["@type"][0] : item["@type"];
return typeof value === "string" && ARTICLE_TYPES.has(value);
}
function extractAuthorFromJsonLd(authorData: unknown): string | null {
if (typeof authorData === "string") return authorData;
if (!authorData || typeof authorData !== "object") return null;
if (Array.isArray(authorData)) {
const names = authorData
.map((author) => extractAuthorFromJsonLd(author))
.filter((name): name is string => Boolean(name));
return names.length > 0 ? names.join(", ") : null;
}
const author = authorData as AnyRecord;
return typeof author.name === "string" ? author.name : null;
}
function extractPrimaryJsonLdMeta(document: Document): Partial<PageMetadata> {
for (const item of parseJsonLdScripts(document)) {
if (!isArticleType(item)) continue;
return {
title: pickString(item.headline, item.name) ?? undefined,
description: pickString(item.description) ?? undefined,
author: extractAuthorFromJsonLd(item.author) ?? undefined,
published: pickString(item.datePublished, item.dateCreated) ?? undefined,
coverImage:
pickString(
item.image,
(item.image as AnyRecord | undefined)?.url,
(Array.isArray(item.image) ? item.image[0] : undefined) as unknown
) ?? undefined,
};
}
return {};
}
function extractPublishedTime(document: Document): string | null {
for (const selector of PUBLISHED_TIME_SELECTORS) {
const el = document.querySelector(selector);
if (!el) continue;
const value = el.getAttribute("content") ?? el.getAttribute("datetime");
if (value && value.trim()) return value.trim();
}
return null;
}
function extractTitle(document: Document): string | null {
const ogTitle = document.querySelector("meta[property='og:title']")?.getAttribute("content");
if (ogTitle && ogTitle.trim()) return ogTitle.trim();
const twitterTitle = document.querySelector("meta[name='twitter:title']")?.getAttribute("content");
if (twitterTitle && twitterTitle.trim()) return twitterTitle.trim();
const title = document.querySelector("title")?.textContent?.trim();
if (title) {
const cleaned = title.split(/\s*[-|–—]\s*/)[0]?.trim();
if (cleaned) return cleaned;
}
const h1 = document.querySelector("h1")?.textContent?.trim();
return h1 || null;
}
function extractMetadataFromHtml(html: string, url: string, capturedAt: string): PageMetadata {
const document = parseDocument(html);
const jsonLd = extractPrimaryJsonLdMeta(document);
const timeEl = document.querySelector("time[datetime]");
return {
url,
title: result.title || "",
description: result.description || undefined,
author: result.author || undefined,
published: result.published || undefined,
coverImage: result.image || undefined,
captured_at: new Date().toISOString(),
title:
pickString(
getMetaContent(document, ["og:title", "twitter:title"]),
jsonLd.title,
document.querySelector("h1")?.textContent,
document.title
) ?? "",
description:
pickString(
getMetaContent(document, ["description", "og:description", "twitter:description"]),
jsonLd.description
) ?? undefined,
author:
pickString(
getMetaContent(document, ["author", "article:author", "twitter:creator"]),
jsonLd.author
) ?? undefined,
published:
pickString(
timeEl?.getAttribute("datetime"),
getMetaContent(document, ["article:published_time", "datePublished", "publishdate", "date"]),
jsonLd.published,
extractPublishedTime(document)
) ?? undefined,
coverImage:
pickString(
getMetaContent(document, ["og:image", "twitter:image", "twitter:image:src"]),
jsonLd.coverImage
) ?? undefined,
captured_at: capturedAt,
};
}
return { metadata, markdown: result.content || "" };
function generateExcerpt(excerpt: string | null, textContent: string | null): string | null {
if (excerpt) return excerpt;
if (!textContent) return null;
const trimmed = textContent.trim();
if (!trimmed) return null;
return trimmed.length > 200 ? `${trimmed.slice(0, 200)}...` : trimmed;
}
function parseJsonLdItem(item: AnyRecord): ExtractionCandidate | null {
if (!isArticleType(item)) return null;
const rawContent =
(typeof item.articleBody === "string" && item.articleBody) ||
(typeof item.text === "string" && item.text) ||
(typeof item.description === "string" && item.description) ||
null;
if (!rawContent) return null;
const content = rawContent.trim();
const htmlLike = /<\/?[a-z][\s\S]*>/i.test(content);
const textContent = htmlLike ? extractTextFromHtml(content) : content;
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(item.headline, item.name),
byline: extractAuthorFromJsonLd(item.author),
excerpt: pickString(item.description),
published: pickString(item.datePublished, item.dateCreated),
html: htmlLike ? content : null,
textContent,
method: "json-ld",
};
}
function tryJsonLdExtraction(document: Document): ExtractionCandidate | null {
for (const item of parseJsonLdScripts(document)) {
const extracted = parseJsonLdItem(item);
if (extracted) return extracted;
}
return null;
}
function getByPath(value: unknown, path: string): unknown {
let current = value;
for (const part of path.split(".")) {
if (!current || typeof current !== "object") return undefined;
current = (current as AnyRecord)[part];
}
return current;
}
function isContentBlockArray(value: unknown): value is AnyRecord[] {
if (!Array.isArray(value) || value.length === 0) return false;
return value.slice(0, 5).some((item) => {
if (!item || typeof item !== "object") return false;
const obj = item as AnyRecord;
return "type" in obj || "text" in obj || "textHtml" in obj || "content" in obj;
});
}
function extractTextFromContentBlocks(blocks: AnyRecord[]): string {
const parts: string[] = [];
function pushParagraph(text: string): void {
const trimmed = text.trim();
if (!trimmed) return;
parts.push(trimmed, "\n\n");
}
function walk(node: unknown): void {
if (!node || typeof node !== "object") return;
const block = node as AnyRecord;
if (typeof block.text === "string") {
pushParagraph(block.text);
return;
}
if (typeof block.textHtml === "string") {
pushParagraph(extractTextFromHtml(block.textHtml));
return;
}
if (Array.isArray(block.items)) {
for (const item of block.items) {
if (item && typeof item === "object") {
const text = pickString((item as AnyRecord).text);
if (text) parts.push(`- ${text}\n`);
}
}
parts.push("\n");
}
if (Array.isArray(block.components)) {
for (const component of block.components) {
walk(component);
}
}
if (Array.isArray(block.content)) {
for (const child of block.content) {
walk(child);
}
}
}
for (const block of blocks) {
walk(block);
}
return parts.join("").replace(/\n{3,}/g, "\n\n").trim();
}
function tryStringBodyExtraction(
content: string,
meta: AnyRecord,
document: Document,
method: string
): ExtractionCandidate | null {
if (!content || content.length < MIN_CONTENT_LENGTH) return null;
const isHtml = /<\/?[a-z][\s\S]*>/i.test(content);
const html = isHtml ? sanitizeHtml(content) : null;
const textContent = isHtml ? extractTextFromHtml(html) : content.trim();
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(meta.headline, meta.title, extractTitle(document)),
byline: pickString(meta.byline, meta.author),
excerpt: pickString(meta.description, meta.excerpt, generateExcerpt(null, textContent)),
published: pickString(meta.datePublished, meta.publishedAt, extractPublishedTime(document)),
html,
textContent,
method,
};
}
function tryNextDataExtraction(document: Document): ExtractionCandidate | null {
try {
const script = document.querySelector("script#__NEXT_DATA__");
if (!script?.textContent) return null;
const data = JSON.parse(script.textContent) as AnyRecord;
const pageProps = (getByPath(data, "props.pageProps") ?? {}) as AnyRecord;
for (const path of NEXT_DATA_CONTENT_PATHS) {
const value = getByPath(data, path);
if (typeof value === "string") {
const parentPath = path.split(".").slice(0, -1).join(".");
const parent = (getByPath(data, parentPath) ?? {}) as AnyRecord;
const meta = {
...pageProps,
...parent,
title: parent.title ?? (pageProps.title as string | undefined),
};
const candidate = tryStringBodyExtraction(value, meta, document, "next-data");
if (candidate) return candidate;
}
if (isContentBlockArray(value)) {
const textContent = extractTextFromContentBlocks(value);
if (textContent.length < MIN_CONTENT_LENGTH) continue;
return {
title: pickString(
getByPath(data, "props.pageProps.content.headline"),
getByPath(data, "props.pageProps.article.headline"),
getByPath(data, "props.pageProps.article.title"),
getByPath(data, "props.pageProps.post.title"),
pageProps.title,
extractTitle(document)
),
byline: pickString(
getByPath(data, "props.pageProps.author.name"),
getByPath(data, "props.pageProps.article.author.name")
),
excerpt: pickString(
getByPath(data, "props.pageProps.content.description"),
getByPath(data, "props.pageProps.article.description"),
pageProps.description,
generateExcerpt(null, textContent)
),
published: pickString(
getByPath(data, "props.pageProps.content.datePublished"),
getByPath(data, "props.pageProps.article.datePublished"),
getByPath(data, "props.pageProps.publishedAt"),
extractPublishedTime(document)
),
html: null,
textContent,
method: "next-data",
};
}
}
} catch {
return null;
}
return null;
}
function buildReadabilityCandidate(
article: ReturnType<Readability["parse"]>,
document: Document,
method: string
): ExtractionCandidate | null {
const textContent = article?.textContent?.trim() ?? "";
if (textContent.length < MIN_CONTENT_LENGTH) return null;
return {
title: pickString(article?.title, extractTitle(document)),
byline: pickString((article as { byline?: string } | null)?.byline),
excerpt: pickString(article?.excerpt, generateExcerpt(null, textContent)),
published: pickString((article as { publishedTime?: string } | null)?.publishedTime, extractPublishedTime(document)),
html: article?.content ? sanitizeHtml(article.content) : null,
textContent,
method,
};
}
function tryReadability(document: Document): ExtractionCandidate | null {
try {
const strictClone = document.cloneNode(true) as Document;
const strictResult = buildReadabilityCandidate(
new Readability(strictClone).parse(),
document,
"readability"
);
if (strictResult) return strictResult;
const relaxedClone = document.cloneNode(true) as Document;
return buildReadabilityCandidate(
new Readability(relaxedClone, { charThreshold: 120 }).parse(),
document,
"readability-relaxed"
);
} catch {
return null;
}
}
function trySelectorExtraction(document: Document): ExtractionCandidate | null {
for (const selector of CONTENT_SELECTORS) {
const element = document.querySelector(selector);
if (!element) continue;
const clone = element.cloneNode(true) as Element;
for (const removeSelector of REMOVE_SELECTORS) {
for (const node of clone.querySelectorAll(removeSelector)) {
node.remove();
}
}
const html = sanitizeHtml(clone.innerHTML);
const textContent = extractTextFromHtml(html);
if (textContent.length < MIN_CONTENT_LENGTH) continue;
return {
title: extractTitle(document),
byline: null,
excerpt: generateExcerpt(null, textContent),
published: extractPublishedTime(document),
html,
textContent,
method: `selector:${selector}`,
};
}
return null;
}
function tryBodyExtraction(document: Document): ExtractionCandidate | null {
const body = document.body;
if (!body) return null;
const clone = body.cloneNode(true) as Element;
for (const removeSelector of REMOVE_SELECTORS) {
for (const node of clone.querySelectorAll(removeSelector)) {
node.remove();
}
}
const html = sanitizeHtml(clone.innerHTML);
const textContent = extractTextFromHtml(html);
if (!textContent) return null;
return {
title: extractTitle(document),
byline: null,
excerpt: generateExcerpt(null, textContent),
published: extractPublishedTime(document),
html,
textContent,
method: "body-fallback",
};
}
function pickBestCandidate(candidates: ExtractionCandidate[]): ExtractionCandidate | null {
if (candidates.length === 0) return null;
const methodOrder = [
"readability",
"readability-relaxed",
"next-data",
"json-ld",
"selector:",
"body-fallback",
];
function methodRank(method: string): number {
const idx = methodOrder.findIndex((entry) =>
entry.endsWith(":") ? method.startsWith(entry) : method === entry
);
return idx === -1 ? methodOrder.length : idx;
}
const ranked = [...candidates].sort((a, b) => {
const rankA = methodRank(a.method);
const rankB = methodRank(b.method);
if (rankA !== rankB) return rankA - rankB;
return (b.textContent.length ?? 0) - (a.textContent.length ?? 0);
});
for (const candidate of ranked) {
if (candidate.textContent.length >= GOOD_CONTENT_LENGTH) {
return candidate;
}
}
for (const candidate of ranked) {
if (candidate.textContent.length >= MIN_CONTENT_LENGTH) {
return candidate;
}
}
return ranked[0];
}
function extractFromHtml(html: string): ExtractionCandidate | null {
const document = parseDocument(html);
const readabilityCandidate = tryReadability(document);
const nextDataCandidate = tryNextDataExtraction(document);
const jsonLdCandidate = tryJsonLdExtraction(document);
const selectorCandidate = trySelectorExtraction(document);
const bodyCandidate = tryBodyExtraction(document);
const candidates = [
readabilityCandidate,
nextDataCandidate,
jsonLdCandidate,
selectorCandidate,
bodyCandidate,
].filter((candidate): candidate is ExtractionCandidate => Boolean(candidate));
const winner = pickBestCandidate(candidates);
if (!winner) return null;
return {
...winner,
title: winner.title ?? extractTitle(document),
published: winner.published ?? extractPublishedTime(document),
excerpt: winner.excerpt ?? generateExcerpt(null, winner.textContent),
};
}
const turndown = new TurndownService({
headingStyle: "atx",
hr: "---",
bulletListMarker: "-",
codeBlockStyle: "fenced",
emDelimiter: "*",
strongDelimiter: "**",
linkStyle: "inlined",
});
turndown.use(gfm);
turndown.remove(["script", "style", "iframe", "noscript", "template", "svg", "path"]);
turndown.addRule("collapseFigure", {
filter: "figure",
replacement(content) {
return `\n\n${content.trim()}\n\n`;
},
});
turndown.addRule("dropInvisibleAnchors", {
filter(node) {
return node.nodeName === "A" && !(node as Element).textContent?.trim();
},
replacement() {
return "";
},
});
function convertHtmlToMarkdown(html: string): string {
if (!html || !html.trim()) return "";
try {
const sanitized = sanitizeHtml(html);
return turndown.turndown(sanitized);
} catch {
return "";
}
}
function fallbackPlainText(html: string): string {
const document = parseDocument(html);
for (const selector of ["script", "style", "noscript", "template", "iframe", "svg", "path"]) {
for (const el of document.querySelectorAll(selector)) {
el.remove();
}
}
const text = document.body?.textContent ?? document.documentElement?.textContent ?? "";
return normalizeMarkdown(text.replace(/\s+/g, " "));
}
function countBylines(markdown: string): number {
return (markdown.match(/(^|\n)By\s+/g) || []).length;
}
function countUsefulParagraphs(markdown: string): number {
const paragraphs = normalizeMarkdown(markdown).split(/\n{2,}/);
let count = 0;
for (const paragraph of paragraphs) {
const trimmed = paragraph.trim();
if (!trimmed) continue;
if (/^!?\[[^\]]*\]\([^)]+\)$/.test(trimmed)) continue;
if (/^#{1,6}\s+/.test(trimmed)) continue;
if ((trimmed.match(/\b[\p{L}\p{N}']+\b/gu) || []).length < 8) continue;
count++;
}
return count;
}
function countMarkerHits(markdown: string, markers: RegExp[]): number {
let hits = 0;
for (const marker of markers) {
if (marker.test(markdown)) hits++;
}
return hits;
}
function scoreMarkdownQuality(markdown: string): number {
const normalized = normalizeMarkdown(markdown);
const wordCount = (normalized.match(/\b[\p{L}\p{N}']+\b/gu) || []).length;
const usefulParagraphs = countUsefulParagraphs(normalized);
const headingCount = (normalized.match(/^#{1,6}\s+/gm) || []).length;
const markerHits = countMarkerHits(normalized, LOW_QUALITY_MARKERS);
const bylineCount = countBylines(normalized);
const staffCount = (normalized.match(/\bForbes Staff\b/gi) || []).length;
return (
Math.min(wordCount, 4000) +
usefulParagraphs * 40 +
headingCount * 10 -
markerHits * 180 -
Math.max(0, bylineCount - 1) * 120 -
Math.max(0, staffCount - 1) * 80
);
}
function shouldCompareWithLegacy(markdown: string): boolean {
const normalized = normalizeMarkdown(markdown);
return (
countMarkerHits(normalized, LOW_QUALITY_MARKERS) > 0 ||
countBylines(normalized) > 1 ||
countUsefulParagraphs(normalized) < 6
);
}
function isMarkdownUsable(markdown: string, html: string): boolean {
const normalized = normalizeMarkdown(markdown);
if (!normalized) return false;
const htmlTextLength = extractTextFromHtml(html).length;
if (htmlTextLength < MIN_CONTENT_LENGTH) return true;
if (normalized.length >= 80) return true;
return normalized.length >= Math.min(200, Math.floor(htmlTextLength * 0.2));
}
async function tryDefuddleConversion(
html: string,
url: string,
baseMetadata: PageMetadata
): Promise<{ ok: true; result: ConversionResult } | { ok: false; reason: string }> {
try {
const [{ JSDOM, VirtualConsole }, { Defuddle }] = await Promise.all([
import("jsdom"),
import("defuddle/node"),
]);
const virtualConsole = new VirtualConsole();
virtualConsole.on("jsdomError", (error: Error & { type?: string }) => {
if (error.type === "css parsing" || /Could not parse CSS stylesheet/i.test(error.message)) {
return;
}
console.warn(`[url-to-markdown] jsdom: ${error.message}`);
});
const dom = new JSDOM(html, { url, virtualConsole });
const result = await Defuddle(dom, url, { markdown: true });
const markdown = normalizeMarkdown(result.content || "");
if (!isMarkdownUsable(markdown, html)) {
return { ok: false, reason: "Defuddle returned empty or incomplete markdown" };
}
return {
ok: true,
result: {
metadata: {
...baseMetadata,
title: pickString(result.title, baseMetadata.title) ?? "",
description: pickString(result.description, baseMetadata.description) ?? undefined,
author: pickString(result.author, baseMetadata.author) ?? undefined,
published: pickString(result.published, baseMetadata.published) ?? undefined,
coverImage: pickString(result.image, baseMetadata.coverImage) ?? undefined,
},
markdown,
rawHtml: html,
conversionMethod: "defuddle",
},
};
} catch (error) {
return {
ok: false,
reason: error instanceof Error ? error.message : String(error),
};
}
}
function convertWithLegacyExtractor(html: string, baseMetadata: PageMetadata): ConversionResult {
const extracted = extractFromHtml(html);
let markdown = extracted?.html ? convertHtmlToMarkdown(extracted.html) : "";
if (!markdown.trim()) {
markdown = extracted?.textContent?.trim() || fallbackPlainText(html);
}
return {
metadata: {
...baseMetadata,
title: pickString(extracted?.title, baseMetadata.title) ?? "",
description: pickString(extracted?.excerpt, baseMetadata.description) ?? undefined,
author: pickString(extracted?.byline, baseMetadata.author) ?? undefined,
published: pickString(extracted?.published, baseMetadata.published) ?? undefined,
},
markdown: normalizeMarkdown(markdown),
rawHtml: html,
conversionMethod: extracted ? `legacy:${extracted.method}` : "legacy:plain-text",
};
}
export async function extractContent(html: string, url: string): Promise<ConversionResult> {
const capturedAt = new Date().toISOString();
const baseMetadata = extractMetadataFromHtml(html, url, capturedAt);
const defuddleResult = await tryDefuddleConversion(html, url, baseMetadata);
if (defuddleResult.ok) {
if (shouldCompareWithLegacy(defuddleResult.result.markdown)) {
const legacyResult = convertWithLegacyExtractor(html, baseMetadata);
const legacyScore = scoreMarkdownQuality(legacyResult.markdown);
const defuddleScore = scoreMarkdownQuality(defuddleResult.result.markdown);
if (legacyScore > defuddleScore + 120) {
return {
...legacyResult,
fallbackReason: "Legacy extractor produced higher-quality markdown than Defuddle",
};
}
}
return defuddleResult.result;
}
const fallbackResult = convertWithLegacyExtractor(html, baseMetadata);
return {
...fallbackResult,
fallbackReason: defuddleResult.reason,
};
}
function escapeYamlValue(value: string): string {
+14 -1
View File
@@ -69,6 +69,12 @@ function formatTimestamp(): string {
return `${now.getFullYear()}${pad(now.getMonth() + 1)}${pad(now.getDate())}-${pad(now.getHours())}${pad(now.getMinutes())}${pad(now.getSeconds())}`;
}
function deriveHtmlSnapshotPath(markdownPath: string): string {
const parsed = path.parse(markdownPath);
const basename = parsed.ext ? parsed.name : parsed.base;
return path.join(parsed.dir, `${basename}-captured.html`);
}
async function generateOutputPath(url: string, title: string, outputDir?: string): Promise<string> {
const domain = new URL(url).hostname.replace(/^www\./, "");
const slug = generateSlug(title, url);
@@ -141,7 +147,7 @@ async function captureUrl(args: Args): Promise<ConversionResult> {
async function main(): Promise<void> {
const args = parseArgs(process.argv);
if (!args.url) {
console.error("Usage: bun main.ts <url> [-o output.md] [--wait] [--timeout ms]");
console.error("Usage: bun main.ts <url> [-o output.md] [--output-dir dir] [--wait] [--timeout ms] [--download-media]");
process.exit(1);
}
@@ -166,7 +172,9 @@ async function main(): Promise<void> {
const result = await captureUrl(args);
const outputPath = args.output || await generateOutputPath(args.url, result.metadata.title, args.outputDir);
const outputDir = path.dirname(outputPath);
const htmlSnapshotPath = deriveHtmlSnapshotPath(outputPath);
await mkdir(outputDir, { recursive: true });
await writeFile(htmlSnapshotPath, result.rawHtml, "utf-8");
let document = createMarkdownDocument(result);
@@ -189,7 +197,12 @@ async function main(): Promise<void> {
await writeFile(outputPath, document, "utf-8");
console.log(`Saved: ${outputPath}`);
console.log(`Saved HTML: ${htmlSnapshotPath}`);
console.log(`Title: ${result.metadata.title || "(no title)"}`);
console.log(`Converter: ${result.conversionMethod}`);
if (result.fallbackReason) {
console.warn(`Fallback used: ${result.fallbackReason}`);
}
}
main().catch((err) => {
@@ -0,0 +1,13 @@
{
"name": "baoyu-url-to-markdown-scripts",
"private": true,
"type": "module",
"dependencies": {
"@mozilla/readability": "^0.6.0",
"defuddle": "^0.10.0",
"jsdom": "^24.1.3",
"linkedom": "^0.18.12",
"turndown": "^7.2.2",
"turndown-plugin-gfm": "^1.0.2"
}
}