2 lines
9.4 KiB
JSON
2 lines
9.4 KiB
JSON
{"schema":"openclaw.skill-workshop.rollback.v1","proposalId":"calibre-cleaner-20260624-0fc374efd7","writtenAt":"2026-07-09T09:56:20.398Z","targetSkillFile":"/home/yangxuan/.openclaw/workspace/storage/skills/calibre-cleaner/SKILL.md","action":"update","previousContent":"---\nname: \"calibre-cleaner\"\ndescription: \"清洗 Calibre 书库：修复损坏书名/提取作者/检测编码损坏、小文件的批量删除\"\n---\n\n# Calibre 书库清洗 Skill\n\n## 概述\n\nCalibre 书库整理三件套：\n\n1. **书名清洗** —— 修复从 txt 导入产生的 HTML 标签/长文本/乱码书名\n2. **作者提取** —— 从书名中自动提取作者名（匹配 `作者：xxx`、`-xxx`、`BY xxx` 等格式）\n3. **脏数据清理** —— 检测并删除内容损坏的 txt（中文 < 10%）以及小于 10KB 的垃圾文件\n\n## 前置条件\n\n- 书库路径：`/home/yangxuan/calibre/books`\n- 元数据文件：`/home/yangxuan/calibre/books/metadata.db`\n- 运行前须停止其他 Calibre 进程（桌面版、Docker Calibre-Web）\n- 运行前 **必须备份** `metadata.db`\n\n---\n\n## 一、书名清洗 & 作者提取\n\n### `shorten(s, max_bytes=120)`\n\n将字符串截断到指定字节数，避免因 UTF-8 中文字符跨字节而损坏。\n\n```python\ndef shorten(s, max_bytes=120):\n encoded = s.encode('utf-8')\n if len(encoded) <= max_bytes:\n return s\n try:\n return encoded[:max_bytes].decode('utf-8')\n except:\n return encoded[:max_bytes-3].decode('utf-8', errors='ignore')\n```\n\n### `clean_title(title)`\n\n```python\ndef clean_title(title):\n t = title\n # 移除 HTML 标签\n t = re.sub(r'<[^>]+>', '', t)\n # 替换 _BR_ / _BR 标记\n t = re.sub(r'_BR_', ' ', t, flags=re.IGNORECASE)\n t = re.sub(r'_BR(?![a-zA-Z])', ' ', t)\n # 移除 _fontxxx_ / _imgxxx_ 标记\n t = re.sub(r'_font[^_]*_', '', t)\n t = re.sub(r'_img[^_]*_', '', t)\n # 合并多余空格\n t = re.sub(r'\\s+', ' ', t).strip()\n # 超长截短\n if len(t) > 100:\n m = re.search(r'[\\u4e00-\\u9fff]{2,}', t)\n if m:\n t = t[m.start():]\n t = t[:60].rstrip('，。；：、！？…\\ufffd\\'\"').rstrip('.,;:!?\\'').strip()\n # 移除控制字符\n t = re.sub(r'[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f]', '', t)\n return shorten(t, 100).strip() or 'Unknown'\n```\n\n### 作者提取规则（从书名中提取）\n\n格式 | 示例 | 正则\n---|---|---\n`作者：xxx`（中文冒号） | `...作者：镜欲` | `作者[：:]\\s*([^\\s,，、\\)）]+)`\n`作者:xxx`（英文冒号） | `...作者:镜欲` | 同上\n`-xxx`（英文 dash 结尾） | `善良的美艳妈妈-糠麸` | `-([^-()【】\\[\\]]+)$`\n`翻译：xxx` | `...翻译：kkmanlg` | `翻译[：:]\\s*([^\\s,，、\\)）]+)`\n`BY xxx`（英文 BY） | `...BY狱厨` | `\\b[Bb][Yy]\\s+([^\\s,，、\\)）]+)`\n`&xxx`（and 符结尾） | `...&axenic` | `&([^&()【】\\[\\] ]+)$`\n\n提取策略：先匹配 `作者：` → `-xxx` → `翻译：` → `BY` → `&xxx`。排除项：`正文`、`内容简介`、`第\\\\d+[章节]`。\n\n```python\nEXCLUDE_AUTHORS = {'正文', '内容简介', ''}\n\ndef extract_author_from_title(title):\n m = re.search(r'作者[：:]\\s*([^\\s,，、\\)）《\\[]+)', title)\n if m:\n name = m.group(1).strip()\n if name not in EXCLUDE_AUTHORS and not re.match(r'^第\\d+[章节]', name):\n return name\n m = re.search(r'翻译[：:]\\s*([^\\s,，、\\)）《\\[]+)', title)\n if m:\n return m.group(1).strip()\n m = re.search(r'\\b[Bb][Yy]\\s+([^\\s,，、\\)）《\\[]+)', title)\n if m:\n name = m.group(1).strip()\n if name not in EXCLUDE_AUTHORS:\n return name\n m = re.search(r'-([^-()【】\\[\\]{}《》\\s]{2,30})$', title)\n if m:\n name = m.group(1).strip()\n if name not in EXCLUDE_AUTHORS and not re.match(r'^\\d+', name):\n return name\n m = re.search(r'&([^&()【】\\[\\]\\s]{2,30})$', title)\n if m:\n name = m.group(1).strip()\n if name not in EXCLUDE_AUTHORS:\n return name\n return None\n```\n\n### 坏书检测条件\n\n```python\nis_bad = (\n len(title) > 100 or\n '<img' in title or '<BR' in title or '<font' in title or\n '\\ufffd' in title or\n len(title.encode('utf-8')) > 150\n)\n```\n\n---\n\n## 二、脏数据清理\n\n### 2.1 检测条件\n\n扫描书库中所有 TXT 文件，读取前 20KB 检测中文比例：\n\n- **内容损坏/二进制垃圾**：中文比例 < 10%（标记为待删除）\n- **小文件垃圾**：`uncompressed_size < 10240`（10KB 以下）\n\n```python\nwith open(full_path, 'rb') as f:\n raw = f.read(20480)\ntext = raw.decode('utf-8', errors='replace')\nchinese = sum(1 for c in text if '\\u4e00' <= c <= '\\u9fff')\nratio = chinese / len(text) if len(text) > 0 else 0\n\nif ratio < 0.10: # 内容损坏，删除\nif size < 10240: # 小文件垃圾，删除\n```\n\n### 2.2 批量删除（通过 SQLite 直写）\n\n停掉 calibre 进程后，通过 SQLite 删除记录并清理文件系统：\n\n```sql\nDELETE FROM data WHERE book = ?\nDELETE FROM books_authors_link WHERE book = ?\nDELETE FROM books_languages_link WHERE book = ?\nDELETE FROM books_publishers_link WHERE book = ?\nDELETE FROM books_series_link WHERE book = ?\nDELETE FROM books_tags_link WHERE book = ?\nDELETE FROM comments WHERE book = ?\nDELETE FROM identifiers WHERE book = ?\nDELETE FROM books WHERE id = ?\n```\n\n同时删除对应目录：`shutil.rmtree(os.path.join('/home/yangxuan/calibre/books', bpath))`\n\n### 2.3 完整删除脚本\n\n```python\nimport sqlite3, os, shutil\n\ndb_path = '/home/yangxuan/calibre/books/metadata.db'\nconn = sqlite3.connect(db_path)\nc = conn.cursor()\n\nc.execute('''\n SELECT b.id, b.title, b.path, d.name, d.uncompressed_size\n FROM books b \n JOIN data d ON d.book = b.id \n WHERE d.format = 'TXT'\n''')\nrows = c.fetchall()\n\nto_delete = []\n\nfor bid, title, bpath, fname, size in rows:\n full_path = os.path.join('/home/yangxuan/calibre/books', bpath, fname + '.txt')\n if not os.path.exists(full_path):\n continue\n \n # 条件1：小文件垃圾\n if size < 10240:\n to_delete.append((bid, bpath))\n continue\n \n # 条件2：内容损坏（中文 < 10%）\n with open(full_path, 'rb') as f:\n raw = f.read(20480)\n text = raw.decode('utf-8', errors='replace')\n chinese = sum(1 for c in text if '\\u4e00' <= c <= '\\u9fff')\n ratio = chinese / len(text) if len(text) > 0 else 0\n if ratio < 0.10:\n to_delete.append((bid, bpath))\n\nprint(f\"待删除: {len(to_delete)}\", flush=True)\nfor bid, bpath in to_delete:\n c.execute('DELETE FROM data WHERE book=?', (bid,))\n c.execute('DELETE FROM books_authors_link WHERE book=?', (bid,))\n c.execute('DELETE FROM books_languages_link WHERE book=?', (bid,))\n c.execute('DELETE FROM books_publishers_link WHERE book=?', (bid,))\n c.execute('DELETE FROM books_series_link WHERE book=?', (bid,))\n c.execute('DELETE FROM books_tags_link WHERE book=?', (bid,))\n c.execute('DELETE FROM comments WHERE book=?', (bid,))\n c.execute('DELETE FROM identifiers WHERE book=?', (bid,))\n c.execute('DELETE FROM books WHERE id=?', (bid,))\n full_dir = os.path.join('/home/yangxuan/calibre/books', bpath)\n if os.path.exists(full_dir):\n shutil.rmtree(full_dir)\n\nconn.commit()\nconn.close()\nprint(f\"完成，删除: {len(to_delete)}\", flush=True)\n```\n\n---\n\n## 三、常用操作\n\n### 3.1 查看 XX 作者的书\n\n```bash\nsqlite3 /home/yangxuan/calibre/books/metadata.db \"\nSELECT b.id, b.title\nFROM books b JOIN books_authors_link l ON b.id = l.book\nJOIN authors a ON l.author = a.id\nWHERE a.name = 'Unknown'\nLIMIT 20;\n\"\n```\n\n### 3.2 统计作者分布\n\n```bash\nsqlite3 /home/yangxuan/calibre/books/metadata.db \"\nSELECT a.name, COUNT(*) as cnt\nFROM books b JOIN books_authors_link l ON b.id = l.book\nJOIN authors a ON l.author = a.id\nGROUP BY a.name ORDER BY cnt DESC LIMIT 20;\n\"\n```\n\n### 3.3 书库统计\n\n```bash\npython3 -c \"\nimport sqlite3\nconn = sqlite3.connect('/home/yangxuan/calibre/books/metadata.db')\nc = conn.cursor()\nc.execute('SELECT COUNT(*) FROM books')\nprint(f'总书籍: {c.fetchone()[0]}')\nc.execute('SELECT COUNT(*) FROM data WHERE format=\\\"TXT\\\"')\nprint(f'TXT 文件: {c.fetchone()[0]}')\nconn.close()\n\"\n```\n\n---\n\n## 四、操作流程（推荐顺序）\n\n```bash\ncd /home/yangxuan/calibre\n\n# 1. 停服务 + 备份\ndocker compose down\ncp books/metadata.db books/metadata.db.backup.$(date +%Y%m%d)\nkillall calibre 2>/dev/null\n\n# 2. 脏数据清理（内容损坏 + 小文件）\npython3 /path/to/clean_dirty_data.py\n\n# 3. 书名清洗 & 作者提取\ncalibre-debug /path/to/fix_titles.py\n\n# 4. 重启\ndocker compose up -d\n```\n\n---\n\n## 五、已知问题\n\n### `set_metadata` 报 `Errno 36 文件名过长`\n\n当 `author_sort` 字段超 80 字节时，Calibre 路径会触及 ext4 255 字节限制。先清理极长作者名：\n\n```bash\nsqlite3 /home/yangxuan/calibre/books/metadata.db \"\nUPDATE authors SET name='Unknown' WHERE LENGTH(name) > 30;\n\"\n```\n\n### WSL `/mnt/f` 删除 NTFS 文件权限不够\n\nWSL 的 drvfs 默认不带 metadata 选项，`sudo rm`、`os.remove`、`cmd.exe /c del` 均无法删除 NTFS 文件。不要在 WSL 下尝试跨分区删除操作，直接让 Windows 侧（文件资源管理器/PowerShell）操作。\n","previousContentHash":"929fdbb7a4de265565ca75bdf3692d33623a634f4991c21a51e5f829b0e9b2c0"}
|