--- name: "calibre-cleaner" description: "清洗 Calibre 书库：修复损坏书名/提取作者/检测编码损坏、小文件的批量删除" status: proposal version: "v1" date: "2026-06-24T03:03:28.166Z" --- # Calibre 书库清洗 Skill ## 概述 Calibre 书库整理三件套： 1. **书名清洗** —— 修复从 txt 导入产生的 HTML 标签/长文本/乱码书名 2. **作者提取** —— 从书名中自动提取作者名（匹配 `作者：xxx`、`-xxx`、`BY xxx` 等格式） 3. **脏数据清理** —— 检测并删除内容损坏的 txt（中文 < 10%）以及小于 10KB 的垃圾文件 ## 前置条件 - 书库路径：`/home/yangxuan/calibre/books` - 元数据文件：`/home/yangxuan/calibre/books/metadata.db` - 运行前须停止其他 Calibre 进程（桌面版、Docker Calibre-Web） - 运行前 **必须备份** `metadata.db` --- ## 一、书名清洗 & 作者提取 ### `shorten(s, max_bytes=120)` 将字符串截断到指定字节数，避免因 UTF-8 中文字符跨字节而损坏。 ```python def shorten(s, max_bytes=120): encoded = s.encode('utf-8') if len(encoded) <= max_bytes: return s try: return encoded[:max_bytes].decode('utf-8') except: return encoded[:max_bytes-3].decode('utf-8', errors='ignore') ``` ### `clean_title(title)` ```python def clean_title(title): t = title # 移除 HTML 标签 t = re.sub(r'<[^>]+>', '', t) # 替换 _BR_ / _BR 标记 t = re.sub(r'_BR_', ' ', t, flags=re.IGNORECASE) t = re.sub(r'_BR(?![a-zA-Z])', ' ', t) # 移除 _fontxxx_ / _imgxxx_ 标记 t = re.sub(r'_font[^_]*_', '', t) t = re.sub(r'_img[^_]*_', '', t) # 合并多余空格 t = re.sub(r'\s+', ' ', t).strip() # 超长截短 if len(t) > 100: m = re.search(r'[\u4e00-\u9fff]{2,}', t) if m: t = t[m.start():] t = t[:60].rstrip('，。；：、！？…\ufffd\'"').rstrip('.,;:!?\'').strip() # 移除控制字符 t = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f]', '', t) return shorten(t, 100).strip() or 'Unknown' ``` ### 作者提取规则（从书名中提取） 格式 | 示例 | 正则 ---|---|--- `作者：xxx`（中文冒号） | `...作者：镜欲` | `作者[：:]\s*([^\s,，、\)）]+)` `作者:xxx`（英文冒号） | `...作者:镜欲` | 同上 `-xxx`（英文 dash 结尾） | `善良的美艳妈妈-糠麸` | `-([^-()【】\[\]]+)$` `翻译：xxx` | `...翻译：kkmanlg` | `翻译[：:]\s*([^\s,，、\)）]+)` `BY xxx`（英文 BY） | `...BY狱厨` | `\b[Bb][Yy]\s+([^\s,，、\)）]+)` `&xxx`（and 符结尾） | `...&axenic` | `&([^&()【】\[\] ]+)$` 提取策略：先匹配 `作者：` → `-xxx` → `翻译：` → `BY` → `&xxx`。排除项：`正文`、`内容简介`、`第\\d+[章节]`。 ```python EXCLUDE_AUTHORS = {'正文', '内容简介', ''} def extract_author_from_title(title): m = re.search(r'作者[：:]\s*([^\s,，、\)）《\[]+)', title) if m: name = m.group(1).strip() if name not in EXCLUDE_AUTHORS and not re.match(r'^第\d+[章节]', name): return name m = re.search(r'翻译[：:]\s*([^\s,，、\)）《\[]+)', title) if m: return m.group(1).strip() m = re.search(r'\b[Bb][Yy]\s+([^\s,，、\)）《\[]+)', title) if m: name = m.group(1).strip() if name not in EXCLUDE_AUTHORS: return name m = re.search(r'-([^-()【】\[\]{}《》\s]{2,30})$', title) if m: name = m.group(1).strip() if name not in EXCLUDE_AUTHORS and not re.match(r'^\d+', name): return name m = re.search(r'&([^&()【】\[\]\s]{2,30})$', title) if m: name = m.group(1).strip() if name not in EXCLUDE_AUTHORS: return name return None ``` ### 坏书检测条件 ```python is_bad = ( len(title) > 100 or ' 150 ) ``` --- ## 二、脏数据清理 ### 2.1 检测条件 扫描书库中所有 TXT 文件，读取前 20KB 检测中文比例： - **内容损坏/二进制垃圾**：中文比例 < 10%（标记为待删除） - **小文件垃圾**：`uncompressed_size < 10240`（10KB 以下） ```python with open(full_path, 'rb') as f: raw = f.read(20480) text = raw.decode('utf-8', errors='replace') chinese = sum(1 for c in text if '\u4e00' <= c <= '\u9fff') ratio = chinese / len(text) if len(text) > 0 else 0 if ratio < 0.10: # 内容损坏，删除 if size < 10240: # 小文件垃圾，删除 ``` ### 2.2 批量删除（通过 SQLite 直写） 停掉 calibre 进程后，通过 SQLite 删除记录并清理文件系统： ```sql DELETE FROM data WHERE book = ? DELETE FROM books_authors_link WHERE book = ? DELETE FROM books_languages_link WHERE book = ? DELETE FROM books_publishers_link WHERE book = ? DELETE FROM books_series_link WHERE book = ? DELETE FROM books_tags_link WHERE book = ? DELETE FROM comments WHERE book = ? DELETE FROM identifiers WHERE book = ? DELETE FROM books WHERE id = ? ``` 同时删除对应目录：`shutil.rmtree(os.path.join('/home/yangxuan/calibre/books', bpath))` ### 2.3 完整删除脚本 ```python import sqlite3, os, shutil db_path = '/home/yangxuan/calibre/books/metadata.db' conn = sqlite3.connect(db_path) c = conn.cursor() c.execute(''' SELECT b.id, b.title, b.path, d.name, d.uncompressed_size FROM books b JOIN data d ON d.book = b.id WHERE d.format = 'TXT' ''') rows = c.fetchall() to_delete = [] for bid, title, bpath, fname, size in rows: full_path = os.path.join('/home/yangxuan/calibre/books', bpath, fname + '.txt') if not os.path.exists(full_path): continue # 条件1：小文件垃圾 if size < 10240: to_delete.append((bid, bpath)) continue # 条件2：内容损坏（中文 < 10%） with open(full_path, 'rb') as f: raw = f.read(20480) text = raw.decode('utf-8', errors='replace') chinese = sum(1 for c in text if '\u4e00' <= c <= '\u9fff') ratio = chinese / len(text) if len(text) > 0 else 0 if ratio < 0.10: to_delete.append((bid, bpath)) print(f"待删除: {len(to_delete)}", flush=True) for bid, bpath in to_delete: c.execute('DELETE FROM data WHERE book=?', (bid,)) c.execute('DELETE FROM books_authors_link WHERE book=?', (bid,)) c.execute('DELETE FROM books_languages_link WHERE book=?', (bid,)) c.execute('DELETE FROM books_publishers_link WHERE book=?', (bid,)) c.execute('DELETE FROM books_series_link WHERE book=?', (bid,)) c.execute('DELETE FROM books_tags_link WHERE book=?', (bid,)) c.execute('DELETE FROM comments WHERE book=?', (bid,)) c.execute('DELETE FROM identifiers WHERE book=?', (bid,)) c.execute('DELETE FROM books WHERE id=?', (bid,)) full_dir = os.path.join('/home/yangxuan/calibre/books', bpath) if os.path.exists(full_dir): shutil.rmtree(full_dir) conn.commit() conn.close() print(f"完成，删除: {len(to_delete)}", flush=True) ``` --- ## 三、常用操作 ### 3.1 查看 XX 作者的书 ```bash sqlite3 /home/yangxuan/calibre/books/metadata.db " SELECT b.id, b.title FROM books b JOIN books_authors_link l ON b.id = l.book JOIN authors a ON l.author = a.id WHERE a.name = 'Unknown' LIMIT 20; " ``` ### 3.2 统计作者分布 ```bash sqlite3 /home/yangxuan/calibre/books/metadata.db " SELECT a.name, COUNT(*) as cnt FROM books b JOIN books_authors_link l ON b.id = l.book JOIN authors a ON l.author = a.id GROUP BY a.name ORDER BY cnt DESC LIMIT 20; " ``` ### 3.3 书库统计 ```bash python3 -c " import sqlite3 conn = sqlite3.connect('/home/yangxuan/calibre/books/metadata.db') c = conn.cursor() c.execute('SELECT COUNT(*) FROM books') print(f'总书籍: {c.fetchone()[0]}') c.execute('SELECT COUNT(*) FROM data WHERE format=\"TXT\"') print(f'TXT 文件: {c.fetchone()[0]}') conn.close() " ``` --- ## 四、操作流程（推荐顺序） ```bash cd /home/yangxuan/calibre # 1. 停服务 + 备份 docker compose down cp books/metadata.db books/metadata.db.backup.$(date +%Y%m%d) killall calibre 2>/dev/null # 2. 脏数据清理（内容损坏 + 小文件） python3 /path/to/clean_dirty_data.py # 3. 书名清洗 & 作者提取 calibre-debug /path/to/fix_titles.py # 4. 重启 docker compose up -d ``` --- ## 五、已知问题 ### `set_metadata` 报 `Errno 36 文件名过长` 当 `author_sort` 字段超 80 字节时，Calibre 路径会触及 ext4 255 字节限制。先清理极长作者名： ```bash sqlite3 /home/yangxuan/calibre/books/metadata.db " UPDATE authors SET name='Unknown' WHERE LENGTH(name) > 30; " ``` ### WSL `/mnt/f` 删除 NTFS 文件权限不够 WSL 的 drvfs 默认不带 metadata 选项，`sudo rm`、`os.remove`、`cmd.exe /c del` 均无法删除 NTFS 文件。不要在 WSL 下尝试跨分区删除操作，直接让 Windows 侧（文件资源管理器/PowerShell）操作。