Files

T

yangxuan c7ff34d012 auto backup 2026-07-02 11:00

2026-07-02 11:00:05 +08:00

8.8 KiB

Raw Blame History

name, description, status, version, date

name	description	status	version	date
calibre-cleaner	清洗 Calibre 书库：修复损坏书名/提取作者/检测编码损坏、小文件的批量删除	proposal	v1	2026-06-24T03:03:28.166Z

Calibre 书库清洗 Skill

概述

Calibre 书库整理三件套：

书名清洗 —— 修复从 txt 导入产生的 HTML 标签/长文本/乱码书名
作者提取 —— 从书名中自动提取作者名（匹配 作者：xxx、-xxx、BY xxx 等格式）
脏数据清理 —— 检测并删除内容损坏的 txt（中文 < 10%）以及小于 10KB 的垃圾文件

前置条件

书库路径：/home/yangxuan/calibre/books
元数据文件：/home/yangxuan/calibre/books/metadata.db
运行前须停止其他 Calibre 进程（桌面版、Docker Calibre-Web）
运行前 必须备份 metadata.db

一、书名清洗 & 作者提取

`shorten(s, max_bytes=120)`

将字符串截断到指定字节数，避免因 UTF-8 中文字符跨字节而损坏。

def shorten(s, max_bytes=120):
    encoded = s.encode('utf-8')
    if len(encoded) <= max_bytes:
        return s
    try:
        return encoded[:max_bytes].decode('utf-8')
    except:
        return encoded[:max_bytes-3].decode('utf-8', errors='ignore')

`clean_title(title)`

def clean_title(title):
    t = title
    # 移除 HTML 标签
    t = re.sub(r'<[^>]+>', '', t)
    # 替换 _BR_ / _BR 标记
    t = re.sub(r'_BR_', ' ', t, flags=re.IGNORECASE)
    t = re.sub(r'_BR(?![a-zA-Z])', ' ', t)
    # 移除 _fontxxx_ / _imgxxx_ 标记
    t = re.sub(r'_font[^_]*_', '', t)
    t = re.sub(r'_img[^_]*_', '', t)
    # 合并多余空格
    t = re.sub(r'\s+', ' ', t).strip()
    # 超长截短
    if len(t) > 100:
        m = re.search(r'[\u4e00-\u9fff]{2,}', t)
        if m:
            t = t[m.start():]
        t = t[:60].rstrip('，。；：、！？…\ufffd\'"').rstrip('.,;:!?\'').strip()
    # 移除控制字符
    t = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f]', '', t)
    return shorten(t, 100).strip() or 'Unknown'

作者提取规则（从书名中提取）

格式	示例	正则
`作者：xxx`（中文冒号）	`...作者：镜欲`	`作者[：:]\s*([^\s,，、\)）]+)`
`作者:xxx`（英文冒号）	`...作者:镜欲`	同上
`-xxx`（英文 dash 结尾）	`善良的美艳妈妈-糠麸`	`-([^-()【】\[\]]+)$`
`翻译：xxx`	`...翻译：kkmanlg`	`翻译[：:]\s*([^\s,，、\)）]+)`
`BY xxx`（英文 BY）	`...BY狱厨`	`\b[Bb][Yy]\s+([^\s,，、\)）]+)`
`&xxx`（and 符结尾）	`...&axenic`	`&([^&()【】\[\] ]+)$`

提取策略：先匹配 作者： → -xxx → 翻译： → BY → &xxx。排除项：正文、内容简介、第\\d+[章节]。

EXCLUDE_AUTHORS = {'正文', '内容简介', ''}

def extract_author_from_title(title):
    m = re.search(r'作者[：:]\s*([^\s,，、\)）《\[]+)', title)
    if m:
        name = m.group(1).strip()
        if name not in EXCLUDE_AUTHORS and not re.match(r'^第\d+[章节]', name):
            return name
    m = re.search(r'翻译[：:]\s*([^\s,，、\)）《\[]+)', title)
    if m:
        return m.group(1).strip()
    m = re.search(r'\b[Bb][Yy]\s+([^\s,，、\)）《\[]+)', title)
    if m:
        name = m.group(1).strip()
        if name not in EXCLUDE_AUTHORS:
            return name
    m = re.search(r'-([^-()【】\[\]{}《》\s]{2,30})$', title)
    if m:
        name = m.group(1).strip()
        if name not in EXCLUDE_AUTHORS and not re.match(r'^\d+', name):
            return name
    m = re.search(r'&([^&()【】\[\]\s]{2,30})$', title)
    if m:
        name = m.group(1).strip()
        if name not in EXCLUDE_AUTHORS:
            return name
    return None

坏书检测条件

is_bad = (
    len(title) > 100 or
    '<img' in title or '<BR' in title or '<font' in title or
    '\ufffd' in title or
    len(title.encode('utf-8')) > 150
)

二、脏数据清理

2.1 检测条件

扫描书库中所有 TXT 文件，读取前 20KB 检测中文比例：

内容损坏/二进制垃圾：中文比例 < 10%（标记为待删除）
小文件垃圾：uncompressed_size < 10240（10KB 以下）

with open(full_path, 'rb') as f:
    raw = f.read(20480)
text = raw.decode('utf-8', errors='replace')
chinese = sum(1 for c in text if '\u4e00' <= c <= '\u9fff')
ratio = chinese / len(text) if len(text) > 0 else 0

if ratio < 0.10:   # 内容损坏，删除
if size < 10240:   # 小文件垃圾，删除

2.2 批量删除（通过 SQLite 直写）

停掉 calibre 进程后，通过 SQLite 删除记录并清理文件系统：

DELETE FROM data WHERE book = ?
DELETE FROM books_authors_link WHERE book = ?
DELETE FROM books_languages_link WHERE book = ?
DELETE FROM books_publishers_link WHERE book = ?
DELETE FROM books_series_link WHERE book = ?
DELETE FROM books_tags_link WHERE book = ?
DELETE FROM comments WHERE book = ?
DELETE FROM identifiers WHERE book = ?
DELETE FROM books WHERE id = ?

同时删除对应目录：shutil.rmtree(os.path.join('/home/yangxuan/calibre/books', bpath))

2.3 完整删除脚本

import sqlite3, os, shutil

db_path = '/home/yangxuan/calibre/books/metadata.db'
conn = sqlite3.connect(db_path)
c = conn.cursor()

c.execute('''
    SELECT b.id, b.title, b.path, d.name, d.uncompressed_size
    FROM books b 
    JOIN data d ON d.book = b.id 
    WHERE d.format = 'TXT'
''')
rows = c.fetchall()

to_delete = []

for bid, title, bpath, fname, size in rows:
    full_path = os.path.join('/home/yangxuan/calibre/books', bpath, fname + '.txt')
    if not os.path.exists(full_path):
        continue
    
    # 条件1：小文件垃圾
    if size < 10240:
        to_delete.append((bid, bpath))
        continue
    
    # 条件2：内容损坏（中文 < 10%）
    with open(full_path, 'rb') as f:
        raw = f.read(20480)
    text = raw.decode('utf-8', errors='replace')
    chinese = sum(1 for c in text if '\u4e00' <= c <= '\u9fff')
    ratio = chinese / len(text) if len(text) > 0 else 0
    if ratio < 0.10:
        to_delete.append((bid, bpath))

print(f"待删除: {len(to_delete)}", flush=True)
for bid, bpath in to_delete:
    c.execute('DELETE FROM data WHERE book=?', (bid,))
    c.execute('DELETE FROM books_authors_link WHERE book=?', (bid,))
    c.execute('DELETE FROM books_languages_link WHERE book=?', (bid,))
    c.execute('DELETE FROM books_publishers_link WHERE book=?', (bid,))
    c.execute('DELETE FROM books_series_link WHERE book=?', (bid,))
    c.execute('DELETE FROM books_tags_link WHERE book=?', (bid,))
    c.execute('DELETE FROM comments WHERE book=?', (bid,))
    c.execute('DELETE FROM identifiers WHERE book=?', (bid,))
    c.execute('DELETE FROM books WHERE id=?', (bid,))
    full_dir = os.path.join('/home/yangxuan/calibre/books', bpath)
    if os.path.exists(full_dir):
        shutil.rmtree(full_dir)

conn.commit()
conn.close()
print(f"完成，删除: {len(to_delete)}", flush=True)

三、常用操作

3.1 查看 XX 作者的书

sqlite3 /home/yangxuan/calibre/books/metadata.db "
SELECT b.id, b.title
FROM books b JOIN books_authors_link l ON b.id = l.book
JOIN authors a ON l.author = a.id
WHERE a.name = 'Unknown'
LIMIT 20;
"

3.2 统计作者分布

sqlite3 /home/yangxuan/calibre/books/metadata.db "
SELECT a.name, COUNT(*) as cnt
FROM books b JOIN books_authors_link l ON b.id = l.book
JOIN authors a ON l.author = a.id
GROUP BY a.name ORDER BY cnt DESC LIMIT 20;
"

3.3 书库统计

python3 -c "
import sqlite3
conn = sqlite3.connect('/home/yangxuan/calibre/books/metadata.db')
c = conn.cursor()
c.execute('SELECT COUNT(*) FROM books')
print(f'总书籍: {c.fetchone()[0]}')
c.execute('SELECT COUNT(*) FROM data WHERE format=\"TXT\"')
print(f'TXT 文件: {c.fetchone()[0]}')
conn.close()
"

四、操作流程（推荐顺序）

cd /home/yangxuan/calibre

# 1. 停服务 + 备份
docker compose down
cp books/metadata.db books/metadata.db.backup.$(date +%Y%m%d)
killall calibre 2>/dev/null

# 2. 脏数据清理（内容损坏 + 小文件）
python3 /path/to/clean_dirty_data.py

# 3. 书名清洗 & 作者提取
calibre-debug /path/to/fix_titles.py

# 4. 重启
docker compose up -d

五、已知问题

`set_metadata` 报 `Errno 36 文件名过长`

当 author_sort 字段超 80 字节时，Calibre 路径会触及 ext4 255 字节限制。先清理极长作者名：

sqlite3 /home/yangxuan/calibre/books/metadata.db "
UPDATE authors SET name='Unknown' WHERE LENGTH(name) > 30;
"

WSL `/mnt/f` 删除 NTFS 文件权限不够

WSL 的 drvfs 默认不带 metadata 选项，sudo rm、os.remove、cmd.exe /c del 均无法删除 NTFS 文件。不要在 WSL 下尝试跨分区删除操作，直接让 Windows 侧（文件资源管理器/PowerShell）操作。

8.8 KiB Raw Blame History Unescape Escape