451 lines
14 KiB
Markdown
451 lines
14 KiB
Markdown
---
|
||
name: "calibre-cleaner"
|
||
description: "清洗 Calibre 书库：修复损坏书名/提取作者/检测编码损坏、小文件的批量删除"
|
||
status: proposal
|
||
version: "v1"
|
||
date: "2026-06-24T03:23:27.062Z"
|
||
---
|
||
|
||
# Calibre 书库清洗 Skill
|
||
|
||
## 概述
|
||
|
||
Calibre 书库整理三件套：
|
||
|
||
1. **书名清洗** —— 修复从 txt 导入产生的 HTML 标签/长文本/乱码书名，截取前 20 个中文字符，移除网站链接
|
||
2. **作者提取** —— 从书名或原始文件名 `data.name` 中自动提取作者名（匹配 `作者：xxx`、`-xxx`、`BY xxx` 等格式）
|
||
3. **脏数据清理** —— 检测并删除内容损坏的 txt（中文 < 10%）、小于 10KB 的垃圾文件、Python 列表格式垃圾标题
|
||
|
||
## 前置条件
|
||
|
||
- 书库路径：`/home/yangxuan/calibre/books`
|
||
- 元数据文件：`/home/yangxuan/calibre/books/metadata.db`
|
||
- 运行前须停止其他 Calibre 进程（桌面版、Docker Calibre-Web）
|
||
- 运行前 **必须备份** `metadata.db`
|
||
|
||
---
|
||
|
||
## 一、书名清洗 & 作者提取
|
||
|
||
### `shorten(s, max_bytes=120)`
|
||
|
||
将字符串截断到指定字节数，避免因 UTF-8 中文字符跨字节而损坏。
|
||
|
||
```python
|
||
def shorten(s, max_bytes=120):
|
||
encoded = s.encode('utf-8')
|
||
if len(encoded) <= max_bytes:
|
||
return s
|
||
try:
|
||
return encoded[:max_bytes].decode('utf-8')
|
||
except:
|
||
return encoded[:max_bytes-3].decode('utf-8', errors='ignore')
|
||
```
|
||
|
||
### `clean_title(title)`
|
||
|
||
```python
|
||
def clean_title(title):
|
||
t = title
|
||
# 移除 HTML 标签
|
||
t = re.sub(r'<[^>]+>', '', t)
|
||
# 替换 _BR_ / _BR 标记
|
||
t = re.sub(r'_BR_', ' ', t, flags=re.IGNORECASE)
|
||
t = re.sub(r'_BR(?![a-zA-Z])', ' ', t)
|
||
# 移除 _fontxxx_ / _imgxxx_ 标记
|
||
t = re.sub(r'_font[^_]*_', '', t)
|
||
t = re.sub(r'_img[^_]*_', '', t)
|
||
# 移除网站链接
|
||
t = re.sub(r'[\(（]www\.[^)）]+[\)）]', '', t)
|
||
t = re.sub(r'^[a-zA-Z0-9.-]+\.com@', '', t)
|
||
t = re.sub(r'[-–—][a-zA-Z0-9.-]+\.com$', '', t)
|
||
t = re.sub(r'【[^】]*?(?:下载|网站|书城|声明|提供)[^】]*】', '', t)
|
||
t = re.sub(r'[：:].*?(?:本文由|本电子书).*$', '', t)
|
||
t = re.sub(r'[/／]\s*www\.[^\s]+.*$', '', t)
|
||
t = re.sub(r'[，,。]?https?://[^\s\)）\]]+', '', t)
|
||
# 移除声明文字当书名
|
||
t = re.sub(r'^内容简介[】】]?', '', t)
|
||
# 移除作者/翻译标记（批量清洗时用）
|
||
t = re.sub(r'[ \s]*?(?:作者|翻译|原?作者)[：:].*$', '', t, flags=re.DOTALL)
|
||
# 移除 Unknown/未知 尾巴
|
||
t = re.sub(r'\s*-\s*(Unknown|未知)\s*$', '', t)
|
||
# 合并多余空格
|
||
t = re.sub(r'\s+', ' ', t).strip()
|
||
# 截取前 20 个中文字符（保留非中文直到第 20 个中文出现）
|
||
chinese_count = 0
|
||
pos = 0
|
||
for i, c in enumerate(t):
|
||
if '\u4e00' <= c <= '\u9fff':
|
||
chinese_count += 1
|
||
if chinese_count == 20:
|
||
pos = i + 1
|
||
break
|
||
if chinese_count > 20:
|
||
t = t[:pos]
|
||
# 超长截短
|
||
if len(t) > 100:
|
||
m = re.search(r'[\u4e00-\u9fff]{2,}', t)
|
||
if m:
|
||
t = t[m.start():]
|
||
t = t[:60].rstrip('，。；：、！？…\ufffd\'"').rstrip('.,;:!?\'').strip()
|
||
# 移除控制字符
|
||
t = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f]', '', t)
|
||
return shorten(t, 100).strip() or 'Unknown'
|
||
```
|
||
|
||
### 作者提取规则（从文件中同时匹配 title 和 data.name）
|
||
|
||
优先从 `data.name`（原始文件名）提取，更准确：
|
||
|
||
```python
|
||
EXCLUDE_AUTHORS = {'正文', '内容简介', ''}
|
||
|
||
def extract_author_from_name(file_name):
|
||
"""从 data.name（原始文件名）提取作者"""
|
||
name = file_name
|
||
# 去掉尾巴
|
||
name = re.sub(r'\s*-\s*(Unknown|未知)\s*$', '', name)
|
||
# 匹配作者：xxx
|
||
m = re.search(r'作者[：:]\s*([^\s_）\)《\[【\]】\-,]+)', name)
|
||
if m:
|
||
candidate = m.group(1).strip()
|
||
if candidate not in EXCLUDE_AUTHORS and len(candidate) < 30:
|
||
return candidate
|
||
# 匹配翻译：xxx
|
||
m = re.search(r'翻译[：:]\s*([^\s_）\)《\[【\]】\-,]+)', name)
|
||
if m:
|
||
candidate = m.group(1).strip()
|
||
if len(candidate) < 20:
|
||
return candidate
|
||
# 匹配原作者：xxx
|
||
m = re.search(r'原?作者[：:]\s*([^\s_）\)《\[【\]】\-,]+)', name)
|
||
if m:
|
||
candidate = m.group(1).strip()
|
||
if len(candidate) < 30:
|
||
return candidate
|
||
return None
|
||
|
||
def extract_author_from_title(title):
|
||
"""从 title 提取作者（降级方案，仅在 data.name 无用时）"""
|
||
m = re.search(r'作者[：:]\s*([^\s,，、\)）《\[]+)', title)
|
||
if m:
|
||
name = m.group(1).strip()
|
||
if name not in EXCLUDE_AUTHORS and not re.match(r'^第\d+[章节]', name):
|
||
return name
|
||
m = re.search(r'翻译[：:]\s*([^\s,，、\)）《\[]+)', title)
|
||
if m:
|
||
return m.group(1).strip()
|
||
m = re.search(r'\b[Bb][Yy]\s+([^\s,，、\)）《\[]+)', title)
|
||
if m:
|
||
name = m.group(1).strip()
|
||
if name not in EXCLUDE_AUTHORS:
|
||
return name
|
||
m = re.search(r'-([^-()【】\[\]{}《》\s]{2,30})$', title)
|
||
if m:
|
||
name = m.group(1).strip()
|
||
if name not in EXCLUDE_AUTHORS and not re.match(r'^\d+', name):
|
||
return name
|
||
m = re.search(r'&([^&()【】\[\]\s]{2,30})$', title)
|
||
if m:
|
||
name = m.group(1).strip()
|
||
if name not in EXCLUDE_AUTHORS:
|
||
return name
|
||
return None
|
||
```
|
||
|
||
### 坏书检测条件
|
||
|
||
```python
|
||
is_bad = (
|
||
len(title) > 100 or
|
||
'<img' in title or '<BR' in title or '<font' in title or
|
||
'\ufffd' in title or
|
||
len(title.encode('utf-8')) > 150
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## 二、脏数据清理
|
||
|
||
### 2.1 检测条件
|
||
|
||
扫描书库中所有 TXT 文件，读取前 20KB 检测中文比例：
|
||
|
||
- **内容损坏/二进制垃圾**：中文比例 < 10%（标记为待删除）
|
||
- **小文件垃圾**：`uncompressed_size < 10240`（10KB 以下）
|
||
- **垃圾格式标题**：Python 列表格式（含 `'xxx', 'R-18'` 单引号列表）、转义字符（`\n`、`\u3000`、`id=`）、HTML 实体（`raquo;`、`laquo;`）
|
||
|
||
```python
|
||
def is_garbage_format_title(title):
|
||
"""检测标题是否为垃圾格式"""
|
||
import re
|
||
if re.search(r"'.*',\s*'R-18'", title):
|
||
return True
|
||
if re.search(r'[\\]u[0-9a-fA-F]{4}', title):
|
||
return True
|
||
if re.search(r'id=|pid=', title):
|
||
return True
|
||
if 'raquo;' in title or 'laquo;' in title:
|
||
return True
|
||
if '【内容简介】' in title or '内容简介】' in title:
|
||
return True
|
||
return False
|
||
|
||
def is_dirty_content(full_path):
|
||
"""检测文件内容是否为垃圾"""
|
||
if not os.path.exists(full_path):
|
||
return True # 文件丢失也算脏
|
||
with open(full_path, 'rb') as f:
|
||
raw = f.read(20480)
|
||
text = raw.decode('utf-8', errors='replace')
|
||
chinese = sum(1 for c in text if '\u4e00' <= c <= '\u9fff')
|
||
ratio = chinese / len(text) if len(text) > 0 else 0
|
||
return ratio < 0.10
|
||
```
|
||
|
||
### 2.2 批量删除（通过 SQLite 直写）
|
||
|
||
停掉 calibre 进程后，通过 SQLite 删除记录并清理文件系统：
|
||
|
||
```sql
|
||
DELETE FROM data WHERE book = ?
|
||
DELETE FROM books_authors_link WHERE book = ?
|
||
DELETE FROM books_languages_link WHERE book = ?
|
||
DELETE FROM books_publishers_link WHERE book = ?
|
||
DELETE FROM books_series_link WHERE book = ?
|
||
DELETE FROM books_tags_link WHERE book = ?
|
||
DELETE FROM comments WHERE book = ?
|
||
DELETE FROM identifiers WHERE book = ?
|
||
DELETE FROM books WHERE id = ?
|
||
```
|
||
|
||
同时删除对应目录：`shutil.rmtree(os.path.join('/home/yangxuan/calibre/books', bpath))`
|
||
|
||
### 2.3 完整删除脚本
|
||
|
||
```python
|
||
import sqlite3, os, shutil
|
||
|
||
db_path = '/home/yangxuan/calibre/books/metadata.db'
|
||
conn = sqlite3.connect(db_path)
|
||
c = conn.cursor()
|
||
|
||
c.execute('''
|
||
SELECT b.id, b.title, b.path, d.name, d.uncompressed_size
|
||
FROM books b
|
||
JOIN data d ON d.book = b.id
|
||
WHERE d.format = 'TXT'
|
||
''')
|
||
rows = c.fetchall()
|
||
|
||
to_delete = []
|
||
|
||
for bid, title, bpath, fname, size in rows:
|
||
full_path = os.path.join('/home/yangxuan/calibre/books', bpath, fname + '.txt')
|
||
|
||
# 条件0：垃圾格式标题（Python列表、转义字符等）
|
||
if is_garbage_format_title(title):
|
||
to_delete.append((bid, bpath))
|
||
continue
|
||
|
||
# 条件1：小文件垃圾
|
||
if size < 10240:
|
||
to_delete.append((bid, bpath))
|
||
continue
|
||
|
||
if not os.path.exists(full_path):
|
||
continue
|
||
|
||
# 条件2：内容损坏（中文 < 10%）
|
||
if is_dirty_content(full_path):
|
||
to_delete.append((bid, bpath))
|
||
|
||
print(f"待删除: {len(to_delete)}", flush=True)
|
||
for bid, bpath in to_delete:
|
||
for table in ['data', 'books_authors_link', 'books_languages_link',
|
||
'books_publishers_link', 'books_series_link', 'books_tags_link',
|
||
'comments', 'identifiers']:
|
||
c.execute(f'DELETE FROM {table} WHERE book=?', (bid,))
|
||
c.execute('DELETE FROM books WHERE id=?', (bid,))
|
||
full_dir = os.path.join('/home/yangxuan/calibre/books', bpath)
|
||
if os.path.exists(full_dir):
|
||
shutil.rmtree(full_dir)
|
||
|
||
conn.commit()
|
||
conn.close()
|
||
print(f"完成，删除: {len(to_delete)}", flush=True)
|
||
```
|
||
|
||
---
|
||
|
||
## 三、书名批量清洗（作者标记去除 + 前 20 中文截取）
|
||
|
||
对于书名中含 `作者：xxx`、`翻译：xxx` 标记但作者字段已有值的书，使用此脚本批量清洗：
|
||
|
||
```python
|
||
import sqlite3, re
|
||
|
||
db_path = '/home/yangxuan/calibre/books/metadata.db'
|
||
conn = sqlite3.connect(db_path)
|
||
c = conn.cursor()
|
||
|
||
c.execute('''
|
||
SELECT b.id, b.title, a.name as author_name, d.name as file_name
|
||
FROM books b
|
||
JOIN data d ON d.book = b.id AND d.format = 'TXT'
|
||
JOIN books_authors_link l ON b.id = l.book
|
||
JOIN authors a ON l.author = a.id
|
||
WHERE (b.title LIKE '%作者：%' OR b.title LIKE '%作者:%' OR b.title LIKE '%翻译：%' OR b.title LIKE '%翻译:%')
|
||
AND a.name NOT IN ('未知', 'Unknown')
|
||
ORDER BY b.id
|
||
''')
|
||
rows = c.fetchall()
|
||
|
||
for bid, title, author_name, file_name in rows:
|
||
# 1. 从 data.name 提取作者并更新 author 字段（如不同）
|
||
name_clean = re.sub(r'\s*-\s*(Unknown|未知)\s*$', '', file_name)
|
||
m = re.search(r'[ \s]*?(?:作者|翻译|原?作者)[：:]\s*([^\s_）\)《\[【\]】\-,]+)', name_clean)
|
||
if m:
|
||
extracted = m.group(1).strip()
|
||
if extracted and extracted != author_name and len(extracted) < 30:
|
||
cur = conn.cursor()
|
||
cur.execute("SELECT id FROM authors WHERE name=?", (extracted,))
|
||
ar = cur.fetchone()
|
||
if ar:
|
||
aid = ar[0]
|
||
else:
|
||
cur.execute("INSERT INTO authors (name, sort) VALUES (?, ?)", (extracted, extracted.replace(',', '').strip()))
|
||
aid = cur.lastrowid
|
||
cur.execute("UPDATE books_authors_link SET author=? WHERE book=?", (aid, bid))
|
||
cur.close()
|
||
|
||
# 2. 裁剪书名：去标记 + 前 20 中文
|
||
clean_title = re.sub(r'[ \s]*?(?:作者|翻译|原?作者)[：:].*$', '', file_name, flags=re.DOTALL)
|
||
clean_title = re.sub(r'\s*-\s*(Unknown|未知)\s*$', '', clean_title).strip()
|
||
|
||
if clean_title:
|
||
count = 0
|
||
pos = 0
|
||
for i, ch in enumerate(clean_title):
|
||
if '\u4e00' <= ch <= '\u9fff':
|
||
count += 1
|
||
if count == 20:
|
||
pos = i + 1
|
||
break
|
||
if count > 20:
|
||
clean_title = clean_title[:pos]
|
||
|
||
if clean_title != title:
|
||
c.execute("UPDATE books SET title=? WHERE id=?", (clean_title, bid))
|
||
|
||
conn.commit()
|
||
conn.close()
|
||
```
|
||
|
||
---
|
||
|
||
## 四、常用查询
|
||
|
||
### 4.1 查看 XX 作者的书
|
||
|
||
```bash
|
||
sqlite3 /home/yangxuan/calibre/books/metadata.db "
|
||
SELECT b.id, b.title
|
||
FROM books b JOIN books_authors_link l ON b.id = l.book
|
||
JOIN authors a ON l.author = a.id
|
||
WHERE a.name = 'Unknown'
|
||
LIMIT 20;
|
||
"
|
||
```
|
||
|
||
### 4.2 统计作者分布
|
||
|
||
```bash
|
||
sqlite3 /home/yangxuan/calibre/books/metadata.db "
|
||
SELECT a.name, COUNT(*) as cnt
|
||
FROM books b JOIN books_authors_link l ON b.id = l.book
|
||
JOIN authors a ON l.author = a.id
|
||
GROUP BY a.name ORDER BY cnt DESC LIMIT 20;
|
||
"
|
||
```
|
||
|
||
### 4.3 书名异常检测
|
||
|
||
```bash
|
||
sqlite3 /home/yangxuan/calibre/books/metadata.db "
|
||
SELECT id, title FROM books WHERE
|
||
title LIKE '%作者%' OR title LIKE '%翻译%' OR
|
||
title LIKE '%.com%' OR title LIKE '%http%' OR
|
||
title LIKE '%内容简介%' OR
|
||
LENGTH(title) > 80;
|
||
"
|
||
```
|
||
|
||
### 4.4 书库统计
|
||
|
||
```bash
|
||
python3 -c "
|
||
import sqlite3
|
||
conn = sqlite3.connect('/home/yangxuan/calibre/books/metadata.db')
|
||
c = conn.cursor()
|
||
c.execute('SELECT COUNT(*) FROM books')
|
||
print(f'总书籍: {c.fetchone()[0]}')
|
||
c.execute('SELECT COUNT(*) FROM data WHERE format=\"TXT\"')
|
||
print(f'TXT 文件: {c.fetchone()[0]}')
|
||
conn.close()
|
||
"
|
||
```
|
||
|
||
---
|
||
|
||
## 五、操作流程（推荐顺序）
|
||
|
||
```bash
|
||
cd /home/yangxuan/calibre
|
||
|
||
# 1. 停服务 + 备份
|
||
docker compose down
|
||
cp books/metadata.db books/metadata.db.backup.$(date +%Y%m%d)
|
||
killall calibre 2>/dev/null
|
||
|
||
# 2. 脏数据清理（内容损坏 + 小文件 + 垃圾格式标题）
|
||
python3 /path/to/clean_dirty_data.py
|
||
|
||
# 3. 书名清洗 & 作者提取（含 data.name 级别提取）
|
||
calibre-debug /path/to/fix_titles.py
|
||
|
||
# 4. 检查是否有遗漏的异常书籍
|
||
sqlite3 books/metadata.db \"SELECT id, title FROM books WHERE title LIKE '%作者%' OR title LIKE '%翻译%' OR title LIKE '%.com%';\" | head -10
|
||
|
||
# 5. 重启
|
||
docker compose up -d
|
||
```
|
||
|
||
---
|
||
|
||
## 六、已知问题
|
||
|
||
### `set_metadata` 报 `Errno 36 文件名过长`
|
||
|
||
当 `author_sort` 字段超 80 字节时，Calibre 路径会触及 ext4 255 字节限制。先清理极长作者名：
|
||
|
||
```bash
|
||
sqlite3 /home/yangxuan/calibre/books/metadata.db "
|
||
UPDATE authors SET name='Unknown' WHERE LENGTH(name) > 30;
|
||
"
|
||
```
|
||
|
||
### WSL `/mnt/f` 删除 NTFS 文件权限不够
|
||
|
||
WSL 的 drvfs 默认不带 metadata 选项，`sudo rm`、`os.remove`、`cmd.exe /c del` 均无法删除 NTFS 文件。不要在 WSL 下尝试跨分区删除操作，直接让 Windows 侧（文件资源管理器/PowerShell）操作。
|
||
|
||
### 书名长度截取规则
|
||
|
||
`title` 字段清洗后最多保留 20 个中文字符。计数规则：只计 Unicode 范围的 `\u4e00-\u9fff` 中文字符，保留第 20 个中文之前的所有字符（含非中文）。如书名不足 20 个中文则全保留。
|
||
|
||
### `title` vs `data.name` 区别
|
||
|
||
- `books.title`：Calibre 界面显示的书名（清洗后的）
|
||
- `data.name`：导入时的原始文件名（不含 `.txt`）
|
||
- 作者提取优先使用 `data.name`，其文件名保留原始信息、更可靠 |