cdcailf / cjkfold

# cjkfold
Text pulled out of a PDF often contains characters that **look** like ordinary
Han characters but are not them. Comparison against them quietly fails.
 [code] 
`cjkfold` folds them back onto the characters you meant:
 [code] 
Zero runtime dependencies. Pure functions. Python 3.9+.
---
**背景阅读 / Background:** PDF 抽出来的中文，为什么搜不到
—— 完整的排查经过（中文），包括我自己踩的四个坑。
社区版：掘金
**Built on this:** corpusfold — this
character layer wired into a full PDF → knowledge-base pipeline (抽取 / 分栏 /
表格 / 分块 / 混合检索).
## The problem
Nothing raises. The text displays correctly, because the glyphs really are
identical. What breaks is everything downstream that compares characters:
- `"紫色" in text` returns `False` for a document that shows 紫色
- a keyword search misses
- a vector retriever ranks an unrelated *short* document above the one that
 contains the query verbatim, because a short document's embedding is tighter
 and scores a higher cosine — measured on a real corpus, the relevant document
 scored **0.29** against a decoy's **0.36**
- deduplication, caching and ground-truth checks all start disagreeing
## Why NFKC is not enough
The reflex is `unicodedata.normalize("NFKC", text)`. It handles the **Kangxi
Radicals** block completely — all 214 characters carry compatibility
decompositions — and then stops:
| Block | Characters | Repaired by NFKC | Repaired by cjkfold |
|---|---|---|---|
| Kangxi Radicals `U+2F00–U+2FDF` | 214 | **214** | 214 |
| CJK Radicals Supplement `U+2E80–U+2EFF` | 115 | **2** | **114** |
Unicode withholds compatibility mappings in the supplement block *on purpose*:
those glyphs are distinct encoded radicals, not compatibility variants of an
existing character. So NFKC leaves 113 of the 115 standing.
`cjkfold` closes that gap with the `EquivalentUnifiedIdeograph` property from
the Unicode Character Database (Unicode 15.1+), which maps 114 of them onto a
unified ideograph. The one remainder, `⺀ U+2E80 CJK RADICAL REPEAT`, has no
ideograph equivalent and is left alone.
## The characters that never show up
There is a second family, and it is harder to catch. A zero-width space leaves no
glyph at all, so two strings that differ only by it look identical:
 数据
 数\u200b据
 [code] 
`fold()` does not repair these, deliberately: half the family is load-bearing. ZWJ
`U+200D` is what holds an emoji sequence together — 👨‍👩‍👧 is five code points rendered as
one glyph, and deleting the joiners does not tidy the text up, it turns one family
into three people.
`check` looks for the first family only, so a file full of zero-width spaces comes
back `clean`. `scan` looks at every family and says what to do with each finding:
 [code] 
The reason is printed once per character rather than once per occurrence: an
emoji-heavy document can hold hundreds of joiners, and repeating the same sentence
hundreds of times buries the lines that differ.
The `fixable` column is the point of the design:
| | |
|---|---|
| `fold` | cjkfold can replace it: a look-alike glyph |
| `nfkc` |