> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenizer Compatibility Algorithm and Checks

> Tokenizer compatibility specification for EMEP. Covers vocab overlap, special token matching, BPE and SentencePiece family checks, and the decision flow to COMPATIBLE, CONDITIONALLY_COMPATIBLE, or INCOMPATIBLE.

Tokenizer compatibility determines whether two models can share a vocabulary or require remapping. This page specifies the algorithm, thresholds, and exact state transitions.

## Check Dimensions

The ModelCompatibilityAnalyzer evaluates tokenizers across these dimensions:

| Dimension | Check |
| - | - |
| Vocab overlap | Intersection size / union size, as percentage |
| Special token match | BOS, EOS, PAD, UNK IDs and strings |
| Tokenizer family | BPE, SentencePiece, WordPiece, Tiktoken |
| Added tokens | Extra tokens beyond base vocab |
| Unicode normalization | NFC, NFD, NFKC, NFKD, or none |
| Pre-tokenizer | Whitespace, ByteLevel, or custom |
| Decoder | Mapping from tokens to text |

## Compatibility Decision Flow

```mermaid theme={null}
flowchart TD
    START([START]) --> INPUT[Input: Tokenizer A + Tokenizer B]
    INPUT --> FAMILY{Same tokenizer family?}
    FAMILY -->|no| NORM{Same Unicode normalization?}
    NORM -->|no| INCOMPATIBLE[INCOMPATIBLE: family and norm mismatch]
    NORM -->|yes| OVERLAP{Vocab overlap >= 95%?}
    OVERLAP -->|no| INCOMPATIBLE
    OVERLAP -->|yes| SPECIAL{Special tokens match?}
    SPECIAL -->|no| CONDITIONALLY_COMPATIBLE_1[CONDITIONALLY_COMPATIBLE: special token mismatch]
    SPECIAL -->|yes| ADDED{Added tokens identical?}
    ADDED -->|no| CONDITIONALLY_COMPATIBLE_2[CONDITIONALLY_COMPATIBLE: added token mismatch]
    ADDED -->|yes| COMPATIBLE[COMPATIBLE]
    FAMILY -->|yes| OVERLAP_2{Vocab overlap >= 98%?}
    OVERLAP_2 -->|no| CONDITIONALLY_COMPATIBLE_3[CONDITIONALLY_COMPATIBLE: low overlap within family]
    OVERLAP_2 -->|yes| SPECIAL_2{Special tokens match?}
    SPECIAL_2 -->|no| CONDITIONALLY_COMPATIBLE_4[CONDITIONALLY_COMPATIBLE: special token mismatch]
    SPECIAL_2 -->|yes| ADDED_2{Added tokens identical?}
    ADDED_2 -->|no| CONDITIONALLY_COMPATIBLE_5[CONDITIONALLY_COMPATIBLE: added token mismatch]
    ADDED_2 -->|yes| COMPATIBLE_2[COMPATIBLE]
    COMPATIBLE --> END([END])
    COMPATIBLE_2 --> END
    CONDITIONALLY_COMPATIBLE_1 --> END
    CONDITIONALLY_COMPATIBLE_2 --> END
    CONDITIONALLY_COMPATIBLE_3 --> END
    CONDITIONALLY_COMPATIBLE_4 --> END
    CONDITIONALLY_COMPATIBLE_5 --> END
    INCOMPATIBLE --> END
```

## Threshold Rationale

| Threshold | Rationale |
| - | - |
| 98% overlap (same family) | Engineering assumption: minor vocab drift is acceptable for merge |
| 95% overlap (different family) | Engineering assumption: cross-family requires substantial overlap |
| Special token exact match | Required for generation correctness; mismatch triggers remapping need |

## CONDITIONALLY\_COMPATIBLE Cases

| Case | Required Action |
| - | - |
| Special token mismatch | Remap special token IDs or retrain embeddings |
| Added token mismatch | Intersect added tokens or retrain |
| Low overlap within family | Vocab remapping or fallback tokenizer |

## INCOMPATIBLE Cases

| Case | Reason |
| - | - |
| Family and norm mismatch | Tokenization diverges at the byte level |
| Overlap below 95% | Too many tokens have no correspondence |

## Cross-Links

* [Model Compatibility](https://doc.blueapi.ir/compatibility/model-compatibility.md) for the full decision tree.
* [Architecture Compatibility](https://doc.blueapi.ir/compatibility/architecture-compatibility.md) for the architecture-only branch.
* [Tensor Shape Validation](https://doc.blueapi.ir/compatibility/tensor-shape-validation.md) for the tensor branch.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.