ModelCompatibilityAnalyzer: Full Specification

The ModelCompatibilityAnalyzer is the gatekeeper of the EMEP merge pipeline. It inspects two or more candidate models and decides whether they can be merged, can be merged with conditions, or must be rejected. This page defines every check, the decision tree, and the three output states: COMPATIBLE, CONDITIONALLY_COMPATIBLE, and INCOMPATIBLE.

​

Purpose

The analyzer prevents invalid merges before they reach the TensorEngine. It validates architectural alignment, tensor consistency, tokenizer equivalence, and licensing constraints. A failed check at any stage can downgrade the result from COMPATIBLE to CONDITIONALLY_COMPATIBLE or INCOMPATIBLE.

​

Problem Definition

Given a list of N candidate models (N ≥ 2), determine whether the models can participate in a merge operation. The output is a compatibility state per model pair and an aggregated state for the full set.

​

Inputs

​

Preconditions

• All models are in REGISTERED or higher lifecycle state.
• Config files are present and parseable as JSON.
• State dicts are loadable into memory or via memory-mapped access.
• Tokenizers are present and loadable.

​

Parameters

​

Decision Tree

​

Full Check List

​

1. Architecture Family

Compare model_type fields. Both must be identical (e.g., both llama, both mistral).

​

2. Model Type

Verify architectures list contains the same class names.

​

3. Parameter Count

Compute total parameters from state dicts. Reject if ratio exceeds max_param_ratio.

​

4. Layer Count

Compare num_hidden_layers. Must match exactly for COMPATIBLE.

​

5. Hidden Size

Compare hidden_size. Must match exactly.

​

6. Intermediate Size

Compare intermediate_size (FFN dimension). Must match exactly.

​

7. Attention Heads

Compare num_attention_heads. Must match exactly.

​

8. KV Heads

Compare num_key_value_heads. Must match exactly for GQA/MQA models.

​

9. Embedding Size

Compare embedding_size or hidden_size for embedding layers.

​

10. Vocabulary Size

Compare vocab_size. Must match exactly if strict_vocab=True.

​

11. Tokenizer

Compare tokenizer vocabularies, special tokens, and chat templates.

​

12. Special Tokens

Compare bos_token_id, eos_token_id, pad_token_id, unk_token_id.

​

13. RoPE Configuration

Compare rope_theta, rope_scaling settings.

​

14. Context Length

Compare max_position_embeddings.

​

15. Parameter Names

Compare state dict keys. Must have identical sets for COMPATIBLE.

​

16. Tensor Shapes

For each shared parameter name, compare tensor shapes.

​

17. Dtype

Compare tensor dtypes. Must match if strict_dtype=True.

​

18. Weight Format

Compare serialization format (Safetensors, PyTorch pickle, GGUF).

​

19. Revision / Source

Record model revision, source repository, and download URL.

​

20. License

Verify licenses permit derivative works and redistribution.

​

Output States

​

Failure Conditions

• Missing config or state dict
• Unparseable JSON
• Unloadable tokenizer
• License prohibits derivative works
• Architecture family mismatch
• Hidden size mismatch
• Layer count mismatch
• Tensor shape mismatch on any shared key

​

Validation

• Unit test: pair of identical models must return COMPATIBLE.
• Unit test: pair with different hidden_size must return INCOMPATIBLE.
• Unit test: pair with same architecture but different tokenizer must return CONDITIONALLY_COMPATIBLE.
• Integration test: feed analyzer output into MergeEngine and verify it rejects INCOMPATIBLE pairs.

​

Complexity

• Time: O(P) where P is total parameter count (shape inspection only, no data reads).
• Space: O(K) where K is number of state dict keys.

​

Memory Requirements

Metadata and configs only. No weight data is loaded into GPU memory during compatibility analysis.

​

Numerical Stability

Not applicable. This is a metadata-only check.

​

Precision Considerations

Dtype comparison is symbolic. No numerical values are inspected.

​

Reproducibility

Given identical inputs, the analyzer must produce identical states. No randomness is permitted.

​

Known Limitations

• Does not inspect weight values (only shapes and metadata).
• Does not verify functional equivalence (two models with identical configs may behave differently due to training).
• Does not check for data contamination or benchmark overlap.

​

Research References

• Model Soups: Wortsman et al. 2022
• Task Arithmetic: Ilharco et al. 2022
• TIES-Merging: Yadav et al. 2023
• DARE: Yu et al. 2023