> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Tensor Operations: Low-Level Merge Primitives

> Specification of every low-level tensor operation used by EMEP merge strategies. Covers input/output shapes, dtype rules, broadcasting, and numerical stability.

TensorEngine exposes a fixed set of low-level operations that all merge strategies use. This page defines each operation, its shape contracts, dtype rules, and numerical stability notes.

## Generic Tensor Op Flow

Every operation follows this pipeline.

```mermaid theme={null}
flowchart TD
    START([START]) --> INPUT[Input A + Input B]
    INPUT --> SHAPE[Shape Check]
    SHAPE -->|mismatch| BROADCAST{Broadcastable?}
    BROADCAST -->|no| FAIL[FAIL: shape incompatible]
    BROADCAST -->|yes| BROADCAST_OP[Apply Broadcasting]
    SHAPE -->|match| DTYPE[Dtype Check]
    BROADCAST_OP --> DTYPE
    DTYPE -->|incompatible| CAST[Cast to higher precision]
    DTYPE -->|compatible| OP[Execute Operation]
    CAST --> OP
    OP --> NUMERICAL[Numerical Validation]
    NUMERICAL -->|NaN/Inf| FAIL_NUM[FAIL: numerical instability]
    NUMERICAL -->|pass| OUTPUT[Output Tensor]
    OUTPUT --> END([END])
    FAIL --> END
    FAIL_NUM --> END
```

## Operation Catalog

### Element-wise Add / Sub

| Property | Value |
| - | - |
| Inputs | A, B: tensors of broadcastable shapes |
| Output | Same broadcasted shape |
| Dtype rule | Promote to higher precision: fp16 + fp32 → fp32 |
| Numerical stability | Watch for catastrophic cancellation in fp16. Use fp32 accumulator. |

### Scalar Mul

| Property | Value |
| - | - |
| Inputs | A: tensor; s: scalar |
| Output | Same shape as A |
| Dtype rule | Scalar cast to A.dtype before multiply |
| Numerical stability | Overflow risk in fp16 for s > 65504. Clamp or use fp32. |

### Weighted Average

| Property | Value |
| - | - |
| Inputs | A, B: tensors; w: scalar weight in \[0, 1] |
| Output | Same broadcasted shape |
| Dtype rule | Compute in fp32, cast output to target dtype |
| Numerical stability | w \* A + (1 - w) \* B. Accumulate in fp32. |

### Normalization

| Property | Value |
| - | - |
| Inputs | A: tensor; norm type: L1, L2, or inf |
| Output | Same shape as A |
| Dtype rule | Compute norm in fp32 regardless of input dtype |
| Numerical stability | Guard against division by zero with epsilon = 1e-8. |

### Linear Interpolation (Lerp)

| Property | Value |
| - | - |
| Inputs | A, B: tensors; t: scalar in \[0, 1] |
| Output | Same broadcasted shape |
| Dtype rule | Compute in fp32, cast to target |
| Numerical stability | Straightforward. No special guards needed. |

### Spherical Interpolation (SLERP)

| Property | Value | | |
| - | - | - | - |
| Inputs | A, B: 1-D parameter vectors; t: scalar in \[0, 1] | | |
| Output | 1-D vector, same length | | |
| Dtype rule | Normalize in fp64 for precision, cast to target | | |
| Numerical stability | Fallback to lerp when theta \< 1e-6 or | theta - pi | \< 1e-6. See [SLERP Math](https://doc.blueapi.ir/math/slerp-math.md). |

### Sparsification

| Property | Value |
| - | - |
| Inputs | A: tensor; p: drop probability in \[0, 1) |
| Output | Same shape as A, with (1 - p) fraction of values kept |
| Dtype rule | Mask in bool, apply to A in original dtype |
| Numerical stability | Rescale survivors by 1 / (1 - p). See [DARE Math](https://doc.blueapi.ir/math/dare-math.md). |

### Sign Masking

| Property | Value |
| - | - |
| Inputs | A: tensor; sign: +1, -1, or 0 |
| Output | Same shape as A, zeroed where sign mismatch |
| Dtype rule | Sign computed in fp32, mask applied in original dtype |
| Numerical stability | No special guards. |

### Rescaling

| Property | Value |
| - | - |
| Inputs | A: tensor; scale: scalar |
| Output | Same shape as A |
| Dtype rule | Scale cast to A.dtype |
| Numerical stability | Overflow risk in fp16. Use fp32 intermediate. |

### Tensor Concat / Slice

| Property | Value |
| - | - |
| Inputs | A, B: tensors; dim: int |
| Output | Concatenated shape along dim |
| Dtype rule | Both tensors must have identical dtype |
| Numerical stability | N/A |

### Layer Selection

| Property | Value |
| - | - |
| Inputs | model: state\_dict; layer\_indices: List\[int] |
| Output | Subset state\_dict with only selected layers |
| Dtype rule | Preserved from input |
| Numerical stability | N/A |

## Cross-Links

* [Merge Engine](https://doc.blueapi.ir/merge/merge-engine.md) for the pipeline that calls these ops.
* [Merge Strategies](https://doc.blueapi.ir/merge/merge-strategies.md) for which strategies use which ops.
* [Merge Math](https://doc.blueapi.ir/math/merge-math.md) for shared notation.
* [SLERP Math](https://doc.blueapi.ir/math/slerp-math.md), [TIES Math](https://doc.blueapi.ir/math/ties-math.md), [DARE Math](https://doc.blueapi.ir/math/dare-math.md) for strategy-specific formulas.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.