> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Task Arithmetic: Task Vectors and Linear Combinations

> Task Arithmetic mathematical specification for EMEP. Defines task vectors as theta_ft minus theta_pre, linear combinations, skill addition, forgetting/negation, and combination rules. Reference: Ilharco et al. 2022.

Task Arithmetic treats fine-tuning as a vector in parameter space. This page defines task vectors, their linear combinations, and the rules for adding skills and removing biases. Reference: Ilharco et al. 2022.

## Task Vector Definition

Given a pre-trained base model theta\_pre and a fine-tuned model theta\_ft:

```text theme={null}
tau = theta_ft - theta_pre
```

tau is a task vector: a set of parameter deltas that encode the skill or behavior learned during fine-tuning.

## Linear Combinations

Multiple task vectors can be combined with scalar coefficients:

```text theme={null}
theta_merged = theta_pre + sum_i (alpha_i * tau_i)
```

| Symbol | Meaning |
| - | - |
| theta\_pre | Pre-trained base model parameters |
| tau\_i | Task vector i |
| alpha\_i | Scalar coefficient for task vector i |
| theta\_merged | Resulting merged model |

## Adding Skills

To add a skill, set the corresponding alpha to a positive value:

```text theme={null}
theta_with_skill = theta_pre + 1.0 * tau_skill
```

Multiple skills can be added simultaneously:

```text theme={null}
theta_multi_skill = theta_pre + 0.5 * tau_math + 0.5 * tau_code
```

## Forgetting and Negation

To remove a bias or unwanted behavior, negate the task vector:

```text theme={null}
theta_without_bias = theta_pre + (-1.0) * tau_bias
```

This is the basis of forgetting controls in EMEP. See [Catastrophic Forgetting Controls](https://doc.blueapi.ir/finetuning/catastrophic-forgetting-controls.md) for extended discussion. Reference: McCloskey & Cohen 1989; Kirkpatrick et al. 2017.

## Combination Rules

| Operation | Formula | Effect |
| - | - | - |
| Add skill | theta\_pre + alpha \* tau\_skill | Enhances skill |
| Remove bias | theta\_pre - alpha \* tau\_bias | Suppresses bias |
| Merge skills | theta\_pre + sum\_i alpha\_i \* tau\_i | Combines multiple skills |
| Average models | theta\_pre + average(tau\_i) | Model Soups (Wortsman et al. 2022) |

## Preconditions

All task vectors must be derived from the same theta\_pre. Merging task vectors from different base models is undefined and produces INVALID.

| Condition | Outcome |
| - | - |
| Different base models | INVALID |
| Missing theta\_pre | INVALID |
| alpha too large | Catastrophic forgetting; QUARANTINE |

## Failure Modes

| Condition | Outcome |
| - | - |
| Base model missing | Cannot compute task vector; FAIL |
| alpha\_i sum much greater than 1 | Parameter drift; QUARANTINE |
| alpha\_i sum much less than 0 | Inverted skills; QUARANTINE |

## Cross-Links

* [Merge Math](https://doc.blueapi.ir/math/merge-math.md) for shared notation and weight shapes.
* [Merge Strategies](https://doc.blueapi.ir/merge/merge-strategies.md) for when Task Arithmetic is selected.
* [Catastrophic Forgetting Controls](https://doc.blueapi.ir/finetuning/catastrophic-forgetting-controls.md) for forgetting mitigation.
* [Tensor Operations](https://doc.blueapi.ir/merge/tensor-operations.md) for the TensorEngine implementation.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.