# Ultralytics YOLO on Axelera Voyager SDK

This document explains how to build and run complete Ultralytics YOLO inference pipelines on Axelera Metis devices using the Axelera runtime.

The typical workflow has two phases:

1. **Model preparation.** You use Ultralytics, PyTorch, and the Voyager SDK
   compiler to train, export, and compile your model to `.axm` format.
2. **Application development.** You use `axelera-rt` to build and run an end-to-end video pipeline. `axelera-rt` is a lightweight, self-contained library with minimal package dependencies. It lets you describe pipelines that read frames from a camera, preprocess them, run model inference, and postprocess the results. The runtime can also fuse pipeline stages together automatically to minimize computation and data transfers.

A pipeline looks like this:

```python
pipeline = op.seq(
    op.colorconvert("RGB", src="BGR"),  # OpenCV reads BGR; models expect RGB
    op.letterbox(640, 640),
    op.totensor(),
    op.load("yolo26n-pose.axm"),
    ConfidenceFilter(threshold=0.25),  # custom operator — see below
    op.to_image_space(keypoint_cols=range(6, 57, 3)),
).optimized()  # runtime fuses ops for maximum throughput

poses = pipeline(frame)  # frame in, results out
```

## Examples

Two examples are provided below. It should be straightforward to apply them to other models and tasks.

| Script                   | Task                                                | Model                   | Description                                                                                                             |
| ------------------------ | --------------------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `yolo26-pose-tracker.py` | Pose estimation with optional multi-object tracking | YOLO26n-pose (NMS-free) | Linear `op.seq()` pipeline with a custom operator and optional `op.tracker()`. Use `--tracker none` for pose-only mode. |
| `yolo11-seg.py`          | Instance segmentation                               | YOLO11n-seg             | Branching with `op.par()` for multi-head models (detections + masks).                                                   |

## Quick Start

### Install

If you followed the [Axelera integration guide](../../docs/en/integrations/axelera.md), `axelera-rt` is already installed — running `yolo predict` or `yolo val` with an Axelera model auto-installs the runtime dependencies. If you need to install manually:

```bash
pip install axelera-rt==1.6.0 --no-cache-dir --extra-index-url https://software.axelera.ai/artifactory/api/pypi/axelera-pypi/simple
```

You will also need `opencv-python` and `numpy` (likely already present).

### Compile

Export and compile your trained model directly to Axelera format using the Ultralytics CLI:

```bash
yolo export model=your-model.pt format=axelera
```

To reproduce these examples, export the pretrained Ultralytics models:

```bash
yolo export model=yolo26n-pose.pt format=axelera
yolo export model=yolo11n-seg.pt format=axelera
```

The compiled models are saved to `yolo26n-pose_axelera_model/` and
`yolo11n-seg_axelera_model/` respectively. Pass the `.axm` file inside to `--model`.

> [!NOTE]
> Each model directory also contains `metadata.yaml` with the model's class names and configuration. You can load this info directly to avoid manual label entry in your own application.

### Run

#### Pose Estimation (Ultralytics YOLO26)

```bash
python yolo26-pose-tracker.py --model yolo26n-pose.axm --source 0 --tracker none         # pose only # webcam
python yolo26-pose-tracker.py --model yolo26n-pose.axm --source image.jpg --tracker none # pose only # image
python yolo26-pose-tracker.py --model yolo26n-pose.axm --source video.mp4 --tracker none # pose only # video
```

#### Pose Tracking (Ultralytics YOLO26 + TrackTrack)

```bash
python yolo26-pose-tracker.py --model yolo26n-pose.axm --source video.mp4              # video with tracking
python yolo26-pose-tracker.py --model yolo26n-pose.axm --source 0 --tracker tracktrack # webcam with tracking
```

#### Instance Segmentation (Ultralytics YOLO11)

```bash
python yolo11-seg.py --model yolo11n-seg.axm --source 0
python yolo11-seg.py --model yolo11n-seg.axm --source video.mp4 --conf 0.3 --iou 0.5
```

### Arguments

| Argument    | Default      | Description                                                              |
| ----------- | ------------ | ------------------------------------------------------------------------ |
| `--model`   | _required_   | Path to compiled `.axm` model                                            |
| `--source`  | `0`          | Image path, video path, or webcam index                                  |
| `--conf`    | `0.25`       | Confidence threshold                                                     |
| `--iou`     | `0.45`       | NMS IoU threshold _(segmentation only)_                                  |
| `--tracker` | `tracktrack` | Tracking algorithm: `bytetrack`, `oc-sort`, `sort`, `tracktrack`, `none` |

## What You'll See in the Code

### Custom Operators

The pipeline is fully composable — drop in your own operators anywhere in the sequence.
Just subclass `op.Operator` and implement `__call__`:

```python
class ConfidenceFilter(op.Operator):
    threshold: float = 0.25

    def __call__(self, x):
        # Each operator receives the output of the previous one in op.seq().
        # Here x is the output of the pose detection model — an array of
        # shape (batch, num_detections, num_values) where each row holds
        # [bbox, confidence, class_id, keypoints…]. Column 4 is the
        # confidence score. Squeeze the batch dim and filter by score.
        if x.ndim == 3:
            x = x[0]
        return x[x[:, 4] >= self.threshold]
```

This means they are fused, scheduled, and optimized alongside built-in ops like `op.letterbox` and the model itself — adding a custom filter carries no special overhead.

### Multi-Object Tracking

Adding tracking to any detection or pose pipeline is a single line via `op.tracker()`:

```python
pipeline = op.seq(
    # ... your detection or pose pipeline ...
    op.tracker(algo="tracktrack"),  # one line adds tracking
)
```

| Algorithm  | Key Strength                                         | Reference                |
| ---------- | ---------------------------------------------------- | ------------------------ |
| TrackTrack | Iterative matching with track-aware NMS (SOTA)       | CVPR 2025                |
| ByteTrack  | Handles low-confidence detections via dual threshold | Zhang et al., ECCV 2022  |
| OC-SORT    | Observation-centric re-update + virtual trajectory   | Cao et al., CVPR 2023    |
| SORT       | Simple, fast IoU-only baseline                       | Bewley et al., ICIP 2016 |

The tracker operates on bounding boxes, but the original detection (with all its
metadata) is preserved — use `tracked.tracked` on each result to access the
original keypoints, masks, or other model outputs. You'll see this pattern in
`yolo26-pose-tracker.py`.

## Learn More

The [Voyager SDK](https://github.com/axelera-ai-hub/voyager-sdk) has everything
beyond these introductory examples: advanced pipeline patterns, ReID and
appearance-based tracking, the high-performance display system, production-ready
application templates, and the full API reference.

Questions or feedback? Join the [Axelera community](https://community.axelera.ai/).
