Skip to content

Pipeline

Vizor is the front door. It owns a primary model and a Refiner, and drives them over a video. Refiner is the part that decides what to send to the secondary and what to do with the answer, and you can use it on its own if you already have detections.

Vizor

Vizor(primary, secondary=None, conf=0.5, mode='full', names=None, **kw)

Run a fast tracker on every frame and refine it with a slower model.

The primary gives boxes and track ids at frame rate. The secondary is only asked about tracks the primary is unsure of, and its answers are cached against the track id, so the cost is paid once per object rather than once per frame.

Parameters:

Name Type Description Default
primary Model

model with a track(img) method returning Tracks.

required
secondary Model | None

model with find (full mode), name (crop mode) or grid (collage mode). Optional.

None
conf float

tracks at or below this confidence go to the secondary.

0.5
mode str

"full" runs the secondary on the whole frame, "crop" on each box, "collage" on several crops of one track tiled together.

'full'
names dict[int, str] | list[str] | None

class id to name mapping. Defaults to whatever the primary reports.

None

Remaining keyword arguments go to Refiner. Pass workers there to stop the frame loop waiting on the secondary.

Source code in vizor/core.py
def __init__(
    self,
    primary: "Model",
    secondary: "Model | None" = None,
    conf: float = 0.5,
    mode: str = "full",
    names: "dict[int, str] | list[str] | None" = None,
    **kw: Any,
):
    self.primary = primary
    self.refiner = Refiner(secondary, conf=conf, mode=mode, names=names, **kw)

secondary property

secondary

The refining model, or None if there is not one.

names property

names

Class id to name mapping, falling back to whatever the primary reports.

reset

reset()

Clear the vote cache and any state the primary keeps between frames.

Source code in vizor/core.py
def reset(self):
    """Clear the vote cache and any state the primary keeps between frames."""
    self.refiner.reset()
    reset = getattr(self.primary, "reset", None)
    if callable(reset):
        reset()

wait

wait()

Block until every background request has come back. See workers.

Source code in vizor/core.py
def wait(self):
    """Block until every background request has come back. See ``workers``."""
    self.refiner.wait()

close

close()

Stop the background workers, dropping anything still in flight.

Source code in vizor/core.py
def close(self):
    """Stop the background workers, dropping anything still in flight."""
    self.refiner.close()

step

step(img, preds=None)

Process one frame and return the refined Tracks.

Source code in vizor/core.py
def step(self, img, preds=None):
    """Process one frame and return the refined [Tracks][vizor.boxes.Tracks]."""
    tracks = self.primary.track(img)
    if not isinstance(tracks, Tracks):
        tracks = Tracks(tracks)
    if tracks.names is None:
        tracks.names = getattr(self.primary, "names", None)
    return self.refiner.run(tracks, img=img, preds=preds)

run

run(src, save=None, show=False, fourcc='mp4v')

Yield refined tracks for every frame of src.

src is anything Video opens: a file path, a camera index, or a stream url. Pass save to also write an annotated video, and show to display it in a window.

Each yielded Tracks carries its frame, so out.draw() needs no argument. Drawing happens on the frame itself, so copy it first if you want the original.

Source code in vizor/core.py
def run(self, src, save=None, show=False, fourcc="mp4v"):
    """Yield refined tracks for every frame of ``src``.

    ``src`` is anything [Video][vizor.utils.video.Video] opens: a file path,
    a camera index, or a stream url. Pass ``save`` to also write an annotated
    video, and ``show`` to display it in a window.

    Each yielded ``Tracks`` carries its frame, so ``out.draw()`` needs no
    argument. Drawing happens on the frame itself, so copy it first if you
    want the original.
    """
    video = Video(src)
    writer = Writer(save, fps=video.fps, fourcc=fourcc) if save else None
    try:
        for frame in video:
            out = self.step(frame)
            yield out
            if writer or show:
                drawn = out.draw(frame)
                if writer:
                    writer.write(drawn)
                if show:
                    import cv2

                    cv2.imshow("vizor", drawn)
                    if cv2.waitKey(1) & 0xFF in (27, ord("q")):
                        break
    finally:
        video.close()
        if writer:
            writer.close()
        if show:
            import cv2

            try:
                cv2.destroyWindow("vizor")
            except cv2.error:
                pass

save

save(src, out, show=False)

Run over src and write the annotated video to out.

Source code in vizor/core.py
def save(self, src, out, show=False):
    """Run over ``src`` and write the annotated video to ``out``."""
    for _ in self.run(src, save=out, show=show):
        pass
    return out

Refiner

Refiner(model=None, conf=0.5, mode='full', iou=0.5, names=None, votes=1, best=True, size=256, hist=25, workers=0, samples=4, every=5, cell=128, cols=None)

Correct low confidence tracks with a second model and cache the answers.

Two modes:

full Run the secondary on the whole frame, match its boxes to the tracks by IoU, and take its box, confidence and class. Best when the secondary is a grounding model or a heavier detector. crop Cut each low confidence track out of the frame and ask the secondary what it is. Best when the secondary is a chat VLM that classifies but does not localise. collage Gather samples crops of the same track, every frames apart, tile them into one image and ask about that. Best when one frame is not enough to tell, so an attribute like which way someone is facing or what they are carrying.

Every answer is a vote against the track id, so a track keeps its corrected class on later frames without the secondary running again.

Parameters:

Name Type Description Default
model Model | None

the secondary. Needs find in full mode, name in crop mode, grid in collage mode.

None
conf float

tracks at or below this confidence are sent to the secondary. Set it to 1.0 to send every track, which is what an attribute question usually wants.

0.5
mode str

"full", "crop" or "collage".

'full'
iou float

minimum IoU to match a secondary box to a track, full mode only.

0.5
names dict[int, str] | list[str] | None

class id to name mapping. Falls back to whatever the primary reports.

None
votes int

stop asking about a track once it has this many votes. Crop and collage modes.

1
best bool

keep only the best IoU match per track instead of every match above iou.

True
size int

how many track ids to keep in the vote cache, and how many may be part way through a collage at once.

256
hist int

how many votes to keep per track id.

25
workers int

run the secondary on this many background threads instead of blocking the frame loop. 0, the default, blocks. Crop and collage modes.

0
samples int

crops per collage, collage mode only.

4
every int

frames between one track's crops, collage mode only. 0 takes one every frame.

5
cell int | tuple[int, int]

collage cell size, one number for a square or a (width, height) pair. Crops are resized to it as they are gathered, so the memory held is size x samples cells at most.

128
cols int | None

columns in the collage. Defaults to a square-ish grid.

None
Source code in vizor/refine.py
def __init__(
    self,
    model: "Model | None" = None,
    conf: float = 0.5,
    mode: str = "full",
    iou: float = 0.5,
    names: "dict[int, str] | list[str] | None" = None,
    votes: int = 1,
    best: bool = True,
    size: int = 256,
    hist: int = 25,
    workers: int = 0,
    samples: int = 4,
    every: int = 5,
    cell: "int | tuple[int, int]" = 128,
    cols: "int | None" = None,
):
    if mode not in MODES:
        raise ValueError(f"mode must be one of {sorted(set(MODES))}, got {mode!r}")
    self.model = model
    self.conf = float(conf)
    self.mode = MODES[mode]
    self.iou = float(iou)
    self.names = names
    self.votes = int(votes)
    self.best = bool(best)
    self.size = int(size)
    self.cache = Vote(size=size, hist=hist)
    self.workers = max(0, int(workers))
    self.samples = max(1, int(samples))
    self.every = max(0, int(every))
    self.cell = (int(cell), int(cell)) if isinstance(cell, (int, float)) else tuple(cell)
    self.cols = cols
    self.pool = None
    self.jobs = []     # (track ids, future) still in flight
    self.busy = set()  # track ids the secondary is already looking at
    self.shots = OrderedDict()  # track id -> [frame of the last crop, tiles]
    self.frame = -1    # frames seen, so collage mode can space its samples

wait

wait()

Block until every request in flight has come back, then bank the votes.

Only useful with workers. The votes land too late for the frames that triggered them, but they are there for whatever you refine next.

Source code in vizor/refine.py
def wait(self):
    """Block until every request in flight has come back, then bank the votes.

    Only useful with ``workers``. The votes land too late for the frames that
    triggered them, but they are there for whatever you refine next.
    """
    for _, fut in list(self.jobs):
        fut.exception()
    self._drain()

close

close()

Stop the workers. Anything still in flight is dropped.

Source code in vizor/refine.py
def close(self):
    """Stop the workers. Anything still in flight is dropped."""
    self.jobs.clear()
    self.busy.clear()
    self.shots.clear()
    if self.pool is not None:
        self.pool.shutdown(wait=False, cancel_futures=True)
        self.pool = None

reset

reset()

Forget every vote and drop anything in flight. Call this between videos.

Source code in vizor/refine.py
def reset(self):
    """Forget every vote and drop anything in flight. Call this between videos."""
    self.cache.clear()
    for _, fut in self.jobs:
        fut.cancel()
    self.jobs.clear()
    self.busy.clear()
    self.shots.clear()
    self.frame = -1

run

run(tracks, img=None, preds=None)

Refine tracks in place and return them.

img is the current frame. preds lets you supply the secondary's output yourself, which skips the model call entirely.

Source code in vizor/refine.py
def run(self, tracks, img=None, preds=None):
    """Refine ``tracks`` in place and return them.

    ``img`` is the current frame. ``preds`` lets you supply the secondary's
    output yourself, which skips the model call entirely.
    """
    if not isinstance(tracks, Tracks):
        tracks = Tracks(tracks)
    if self.names is None:
        self.names = tracks.names
    self.frame += 1
    if len(tracks):
        if self.mode == "full":
            self._full(tracks, img, preds)
        elif self.mode == "crop":
            self._crop(tracks, img)
        else:
            self._collage(tracks, img)
        self._apply(tracks)
    tracks.names = self.names if self.names is not None else tracks.names
    tracks.img = img if img is not None else tracks.img
    return tracks

Vote

Vote(size=256, hist=25)

Majority vote per track id, capped at size ids and hist votes each.

Ids below zero are untracked detections that share the same placeholder id, so they are ignored instead of being pooled together.

Source code in vizor/vote.py
def __init__(self, size=256, hist=25):
    self.size = int(size)
    self.hist = int(hist)
    self.data = OrderedDict()

add

add(id, cls)

Record one vote of cls for track id.

Source code in vizor/vote.py
def add(self, id, cls):
    """Record one vote of ``cls`` for track ``id``."""
    id = int(id)
    if id < 0:
        return
    dq = self.data.get(id)
    if dq is None:
        dq = self.data[id] = deque(maxlen=self.hist)
        while len(self.data) > self.size:
            self.data.popitem(last=False)
    self.data.move_to_end(id)
    dq.append(int(cls))

get

get(id, default=None)

Most common class voted for id, or default if it has no votes.

Source code in vizor/vote.py
def get(self, id, default=None):
    """Most common class voted for ``id``, or ``default`` if it has no votes."""
    id = int(id)
    dq = self.data.get(id) if id >= 0 else None
    if not dq:
        return default
    self.data.move_to_end(id)
    return Counter(dq).most_common(1)[0][0]

count

count(id)

How many votes id has, capped at hist.

Source code in vizor/vote.py
def count(self, id):
    """How many votes ``id`` has, capped at ``hist``."""
    dq = self.data.get(int(id))
    return len(dq) if dq else 0

drop

drop(id)

Forget one track id. Does nothing if it was never seen.

Source code in vizor/vote.py
def drop(self, id):
    """Forget one track id. Does nothing if it was never seen."""
    self.data.pop(int(id), None)

clear

clear()

Forget every track id.

Source code in vizor/vote.py
def clear(self):
    """Forget every track id."""
    self.data.clear()