← Back to blog
Blog
2026-07-28

Building CoachYu's Bar Path Tracker: A YOLO-Powered Microservice for Weightlifting Analysis

If you've ever filmed yourself squatting or deadlifting and wished you could see exactly how the bar moved through space, that's the problem CoachYu's AI service solves. Here's how I built it.

The problem

Barbell path tracking is a well-known concept in strength training: overlay the trajectory of the bar's center over a lift video, and you can immediately spot inefficiencies — bar drifting forward on a squat, looping on a deadlift, etc. To do this automatically from a phone video, you need to:

  1. Detect the barbell plate in every frame
  2. Track it reliably even as the lifter moves, occludes it, or the camera shakes
  3. Turn that into a compact, easy-to-consume trajectory for a mobile app

Why this had to run server-side rather than on-device: the real trigger was simple — not every user's phone can actually run the AI locally. On iOS, this kind of object tracking can lean on Apple's Vision framework and CoreML running directly on the phone. Android has no equivalent first-party framework with the same maturity, and even where something similar exists, device fragmentation makes on-device model performance wildly inconsistent across the Android install base — a flagship phone might handle it fine, a budget device won't. Rather than gate the feature behind "does your phone support this," I moved the AI to the server so it works identically no matter what phone someone owns. The client just records and uploads a video; all the heavy lifting happens on a server I control.

I built this as a standalone Python microservice (coachyu-ai-service) that a Go backend worker calls over HTTP, rather than baking model inference into the main application. It's live and in beta now.

Architecture at a glance

  • FastAPI app (main.py) exposing /health and /process
  • Ultralytics YOLO model for plate detection, loaded once at startup via FastAPI's lifespan hook
  • OpenCV for frame-by-frame video decoding
  • Google Cloud Storage for input video retrieval (the Go worker uploads the video, hands this service a gs:// URL)
  • Docker + Cloud Run for deployment — a lean Python 3.11-slim image with just the native libs OpenCV needs (libgl1, ffmpeg, etc.)

The request/response contract is intentionally minimal:

class ProcessRequest(BaseModel):
    videoUrl: str          # gs://bucket/object
    recordingClientId: str
    tasks: list[str]       # ["barbell_detection", "pose_estimation"]
    mode: str = "auto"
    lock: dict[str, float] | None = None

tasks is forward-looking — pose estimation is scaffolded in the response shape (poseJson) but not implemented yet, so the Go worker already knows how to handle it gracefully when it ships.

The interesting part: auto vs. lock tracking modes

The core design decision was how to pick which plate to track when YOLO detects several (both ends of the bar, plates in the background rack, etc.).

I split this into a dedicated BarPathSelector with two modes:

  • auto: just take the highest-confidence detection each frame. Good for a first pass / simple scans.
  • lock: track a specific plate by proximity to a previously known center point, rejecting jumps beyond a max distance threshold (lock_max_distance = 0.20 of frame size). This lets a client lock onto this specific plate and not get fooled by a plate on a rack in the background suddenly having higher confidence.
best = min(detections, key=lambda detection: distance(detection.center, self.locked_center))
if distance(best.center, self.locked_center) > self.lock_max_distance:
    return None

This mirrors how the API is used in practice: the mobile client can send mode="auto" for a quick scan to find the plate, then re-send mode="lock" with a locked x, y centroid once the user (or client-side heuristic) confirms which plate to follow for the full analysis pass.

Smoothing detections without over-engineering it

Raw YOLO boxes jitter frame to frame. Rather than reach for a full Kalman filter, I used a simple exponential moving average (BoxSmoother) keyed by label, gated by IoU:

if previous and previous.iou(detection.rect) >= self.iou_threshold:
    rect = previous.ema(detection.rect, alpha=self.alpha)

If the new detection doesn't overlap enough with the last one (IoU < 0.3), it's treated as a fresh object rather than smoothed — this avoids smearing the box across a sudden real jump (e.g. a fast rep).

Normalized coordinates, Vision-framework style

All boxes are stored as NormalizedRect — x/y/width/height in [0, 1], with y measured from the bottom of the frame. That's a deliberate match to Apple's Vision framework coordinate convention. Even though the whole point of this service is to give Android parity with iOS, keeping the wire format aligned with Vision's convention meant the existing iOS client code didn't need to change at all — only Android's overlay rendering needed to account for the bottom-up y-axis. One coordinate convention, two platforms, no server-side special-casing per client.

Fail fast, ship less data

The /process endpoint does a "scan" pass over the whole video before returning anything. If no plate was ever detected, it returns a 422 with a structured error (NO_PLATES_DETECTED) rather than a silent empty result — so the Go worker and mobile client can distinguish "no bar found" from "processing succeeded, nothing to show."

The successful payload itself is deliberately compact: rather than shipping big detection objects, points are flattened into [x, y, time, x, y, time, ...] triples and the whole thing is gzipped + base64-encoded before going back over HTTP:

def encode_result(data: dict[str, Any]) -> str:
    json_bytes = json.dumps(data, separators=(",", ":")).encode("utf-8")
    compressed = gzip.compress(json_bytes)
    return base64.b64encode(compressed).decode("ascii")

For a video with thousands of tracked points, this keeps the response small and lets the Go worker just pass the blob through to storage/the client without re-parsing it.

New territory: my first AI server in Python

Worth being honest about: this was my first time building an AI-serving backend, and my first time working in Python in this kind of production, long-running-process way (loading a model once at startup and keeping it warm across requests, rather than the more scripty one-shot Python I was used to). Things that weren't obvious going in:

  • Model loading has to happen once, not per-request. Loading a YOLO model takes real time, so it has to happen at process startup (FastAPI's lifespan hook) and be reused across every /process call, not reloaded per request. Get this wrong and every request pays a multi-second tax.
  • Python's implicit state is different from what I was used to. Keeping a module-level _tracker global that gets set once and read many times isn't a pattern I'd normally reach for, but it's the standard way to share an expensive resource across FastAPI request handlers.
  • The GIL means model inference blocks the event loop. FastAPI is async, but YOLO inference via model.predict() is a synchronous, CPU/GPU-bound call — running it inside an async def route just blocks that worker for the duration of the call. It's fine at low concurrency but was a "wait, why is everything single-threaded during inference" moment.
  • Numeric/CV libraries have their own dialect. Coming from a different tech stack, getting comfortable with OpenCV's BGR frames, NumPy array shapes, and normalized coordinate conventions took more trial and error than the actual tracking logic did.

Where the model came from

The plate detector itself (models/plate-detector/best.pt) is a YOLO model fine-tuned from a public dataset, rather than trained from scratch or used off-the-shelf. Starting from an existing barbell/gym-equipment dataset meant not having to hand-label thousands of frames of lifting footage before getting a usable detector.

Deployment

It's a plain Cloud Run service: Dockerfile installs the OpenCV/ffmpeg system deps, copies the app and the trained model weights (models/plate-detector/best.pt) into the image, and runs uvicorn on Cloud Run's $PORT. Model path and confidence threshold are env-configurable, so swapping in a retrained model is a config change, not a code change.

What's next

  • Pose estimation (already reserved in the API contract via tasks and poseJson)
  • Possibly a lighter-weight scan mode using SCAN_FRAMES to sample rather than decode every frame for the initial "is there a plate here at all" check