Building Something That Didn't Exist: A Phone-Only VBT System for Olympic Weightlifting

Velocity-Based Training (VBT) is the practice of measuring how fast the bar moves on a lift, rep to rep, and using that speed to guide training decisions — whether to add weight, whether to stop the set, whether fatigue is creeping in before it shows up anywhere else. In Olympic weightlifting it's closer to essential than optional: bar speed on the Snatch and Clean & Jerk tells you things RPE and a stopwatch can't.

Normal VBT means hardware. A linear position transducer clips to the bar and a cable pot measures displacement directly; camera-based systems like GymAware or Vmaxpro use dedicated depth or stereo cameras built for exactly one job. They're accurate because they're purpose-built, and they cost accordingly — often several hundred to a couple thousand dollars — and they're one more piece of kit to bring to the gym, charge, and calibrate.

This system doesn't use any of that. There is no transducer, no dedicated camera, no depth sensor — just a phone's camera and a video. That's the thing that didn't exist before this: velocity, force, power, and full phase-by-phase breakdown (First Pull, Turnover, Catch, Dip/Drive/Receive) computed entirely from footage an ordinary camera can shoot, with the entire pipeline — object detection, pose estimation, filtering, physics — running on-device, frame by frame, no server round-trip. Standard VBT tools measure a transducer's cable or a purpose-built camera's depth data. This measures pixels.

This post is about how that pipeline works — and about VBT Lab, the Mac debug harness I built alongside it, which is what actually made building the pipeline possible without a phone glued to my hand for every parameter tweak.

The pipeline, top to bottom

Detect — a custom-trained Core ML object detector finds the barbell plate in each frame via Vision (VNCoreMLRequest / VNRecognizedObjectObservation).
Track — detections get smoothed frame-to-frame (IOU-based box smoothing) and can be locked onto a specific plate with a tap, so the tracker doesn't jump to a plate in the background.
Pose — Apple Vision's body pose estimation runs on the same decoded frame, at the same timestamp, as the plate detector.
Analyze — bar-path samples get resampled to a uniform 60Hz, filtered, differentiated into velocity, and combined with the athlete's height (as a real-world scale reference) to get force and power per rep.
Segment — a lift is broken into phases (First Pull, Turnover, Catch, Recovery, Dip/Drive/Receive, etc.), each rendered as a distinct color along the bar path.

None of this touches a server. It runs frame-by-frame on-device.

Running detection and pose in lockstep, not in parallel

The obvious way to add pose tracking after the bar tracker already existed is to run a second, independent decode pass over the video and merge the two streams by timestamp afterward. I didn't do that, because two independent AVAssetReader passes drift — even small frame-timing differences mean the bar position and the joint positions you're comparing were never actually the same instant.

Instead, VideoDetectionViewModel exposes an onFrameForExternalProcessing hook that fires with the exact pixel buffer being fed to the Core ML detector. Pose extraction subscribes to that hook rather than opening its own decode session. Same buffer, same timestamp, every frame. It's a small architectural choice, but it removes an entire class of sync bugs before they can happen.

Detection itself is adaptively throttled — instead of running the model at a fixed rate, the throttle targets a CPU utilization ceiling, backing off frame-analysis frequency when the device is under load rather than dropping into a fixed low frame rate up front.

How phase segmentation actually works

Each colored segment on the bar path (First Pull in yellow, Turnover in purple, Catch in green, Recovery in blue, Dip in orange, and so on) comes from a small, independent function called a "colorer" — there's no shared LiftPhaseColorer protocol, just a family of enums like SnatchFirstPullColorer, CleanTurnoverColorer, and JerkDipColorer, each exposing one static function that takes the bar path, the pose data, and body metadata (bar weight, athlete height), and returns an array of point runs to draw.

First Pull is the simplest: it uses the knee joint from pose data as a literal reference line. KneeHeightSignal flips the bar path's bottom-left coordinate origin to match pose's top-left origin, then a sample counts as "below the knee" when its flipped Y is greater than the knee's Y. The colorer walks forward from the start of the lift up to the timestamp of peak bar velocity, and keeps every contiguous stretch where the bar is below that knee line as the First Pull run.

Turnover (Clean) and Dip (Jerk) both lean on the same underlying primitive: a confirmed local minimum in bar height. BarDescentSignal.findFirstConfirmedBottom scans forward from a starting point and only accepts a candidate bottom once the bar has been rising for requiredConfirmations consecutive samples and stayed down for at least minDwellSeconds (0.08s by default) — the same "don't trust a reversal until it's sustained" pattern used in the lift-start detector, applied again here to stop a jittery frame from being read as the bottom of a squat-under. Turnover's boundaries chain three signals in sequence: peak velocity → first confirmed peak in bar height after that → first confirmed bottom after that (the catch). Because a numerically "confirmed" bottom lands slightly later than where a human eye would call the catch, the actual catch-start timestamp is nudged earlier by a small tunable catchPullbackSeconds (0.075s by default): catchStartTime = max(peakVelocityTime, catchBottomTime - catchPullbackSeconds). Dip reuses the exact same confirmed-bottom primitive, just anchored at the start of the recording instead of after peak velocity, since a jerk dip has no prior pull to reference.

Critically, boundary computation is split out from the colorers into shared helper structs (CleanTurnoverBoundaries, JerkBoundaries) so that, say, Turnover's end and Catch's start are computed once and handed to both colorers — they can't quietly disagree about where one phase ends and the next begins.

There's no central "phase machine" walking through Snatch → First Pull → Turnover → Catch → Recovery in one pass. ContentView switches on the lift type and calls each colorer independently into its own state slot; for a Clean & Jerk, that means running all of Clean's colorers and all of Jerk's colorers back to back. The segments come back as plain arrays of points and get stroked directly as SwiftUI Paths with fixed colors over the video — the "phase overlay" is just several independently-computed line segments layered on the same coordinate space.

Scoring runs on a separate track from the colorers entirely. LiftScorer slices the trimmed bar-path samples between manually-placed phase markers (not the colorer output) and scores each slice on two axes via RecordingFormScorer: 60% weight on lateral drift (how much the bar wanders side to side, penalized past a tolerance of 0.12 standard deviations) and 40% on vertical smoothness (how far the bar path strays from a straight line, penalized past an RMS residual of 0.06). The visual phase colors and the numeric form score are deliberately decoupled — one is for a coach's eye, the other for a number the lifter can track over time.

The hard part wasn't the ML model

Object detection, once you have a labeled dataset, is a solved problem you consume rather than build. The interesting bugs were all in the signal processing layer that turns a wobbly bar-position trace into "this is where the pull started."

BarPathLiftStartSignal answers one question: given noisy bar-height-over-time data for a Snatch or Clean, at what timestamp did the lifter actually start pulling? The algorithm:

Requires the first minRestSeconds of filtered position to sit flat within a restBandMeters band — this establishes a resting baseline. If the clip doesn't open on a settled bar, it returns nil rather than guessing.
Scans forward for the first point that rises minRiseMeters above that baseline.
Requires the rise to be sustained — requiredConfirmations consecutive samples over a minimum dwell time — before committing, so a camera shake or detection jitter can't masquerade as the start of a pull.

I went through two approaches that quietly produced wrong answers before landing here: one relied on an internal index from the velocity analyzer that didn't line up with the raw bar-path timeline, and another tried to reverse the existing descent-detection scan, which found the wrong local extremum on lifts with a slow, controlled setup. Both failed silently — no crash, just a start time that was off by a fraction of a second in a way that only showed up when you overlaid the phase colors on the video and watched the boundary land in the wrong place. That's the class of bug that on-device signal processing keeps producing: the code runs fine, the output is just quietly not what happened.

Rep counting had the same flavor of bug. Two failure modes showed up in testing: standing up out of the catch could get counted as a second rep, and a bar bouncing off the floor after a drop could get counted as a rep on its own. Both fixes came from watching the phase overlay against real footage and tightening the state machine, not from a unit test — the ground truth here is a video, not an assertion.

Turning a bar-position trace into velocity, force, and power

VBTAnalyzer is where the bar path stops being geometry and becomes a physics readout. The pipeline: resample the raw (and irregularly-timed) bar-path samples to a uniform 60Hz, run them through a hand-derived 2nd-order Butterworth low-pass filter (12Hz cutoff, coefficients b=[0.2066, 0.4132, 0.2066], a=[1.0, -0.3695, 0.1958]) applied forward-backward (filtfilt, with a 9-sample reflect-pad at each edge) so filtering removes sensor noise without introducing phase lag or corrupting the first/last few frames. Velocity and acceleration come from central-difference derivatives of the filtered position, and force falls out of Newton's second law directly: force = mass × (acceleration + 9.81).

Two bugs from this layer are worth calling out because they're the kind that don't crash, they just quietly produce a plausible-looking wrong number. A sign-flip bug: an old (1.0 - y) coordinate flip inverted the velocity sign, which made the concentric-phase detector latch onto the bar's descent instead of its ascent — the rep counter was, for a while, confidently timing the wrong half of the lift. And the snatch double-counting bug: a snatch has two upward-velocity phases per rep — the pull/turnover, then standing up out of the catch — separated by a support hold that looks structurally identical to a real inter-rep rest. The fix checks whether the bar has actually returned near its floor position before starting a new rep, combined with a minimum-displacement filter that rejects a dropped bar bouncing off the floor as if it were a rep.

There's also a diagnose() pass that exists purely to catch its own pipeline lying to it: bar velocities over 2.0 m/s get flagged as world-record-implausible (almost certainly a detection glitch, not a real lift), and effective sample rates under 10Hz get flagged as "mostly interpolated" so a low-quality video doesn't get presented with the same confidence as a clean 60fps one.

Splitting a combined Clean & Jerk, and tracking velocity over time

A Clean & Jerk is usually filmed as one continuous clip, but it's really two separate lifts stitched together by a rack hold. CleanJerkSplitSignal finds that seam by reusing the same boundary primitives as the phase colorers — it locates the Clean's lockout via CleanTurnoverBoundaries and BarPeakHeightSignal, then scans the velocity curve after lockout for a sustained near-zero stretch. It splits at the midpoint of that plateau rather than either edge, specifically to avoid the split landing during bar-settling right after the catch or during the earliest onset of the jerk dip. For an efficient lifter with barely any pause between the two lifts, there's no plateau to find, and the signal just returns nil — the split isn't attempted rather than guessed at.

Above the single-lift level, SpeedVBTTrendEngine tracks velocity trends across sessions. Sets at the same working weight get averaged into "rungs," and a session missing exactly one rung from its usual pattern gets that rung linearly interpolated — but only when it's short by exactly one, and only against a "canonical wave size" computed as the mode of the lifter's historical rung counts (ties break toward the larger wave). Whether to tell a lifter they're ready to add weight requires a run of consecutive trending-up session-pairs, independently confirmed on every rung of the wave — one strong rung doesn't override a flat or declining one elsewhere in the same session.

The MediaPipe detour, and why Vision won out

Pose tracking wasn't always Vision-based. There's a DeferredPose/ folder still in the repo containing a full MediaPipe-based pose implementation (LivePoseTrackingViewModel, a bundled .task model) that's deliberately excluded from the build target. The reason is unglamorous but real: MediaPipe's MediaPipeTasksVision.xcframework only ships iOS device and simulator binary slices — no Mac Catalyst slice exists — so it simply cannot link into a Catalyst target no matter what the code does. Rather than rebuild MediaPipe from source for Catalyst, the pose layer was already written behind a PoseBackend abstraction, so swapping the implementation over to AppleVisionPoseService — Vision, native to every Apple platform including Catalyst — was a backend swap, not a rewrite. It's a small case study in why abstracting a dependency you don't control pays off the moment that dependency's platform support doesn't match yours.

That abstraction is also why VBT Lab — the Mac tool in this repo — exists at all in its current form. Per the project's HANDOFF.md, it's a debug harness extracted wholesale from a separate production iOS weightlifting app, specifically so the bar-detection and VBT pipeline could be iterated on without a phone deploy in the loop. Going Catalyst instead of a native macOS rewrite meant the UIKit/AVFoundation/Vision/CoreML code that already existed could run unmodified; the tradeoff shows up in two Catalyst-only stubs — WatchSessionManager (WatchConnectivity doesn't exist on Catalyst at all) and AuthenticationManager (stubbed to fall back to the most recently used UserProfile) — plus a short, explicit list of screens and program-linkage types left behind because they weren't needed for pipeline debugging.

Storage: leave the file where it is

Videos come from two places: the in-app camera, and imports from Files/iCloud. For camera recordings, the app owns the file and manages its lifecycle normally. For imports, RecordingStorageManager deliberately does not copy the file into the app sandbox — it keeps a security-scoped bookmark to the original location (startAccessingSecurityScopedResource around every access) and stores the absolute path. That avoids silently duplicating what might be a multi-hundred-megabyte 4K clip just to analyze it once.

The tradeoff is that storage now has to track two different provenance types instead of one, and evictLocalFiles — which prunes older local recordings to reclaim space once they're uploaded, while keeping the SwiftData record and thumbnail for gallery display — has to know which files it's actually allowed to delete.

Exporting exactly what's on screen

The app renders bar-path overlays and phase-colored trails as SwiftUI views on top of the video. When a user exports a still image of a rep, it needs to look exactly like what they saw live. Rather than building a second rendering path for export, the same overlay views get handed to SwiftUI's ImageRenderer and composited straight to file — raw frame, bar-path-only, and full-phase variants. There's no separate export renderer to keep in sync with the live one, because there isn't one.

Debugging an on-device ML pipeline without a phone in your hand

Tuning a Core ML detector's confidence thresholds, a Butterworth filter's cutoff, or the rest/rise thresholds in the lift-start signal means changing one number and immediately wanting to see it against ten different real lift videos. Doing that by deploying to a physical iPhone every time is slow enough to kill the iteration loop.

The fix was almost embarrassingly simple given the app is already SwiftUI: the entire Minerva module — detection, pose, lift analysis, VBT math — is portable enough that it runs unmodified as a Mac Catalyst target (project.yml generated via XcodeGen, SUPPORTS_MACCATALYST: YES). "VBT Lab" itself, the app in this repo, is that Mac debug harness: a UI to pick a lift type and bar weight, import a recorded clip, and watch bar-path, pose, and phase segmentation update live as parameters change — no phone required. It's not the shipping product; it's the tool that made the shipping product's pipeline debuggable.

What I'd take away from this

The ML model was the part that felt hardest going in and turned out to be the least interesting part of the system once trained. The actual engineering effort went into: keeping two real-time signal sources in sync without drift, writing a rest-then-rise detector that fails loudly (returns nil) rather than quietly guessing wrong, and building a second target for the exact same code so debugging didn't require a device in hand. None of that shows up in a demo GIF, but it's the difference between a bar-tracking app that works on the videos you tested it on, and one that works on the video a lifter hands you cold.