Developer guide

Using your exercise JSON

Markerless exports an exercise as data: the stages that make up a rep, the joint angles each one requires, any corrective feedback you defined, and the exact definition of every angle. Anything that gives you 2D body keypoints can run it — MediaPipe, ML Kit, Apple Vision, MoveNet, YOLO-pose, or a model of your own.

01

What is in the file

Schema version 1. Every field, annotated:

json
{
  "schema": 1,
  "generator": "Markerless AI",
  "exercise": {
    "id": "bicep_curl",           // file name to load, and the CustomExercise id
    "name": "Bicep Curl",
    "description": "Custom exercise created with Markerless AI"
  },
  "angle_convention": "degrees, clockwise; min > max wraps through 0°",

  // How to reproduce every angle from raw keypoints, on any pose engine.
  "angle_definitions": {
    "method": "The directed angle at the vertex keypoint, swept clockwise …",
    "joints": {
      "shoulder": { "from": "elbow",    "vertex": "shoulder", "to": "hip" },
      "elbow":    { "from": "shoulder", "vertex": "elbow",    "to": "wrist" },
      "hip":      { "from": "shoulder", "vertex": "hip",      "to": "knee" },
      "knee":     { "from": "hip",      "vertex": "knee",     "to": "ankle" }
    }
  },

  "options": { "tolerance": 20, "either_side": false },

  // Only the joints your stages constrain — these become requiredFeatures.
  "required_joints": [{ "joint": "elbow", "side": "left" }],

  // The keypoints a pose engine must provide to run this file.
  "required_keypoints": ["left_shoulder", "left_elbow", "left_wrist"],

  "stages": [
    {
      "id": "stage_1",            // positional; corrections point at these
      "name": "Extended",
      "description": "Extended",
      "requirements": [
        {
          "joint": "elbow",       // shoulder | elbow | hip | knee
          "side": "left",         // left | right
          "min": 154,             // inclusive, degrees
          "max": 194,
          "captured": 174         // the angle Markerless measured, before tolerance
        }
      ]
      // "alternative_requirements": [[ ... ]]  — only when Either side is on
    }
  ],

  "corrections": [
    {
      "id": "…",
      "stage_id": "stage_2",      // which stage this fault belongs to
      "name": "Elbow drifting forward",
      "message": "Keep your elbow tucked in",
      "requirements": [ /* the WRONG pose, same shape as a stage */ ]
    }
  ]
}

Three things worth knowing:

  • Angles are clockwise degrees, 0–360, matching the readouts in Markerless. A requirement where min is greater than max wraps through zero — 340 to 20 is a 40° band around the top, not an empty one.
  • Tolerance is already baked in. min and max are the captured angle ± the ROM tolerance you chose at export. captured is kept so you can widen a band by hand later without going back to Markerless.
  • Not every joint appears. Markerless constrains only joints that actually move during the exercise, so a curl does not force the user to hold their knees still.

Any pose engine

The code below is TypeScript because it reads easily. The same runtime ships ready-made in Swift, Kotlin and Python too — every version is tested against the engine’s own angles on the same clip.

02

Read twelve keypoints

A rep counter uses four joints on each side — shoulder, elbow, hip and knee — and each angle is made from three keypoints. That needs the shoulders, elbows, wrists, hips, knees and ankles, which every mainstream pose model provides:

Keypoint (left / right)MediaPipe, ML KitCOCO-17: MoveNet, YOLO-poseApple Vision
shoulder11 / 125 / 6.leftShoulder / .rightShoulder
elbow13 / 147 / 8.leftElbow / .rightElbow
wrist15 / 169 / 10.leftWrist / .rightWrist
hip23 / 2411 / 12.leftHip / .rightHip
knee25 / 2613 / 14.leftKnee / .rightKnee
ankle27 / 2815 / 16.leftAnkle / .rightAnkle

Left and right are the subject’s own, not the viewer’s, in all of them. Wrists and ankles are only there to give the elbow and knee their far segment — a counter never constrains them directly.

Each file’s required_keypointssays exactly which of the twelve that exercise needs, so an app on a custom or partial model can check before it counts — the runtimes’ missingKeypoints() does it in one call.

You rarely need to write this mapping yourself. Pick your engine under Run on in the Export tab and it hands you the runtime plus an adapter that already does it — for Apple Vision, ML Kit, MediaPipe on Android, the web or Python, TensorFlow.js, COCO-17 models such as YOLO-pose, and QuickPose on either platform — in Swift, Kotlin, TypeScript or Python.

03

Measure the angles the same way

Each angle is the directed sweep at the middle keypoint, clockwise on screen from the first segment to the second. That definition travels inside every exported file, under angle_definitions, so the file is enough on its own:

typescript
export type Point = { x: number; y: number };

/** Directed angle at `vertex`, swept clockwise on screen from `from` to `to`, 0–360°. */
export function sweep(from: Point, vertex: Point, to: Point): number {
  const ux = from.x - vertex.x, uy = from.y - vertex.y;
  const vx = to.x - vertex.x, vy = to.y - vertex.y;
  const deg = (Math.atan2(ux * vy - uy * vx, ux * vx + uy * vy) * 180) / Math.PI;
  return deg < 0 ? deg + 360 : deg;
}

// The three keypoints behind each angle — all on the same side.
// Also in the file itself, under angle_definitions.joints.
export const ANGLES = {
  shoulder: ["elbow", "shoulder", "hip"],
  elbow: ["shoulder", "elbow", "wrist"],
  hip: ["shoulder", "hip", "knee"],
  knee: ["hip", "knee", "ankle"],
} as const;

/** `keypoints` in PIXELS, y pointing down, keyed "left_elbow", "right_knee", … */
export function jointAngles(keypoints: Record<string, Point>): Record<string, number> {
  const out: Record<string, number> = {};
  for (const side of ["left", "right"]) {
    for (const [joint, names] of Object.entries(ANGLES)) {
      const [a, b, c] = names.map((n) => keypoints[`${side}_${n}`]);
      if (a && b && c) out[`${side}_${joint}`] = sweep(a, b, c);
    }
  }
  return out;
}

Three things change the number, so get them right once:

  • Pixels, not normalised coordinates. Multiply x by the frame width and y by its height. On a portrait phone frame, normalised coordinates squash the picture and put angles 8–16° out.
  • y points down. Engines that put the origin at the bottom left — Apple Vision among them — need height − y, or every angle comes out as 360° minus the right one.
  • Un-mirrored frames. A mirrored front-camera preview reverses the sweep the same way. Measure on frames the right way round, as the clip you built the exercise from was.
04

Count reps and show cues

Keep a pointer to the stage you are waiting for. When a frame’s angles meet it — its requirements, or any one of its alternative sets — move on to the next. Meeting the last stage completes a rep and sends you back to the first. The holding flag stops one pose satisfying two neighbouring stages in the same moment.

typescript
// The parts of the exported file the counter reads.
export type Requirement = { joint: string; side: string; min: number; max: number };
export type Stage = { id: string; name: string; requirements: Requirement[]; alternative_requirements?: Requirement[][] };
export type Correction = { stage_id: string | null; name: string; message: string; requirements: Requirement[] };
export type Exercise = { schema: number; stages: Stage[]; corrections: Correction[]; required_keypoints?: string[] };

// min > max means the band wraps through 0°, e.g. 340–20.
const inBand = (angle: number, r: Requirement) =>
  r.min > r.max ? angle >= r.min || angle <= r.max : angle >= r.min && angle <= r.max;

const meets = (reqs: Requirement[], angles: Record<string, number>) =>
  reqs.every((r) => {
    const angle = angles[`${r.side}_${r.joint}`];
    return angle !== undefined && inBand(angle, r);
  });

export class RepCounter {
  reps = 0;
  private exercise: Exercise;
  private waitingFor = 0; // index of the stage we need next
  private holding = false; // still in the stage we just matched

  constructor(exercise: Exercise) {
    this.exercise = exercise;
  }

  /** Name of the stage the counter is waiting for — handy while testing. */
  get stageWaitingFor(): string {
    return this.exercise.stages[this.waitingFor].name;
  }

  /** Call once per frame. Returns the running rep count. */
  update(angles: Record<string, number>): number {
    const stage = this.exercise.stages[this.waitingFor];
    const met =
      meets(stage.requirements, angles) ||
      (stage.alternative_requirements ?? []).some((alt) => meets(alt, angles));

    if (!met) {
      this.holding = false;
    } else if (!this.holding) {
      this.holding = true;
      this.waitingFor = (this.waitingFor + 1) % this.exercise.stages.length;
      if (this.waitingFor === 0) this.reps += 1; // meeting the last stage completes a rep
    }
    return this.reps;
  }

  /** The first fault the pose matches, if any — show its `message` as the cue. */
  fault(angles: Record<string, number>): Correction | undefined {
    return this.exercise.corrections.find((c) => c.requirements.length > 0 && meets(c.requirements, angles));
  }
}

// Once, with the exported file — bundled, or fetched from your server:
const counter = new RepCounter(loadExercise(json));

// Then per frame, with your engine's keypoints converted to pixels:
const angles = jointAngles(keypoints);
const reps = counter.update(angles);
const cue = counter.fault(angles)?.message;

Faults are checked separately from the count, against the same angles. Use stage_id if you only want a cue while its stage is the one being waited on. The downloadable runtime adds loadExercise(), which refuses a schema newer than it understands, and missingKeypoints().

05

Check it on your engine

The formula is exact: computed from the QuickPose engine’s own keypoints, it agrees with that engine’s angles to within 0.1° on our reference clip. What differs between engines is where each model puts the keypoints — one model’s hip can sit a little higher than another’s — and the requirement bands were captured from Markerless’s tracking.

The tolerance band, ±20° by default, absorbs most of that. Still, try each exercise on your engine before shipping it. If a stage is missed, widen its band around captured, or export again with a larger ROM ±.

QuickPose SDK, iOS and Android

The shortcut on either platform: a ready-made runtime that reads the angles straight out of QuickPose’s features, so there is no pose code to write. Steps 06–09 are iOS; step 10 is Android. The QuickPose SDK is licensed separately.

06

Add the two runtime files (once per project)

Open an exercise, go to the Export tab, and copy these into your Xcode project:

  • CustomExerciseEngine.swift — the rep-counting runtime. It tracks which stage you are on and counts a rep each time the sequence completes.
  • QuickPoseTrainLoader.swift — decodes an exported JSON file into the CustomExercise the engine expects.

Neither file is exercise-specific, and neither needs editing. You only replace them when you update to a newer schema.

07

Add the exercise itself

Back in the Export tab, hit Download. The file is named after the exercise id — bicep_curl.json — which is also what you pass to the loader.

Drag it into Xcode and make sure it is ticked for your app target, so it lands in Copy Bundle Resources. Or skip the bundle entirely:

swift
// Fetch exercises instead of bundling them, and you can add or retune
// movements without shipping an app release.
let data = try await URLSession.shared.data(from: exerciseURL).0
let loaded = try QuickPoseTrain.load(data: data)

That is the real reason to prefer JSON over generated code — retuning an angle or rewording a cue stops being an App Store release. The same goes for any engine: fetch the file rather than bundling it.

08

Load it and count reps

swift
import QuickPoseCore

// Once, when the exercise screen appears:
let loaded = try QuickPoseTrain.load(named: "bicep_curl")
let engine = CustomExerciseEngine(exercise: loaded.exercise)

// `exerciseDefinition` is the SDK `Exercise`, built from the joints your
// stages actually constrain. Hand it to QuickPose exactly as you do today.
let definition = loaded.exercise.exerciseDefinition

Then feed the engine each frame:

swift
// In your per-frame callback, where `features` is the
// [QuickPose.Feature: QuickPose.FeatureResult] dictionary QuickPose gives you:
let reps = engine.processFrame(features: features)

// The engine also publishes these, so SwiftUI updates on its own:
//   engine.currentReps       Int
//   engine.currentStage      String   — name of the stage being waited on
//   engine.feedbackMessage   String   — empty when hideFeedback is true
//   engine.incorrectJoints   Set<JointType>
//   engine.newRepCompleted   Bool

Setting up the QuickPose session itself does not change — the loader only builds the exercise and its required features for you.

09

Show corrective feedback

Each fault you recorded in Markerless carries the wrong pose and the cue to show for it. The engine does not check faults itself, so test them where you process the frame:

swift
// Faults you defined in Markerless come back alongside the exercise.
let angles = QuickPoseTrain.angles(from: features)

if let fault = loaded.corrections.first(where: { $0.matches(angles) }) {
    cue = fault.message      // e.g. "Keep your elbow tucked in"
} else {
    cue = nil
}

QuickPoseTrain.angles(from:) reads the joint angles out of the same features dictionary you already pass to processFrame. Use fault.stageId if you only want to show a cue while its stage is the one being waited on.

10

On Android

Pick QuickPose SDK under Android in the Export tab. You get the Kotlin runtime, MarkerlessRepCounter.kt, and QuickPoseAngles.kt, which asks QuickPose for the range-of-motion features the exercise needs — clockwise, to match Markerless — and hands their values to the counter:

kotlin
val json = context.assets.open("bicep_curl.json").bufferedReader().use { it.readText() }
val exercise = MarkerlessExercise.parse(json)
val counter = MarkerlessRepCounter(exercise)

quickPose.start(QuickPoseAngles.features(exercise), onStart = {}) { status, overlay, features, feedback, landmarks ->
    val angles = QuickPoseAngles.from(features)
    val reps = counter.update(angles)
    val cue = counter.fault(angles)?.message
}

Faults work the same way as everywhere else: counter.fault(angles) returns the first one the pose matches.

11

Changing an exercise later

Edit the stages or feedback in Markerless, export again, and replace the JSON. Stage ids are positional (stage_1, stage_2, …), so reordering stages reassigns them — if you store anything keyed by stage id on your side, re-check it after a reorder.

Check schema before you trust a file. The Swift loader refuses a schema newer than it understands rather than guessing, so a stale copy fails loudly instead of silently mis-counting — do the same in your own decoder.

When it does not work

Reps never count
Usually the tolerance is too tight for real movement. Raise ROM ± in the Export tab, re-download, and try again. Log which stage the counter is waiting on while you test — on the QuickPose runtime, load with hideFeedback: false and engine.feedbackMessage says so.
Reps count instantly, without moving
A stage whose requirements are always satisfied. That happens when the stages are too similar to each other, or the tolerance is wide enough that two stages overlap. Add a stage at a more distinct position, or lower the tolerance.
It counts only when I lead with one side
Turn on Either side before exporting. Each stage then carries a mirrored alternative_requirements set, and the rep counts whichever limb leads.
On my engine, every angle is roughly 360° minus what Markerless shows
The sweep is running the other way. Either your y axis points up — Apple Vision puts the origin at the bottom left, so use height − y — or the frames are mirrored, as a front-camera preview often is. Feed the engine un-mirrored frames, or flip x with width − x before measuring.
“Could not find …json in the app bundle” (QuickPose runtime)
The file is in your project but not in the target. In Xcode, select the JSON and check the target under Target Membership, or find it under Build Phases → Copy Bundle Resources.
“This exercise uses JSON schema N” (QuickPose runtime)
The file was exported by a newer version of Markerless than your copy of the loader. Re-download QuickPoseTrainLoader.swift from the Export tab.

Ready to build one?

Upload a clip of the movement, tag the stages that make up a rep, and export.

Open Markerless