Build a video intake QC gate

This workflow applies the API to an intake-quality-control pipeline. It focuses on detecting data that is unusable or unlikely to contain the promised work, before attempting to score the relative value of accepted datasets. Use three outcomes: pass, manual_review, and reject.

Recommended decision order

Check measurement coverage. If the planned analysis is incomplete, re-run it or use manual_review. Do not interpret low coverage as bad footage.
Check visual usability. Use distorted_share to detect unreadable footage. Use jerky_share as a review signal for unstable motion.
Check task relevance. When the measured view is expected to show a human operator’s hands, low active_manipulation_share can reveal breaks, idle footage, or recordings that contain little of the promised activity.
Review repetition and content. Use vocabulary metrics to find deliveries dominated by a small number of environments, objects, or actions.

QC question	Metric	Suggested pipeline action
Did enough of the planned analysis succeed?	`coverage_share`	Re-measure or review when coverage is below policy.
Is the video readable?	`distorted_share`	Reject or review excessive blur, occlusion, darkness, or exposure loss.
Is motion stable enough for this capture setup?	`jerky_share`	Review before rejecting because fast valid motion can trigger the signal.
Does a view that should show the operator’s hands contain interaction rather than idle time?	`active_manipulation_share`	Review or reject deliveries with too little task activity.
Is the delivery dominated by repetitive content?	`effective_distinct_terms` and `terms`	Review dominant or unrelated terms against the collection brief.

See Understand quality metrics for complete field definitions and denominator rules.

Apply metrics by capture setup

Frame quality and vocabulary apply across robotics capture setups. Use them to check visual usability and whether the observed content matches the collection brief. Add hand activity only when the measured camera view is expected to show the human operator. Low active manipulation can expose long breaks, transit, or footage where the operator is not performing the promised task. Do not use it as a proxy for robot-gripper activity. Use vocabulary after the baseline checks. A low effective term count or unrelated dominant terms can trigger a content review, but vocabulary concentration alone does not prove that recordings are duplicates or that a demonstration failed.

Integrate a conservative gate

Keep thresholds in a versioned policy rather than embedding them in API client code. This example rejects clearly unreadable footage and sends ambiguous motion or activity failures to human review:

type QualityControlDecision = "pass" | "manual_review" | "reject";

interface QualityControlPolicy {
	minimumCoverageShare: number;
	maximumDistortedShare: number;
	maximumJerkyShare: number;
	minimumActiveManipulationShare: number;
}

interface QualityControlInputs {
	frameQuality: {
		coverage_share: number | null;
		distorted_share: number | null;
		jerky_share: number | null;
	};
	handActivity?: {
		coverage_share: number | null;
		active_manipulation_share: number | null;
	};
}

interface QualityControlResult {
	decision: QualityControlDecision;
	reasons: string[];
}

export function evaluateDatasetQualityControl(
	inputs: QualityControlInputs,
	policy: QualityControlPolicy,
): QualityControlResult {
	if (
		inputs.frameQuality.coverage_share === null ||
		inputs.frameQuality.coverage_share < policy.minimumCoverageShare
	) {
		return {
			decision: "manual_review",
			reasons: ["frame-quality measurement is incomplete"],
		};
	}

	if (inputs.frameQuality.distorted_share === null) {
		return {
			decision: "manual_review",
			reasons: ["frame quality has no measured-frame denominator"],
		};
	}

	if (inputs.frameQuality.distorted_share > policy.maximumDistortedShare) {
		return {
			decision: "reject",
			reasons: ["distorted-frame share exceeds the acceptance policy"],
		};
	}

	const reviewReasons: string[] = [];
	if (
		inputs.frameQuality.jerky_share !== null &&
		inputs.frameQuality.jerky_share > policy.maximumJerkyShare
	) {
		reviewReasons.push("jerky-frame share requires visual review");
	}

	if (inputs.handActivity !== undefined) {
		if (
			inputs.handActivity.coverage_share === null ||
			inputs.handActivity.coverage_share < policy.minimumCoverageShare
		) {
			reviewReasons.push("hand-activity measurement is incomplete");
		} else if (
			inputs.handActivity.active_manipulation_share === null ||
			inputs.handActivity.active_manipulation_share <
				policy.minimumActiveManipulationShare
		) {
			reviewReasons.push(
				"too little active manipulation for automatic acceptance",
			);
		}
	}

	return reviewReasons.length > 0
		? { decision: "manual_review", reasons: reviewReasons }
		: { decision: "pass", reasons: [] };
}

Omit handActivity whenever human hands are not expected in the measured view. Otherwise, include it and set the minimum from the collection protocol. Store the decision, reasons, response tokens (analysis, frame_quality, and vocabulary), classification_version, and the policy version together. This makes a later decision reproducible when either the measurements or the acceptance policy changes.

Choose thresholds from calibration data

Do not copy one threshold across capture setups without validation.

Label a representative sample as acceptable, review, or reject according to the collection contract.
Compare metric distributions for those groups.
Choose a narrow automatic-reject region and a wider manual-review region.
Record false accepts and false rejects during rollout, then revise the versioned policy.
Recalibrate when the camera, task family, or metric version changes.

Current limits

The current customer endpoints return dataset-level summaries. They can decide that a delivery needs review, but they do not identify the exact file or time range to remove. The current API also does not directly measure exact or near-duplicate video, task completion, unsafe behavior, or demonstration quality. Vocabulary concentration and active manipulation can expose symptoms of repetitive or idle data, but they are not direct duplicate or behavior-quality detectors. A corrupted or unreadable recording may reduce coverage_share, but the current customer response does not diagnose which recording failed or why. Treat incomplete coverage as a re-measurement or review outcome rather than as proof that the delivered data itself is bad.

Getting started

Concepts

Workflows

Recommended decision order

Apply metrics by capture setup

Integrate a conservative gate

Choose thresholds from calibration data

Current limits

​Recommended decision order

​Apply metrics by capture setup

​Integrate a conservative gate

​Choose thresholds from calibration data

​Current limits

Recommended decision order

Apply metrics by capture setup

Integrate a conservative gate

Choose thresholds from calibration data

Current limits