> ## Documentation Index > Fetch the complete documentation index at: https://docs.hebbianrobotics.com/llms.txt > Use this file to discover all available pages before exploring further. # Get duplicates in a dataset > Reports data that appears more than once. `exact` compares media bytes over the same span, so it needs no threshold and makes no judgment: a group is a fact about the collection. Other notions of duplicate will arrive as sibling blocks in this response rather than as new endpoints. Expect `redundant_percent` to be `0` on most datasets. A non-zero value is usually a delivery that included the same footage twice. For denominator and comparison guidance, see [Understand quality metrics](https://docs.hebbianrobotics.com/quality-metrics). ## OpenAPI ````yaml https://api.hebbianrobotics.com/public-openapi.json get /api/v1beta/quality/duplicates openapi: 3.1.0 info: description: >- Measure video characteristics, hand activity, and content diversity in robotics video datasets. Start with `GET /api/v1beta/access`, then list available datasets with `GET /api/v1beta/quality/datasets`. license: identifier: Apache-2.0 name: Apache-2.0 title: Robotics Data Quality API version: v1beta servers: - description: Production API url: https://api.hebbianrobotics.com security: - {} - bearer_token: [] tags: - description: >- Video-characteristics, hand-activity, and content-diversity measurements for robotics video data. name: quality - description: Scope and capabilities granted by a bearer token. name: access paths: /api/v1beta/quality/duplicates: get: tags: - quality summary: Get duplicates in a dataset description: >- Reports data that appears more than once. `exact` compares media bytes over the same span, so it needs no threshold and makes no judgment: a group is a fact about the collection. Other notions of duplicate will arrive as sibling blocks in this response rather than as new endpoints. Expect `redundant_percent` to be `0` on most datasets. A non-zero value is usually a delivery that included the same footage twice. For denominator and comparison guidance, see [Understand quality metrics](https://docs.hebbianrobotics.com/quality-metrics). operationId: getDuplicates parameters: - description: Dataset ID from `GET /api/v1beta/quality/datasets`. in: query name: dataset required: true schema: type: string - description: >- Opaque `media_digest` token from an earlier response. Omit to use the latest comparison. in: query name: media_digest required: false schema: type: string responses: '200': content: application/json: schema: $ref: '#/components/schemas/DuplicatesResponse' description: Duplicate footage and the clips compared headers: x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '400': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: The dataset ID is invalid headers: x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '401': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: A valid bearer token is required headers: WWW-Authenticate: description: Bearer authentication challenge schema: type: string x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '403': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: The bearer token cannot read quality measurements headers: x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '404': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: The dataset or duplicate comparison is unavailable headers: x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '500': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: The server could not read the comparison headers: x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '503': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: The server is temporarily unavailable or overloaded headers: Retry-After: description: Seconds to wait before retrying an overloaded request schema: type: string x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string '504': content: application/json: schema: $ref: '#/components/schemas/ApiErrorBody' description: The request exceeded the server processing deadline headers: x-request-id: description: >- Request correlation identifier supplied by the caller or generated by the API schema: type: string security: - bearer_token: [] components: schemas: DuplicatesResponse: description: >- Duplicate footage in a dataset, one block per notion of duplicate. The block is named for the notion, not the endpoint, because more are coming: `exact` compares bytes and needs no threshold, while a perceptual or re-encoded comparison would need one and would answer a different question. Reading `exact` therefore never becomes ambiguous when a sibling ships, and both would share the one denominator below. properties: compared_clip_count: description: >- Clips compared, which is the denominator every percentage here is taken over. Clips the comparison has not covered are excluded rather than assumed unique. format: int64 minimum: 0 type: integer dataset: description: Dataset ID used in the request. type: string exact: oneOf: - type: 'null' - $ref: '#/components/schemas/ExactDuplicatesDto' description: >- Byte-for-byte identical footage. `null` when nothing in this dataset has been compared that way yet, which is not the same as no duplicates. media_digest: description: >- Opaque measurement token. Matching tokens identify results from the same comparison. type: string required: - dataset - media_digest - compared_clip_count - exact type: object ApiErrorBody: description: Error payload returned for handled API failures. properties: code: description: Machine-readable error code, e.g. `dataset_not_found`. type: string message: description: Human-readable message. type: string required: - code - message type: object ExactDuplicatesDto: description: Footage that is byte-for-byte the same over the same span. properties: groups: description: >- The groups themselves, largest first. Each lists every clip that is the same footage, so a caller can choose which copy to keep rather than being told. items: $ref: '#/components/schemas/ExactDuplicateGroupDto' type: array redundant_clip_count: description: >- Clips that could be removed without losing any footage: every group's members except the one that survives it. Deliberately not the number of clips INVOLVED in a duplicate, which is larger whenever a group holds more than two copies and overstates what deleting them recovers. format: int64 minimum: 0 type: integer redundant_percent: description: >- [`Self::redundant_clip_count`] over the clips compared, rounded to two decimal places. Expect `0` on most datasets. format: double maximum: 100 minimum: 0 type: number required: - redundant_clip_count - redundant_percent - groups type: object ExactDuplicateGroupDto: description: One set of clips that are all the same footage. properties: clips: description: >- Every clip in the group, sorted, so the same dataset always reports the same order. items: type: string type: array required: - clips type: object securitySchemes: bearer_token: description: Bearer access token issued through OAuth 2.0 client credentials. scheme: bearer type: http ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.