Define the review object
State whether acceptance applies to files, labels, individual objects, or complete image records. An image can have correct labels for some objects and missing labels for others. Choose review units that reflect the task. Explain how an item with multiple annotations is counted so the reported result does not hide serious omissions behind numerous easy correct decisions.
Sample the difficult slices
A purely convenient sample can miss rare classes and complex scenes. Combine a broad representative review with targeted checks for small objects, crowded images, uncertain boundaries, and unusual formats. Keep the sampling method documented. Report what was inspected and what remains unreviewed instead of describing a sample result as proof that every annotation in the collection is correct.
Classify errors consistently
Separate wrong class, missed instance, geometry mismatch, duplicate annotation, and unsupported certainty. Give examples for reviewers and establish a way to adjudicate ambiguous cases. The correction process should explain whether similar errors trigger broader reinspection. This produces a more actionable conversation than a single quality adjective or an aggregate percentage without a defined denominator.
Keep revisions reconcilable
Request stable image and annotation identifiers, a corrected export, and a change log. Recheck the affected error categories after correction and confirm that changes did not break file references or split assignments. Agree on these procedures during the pilot. Availability, annotation scope, and acceptance terms should be established with the specific proposed supplier material rather than inferred from a site headline.
Review checklist
- Choose the acceptance unit and denominator.
- Mix broad sampling with targeted difficult slices.
- Define error categories and adjudication rules.
- Require traceable corrected exports and change logs.