Count beyond filenames
Different filenames do not necessarily represent different visual examples. A collection may contain repeated downloads, resized exports, bursts, or crops of the same original. Ask for both file counts and source-group counts. Define the kind of novelty your task requires before deciding which related images should be removed and which should be retained.
Separate exact from related matches
Byte-identical files, visually identical recompressions, and similar neighboring shots are different relationships. Use separate flags rather than forcing all relationships into one duplicate decision. A new angle can be useful for recognition even when it belongs to an existing product group. Preserve the evidence and review ambiguous clusters before deleting material.
Review clusters in context
A similarity score is a screening tool, not a complete editorial judgment. Inspect representative clusters and the tails of large groups. Repeated backgrounds may connect different objects, while altered crops can obscure a common original. Document which transformations were considered so future additions can be checked using a compatible interpretation of duplication. Keep a reversible exclusion list so a useful related view can be restored without rebuilding the original collection or losing its established source-group relationship.
Apply the policy to splits
Related photographs can affect evaluation even when retaining them is reasonable for training. Assign related source groups together when independence requires it. Keep the grouping table in the delivery manifest and preserve stable identifiers after filtering. The resulting count should explain retained relationships rather than promise that every remaining image is universally independent.
Review checklist
- Report files and source groups separately.
- Distinguish exact duplicates from related photographs.
- Inspect similarity clusters before removal.
- Apply the agreed grouping policy across splits.