Real photographs versus synthetic images for computer vision: how to design a fair pilot

Separate three source types before comparing them

A photograph is an observation made through a camera pipeline. A rendered image is produced from a specified virtual scene and renderer. A generative image is produced by a model conditioned on prompts or other inputs; its internal scene geometry may not be explicitly available. These origins affect what can be controlled and which labels can be derived. The broad term synthetic can obscure the most important differences.

For example, a renderer may expose object masks directly from scene assets, while a generated picture of a person may require a separate, fallible pose annotation step. A photograph can have accurate human-reviewed labels or poor scraped captions. Ask how every image and every label were produced. Treat source origin, annotation method and review status as separate manifest fields rather than assuming one determines the other.

What published experiments establish—and what they do not

Tobin and colleagues’ domain-randomization research demonstrates a particular simulation-to-real approach for object localization and robotic transfer. Playing for Data studies game-derived imagery and semantic segmentation. These are useful examples of explicit experimental designs, not evidence that an arbitrary synthetic archive will improve every model or that its publisher permits every commercial use.

Use the papers to ask better questions: Which target task was measured? What real evaluation set was held out? How was the synthetic source varied? What training budget and real-data baseline were used? A result cannot be transplanted to a different camera, object class, model or source mix without a new evaluation. A vendor’s realistic-looking sample grid is evidence of appearance, not a measured reduction in deployment errors.

Map the coverage advantage to a specific failure

Synthetic controls can be useful when a team needs systematic variation in viewpoint, object placement, lighting or rare configurations. They can also encourage unrealistic correlations if the generator repeats a background, texture, camera model or compositional habit. Real photography can reveal unplanned clutter, wear, reflections and interactions, while still being biased by its capture brief and selection process. Neither approach automatically provides representative coverage.

Write the target failure as an observable condition before requesting a source. “Small product labels under glare” is more actionable than “more realism.” Then identify which variables can be varied independently and which must occur together naturally. A rare scene that looks plausible may still lack the physics, geometry or appearance cues required by the downstream task. Decide which combinations need actual capture and which can be explored with generated or rendered augmentation.

Ask what makes each label true

A renderer-defined object mask can be internally exact for the virtual scene and still fail to represent the target camera’s image boundaries. A generated caption can be fluent while naming an object that is absent or miscounting instances. A photographic annotation can also contain unsupported assumptions. Make label evidence visible: scene-graph export, direct image inspection, an independent measurement, a model prediction or a reviewer decision.

COCO’s separation of image tasks is a useful reminder that labels serve distinct objectives. Detection, keypoints, segmentation and captions should not be bundled under an unexplained accuracy claim. For each proposed source, request a label policy and difficult cases. Review negatives, occlusions and ambiguous examples with the same care as positive ones. Keep uncertainty and correction history so a later model failure can be traced to the actual supervision used.

Design evaluation before choosing the winning source

Hold out real material that reflects the intended deployment and preserve source groups such as sessions, locations, products or physical objects. Do not select the evaluation set after inspecting which condition performs best. Choose the primary metric, important failure slices and acceptance tradeoffs in advance. An average score may conceal a regression on a rare but costly condition.

Keep related crops, augmented variants, rendered views of the same asset and repeated photographs connected. If a generative model may already contain evaluation imagery or highly related source content, state the uncertainty rather than promising a contamination-free test. Synthetic images that imitate a target benchmark can make a comparison look better while weakening its independence. The evaluation should answer the deployment question, not reward the ability to resemble a public test set.

A mixed-data experiment plan with no invented results

Download real-synthetic-evaluation-worksheet.csv proposes three illustrative conditions: a small real baseline, a real-plus-synthetic expansion and a larger-real alternative. The counts are chosen planning placeholders, not available inventory or recommended universal sample sizes. Every result field is marked not run. The point is to make assumptions and missing evidence visible before a source purchase.

Run the same architecture and preprocessing under a declared budget rule. Fixed optimizer steps, fixed wall-clock time and fixed acquisition cost answer different questions; report which is held constant. When comparing data volumes, specify whether examples are repeated and how batches mix source types. Include repeated runs where needed to understand variation. Report absolute results, uncertainty and slice-level failures, not only a favorable percentage improvement over a weak or mismatched baseline.

Measure the cost of accepted utility

A cheap image is not necessarily a cheap usable training example. Compare scene or prompt design, generation or rendering effort, capture costs, label production, review, correction, storage and rights work. If a source requires heavy filtering, retain the accepted fraction and rejection reasons. If real collection needs access to a difficult location, make that constraint explicit rather than treating its cost as an unexplained image premium.

Avoid an invented market-wide price comparison. One project may value controlled masks, another rare physical interactions and another faithful text. Prepare a quote sheet with source-specific quantities and accepted output definitions. Model-training compute also belongs in the comparison when one condition requires substantially more examples or iterations. A pilot should reveal whether the proposed source contributes useful signal at an acceptable total cost, not just whether its invoice is smaller.

Keep provenance and permissions independent of realism

Rendered imagery can include licensed assets, textures or likenesses; generative outputs can be subject to model-provider terms; photographs can contain third-party material or participant interests. Visual realism does not resolve these questions. Preserve source tools, versions, asset relationships and the records supporting the proposed use. Avoid making a blanket legal conclusion from the word synthetic, public or owned.

The Playing for Data project page states a research/educational access limitation for its released material. That is a concrete example of why a useful technical dataset is not automatically a commercial supply source. In a Fiund inquiry, describe the intended training, fine-tuning or evaluation role and the documentation the buyer needs for the actual material. The source review should be grounded in that proposal rather than a general assurance that all images of one origin are safe.

Make the pilot decision reviewable

At the end of a pilot, retain the selected source versions, grouping rules, preprocessing, training conditions, evaluation records and known limitations. Explain whether an apparent gain came from source diversity, additional labels, a larger training budget or some combination. If the result is mixed, report which deployment slices improved and which worsened. A qualified “useful for these conditions” conclusion is more actionable than declaring one origin universally superior.

Fiund can help scope a photographic sourcing discussion and compare the evidence needed for a proposed collection. The next brief should identify the target failure, real evaluation conditions, candidate source types, annotation requirements, expected exclusions and cost units. The images and schemas on these specialist pages are labeled according to their actual role. An illustrative experiment is not a claim that Fiund has conducted it or has a particular synthetic or photographic archive ready to license.

Questions buyers ask