Indoor RGB-D image datasets: depth units, registration and spatial evidence

Specify a spatial task rather than asking for depth in general

Depth can support object distance, indoor segmentation, surface understanding, reconstruction or camera localization. Each task requires a different relationship among color, geometry and labels. A depth map useful for rough foreground separation may be unsuitable for measuring a small object or evaluating metric reconstruction. Describe the spatial decision, working range, objects and environmental conditions before choosing a source.

This guide concerns still-image pairs and scene-associated views. A collection may originate from a sensor stream, but it should not be confused with a robot navigation trajectory or a wearer-synchronized multimodal sequence. If actions, inertial signals or continuous motion matter, include those requirements explicitly and refer to the appropriate specialist workflow. A single RGB-D frame does not preserve the history that preceded it or establish an action that caused the observed scene.

Read benchmark releases at the artifact level

The NYU Depth V2 documentation distinguishes labeled aligned image pairs, raw sensor data and preprocessed depth. ScanNet’s official repository describes scan-associated color/depth streams, poses, meshes and annotation files. The practical lesson is to compare the artifact your method will consume, not just the name of a dataset or the number of frames quoted in a summary.

A benchmark can be valuable for prototyping a loader or understanding a label convention while remaining a poor match for a particular commercial collection. Check the current release, access agreement, retained source fields and intended use directly with its publisher. Neither research access nor a software repository’s license should be treated as evidence that Fiund has the underlying source available for redistribution or commercial model training.

Make depth scale and invalid values machine-readable

A numeric depth value is incomplete without its encoding. Define the storage type, units or conversion factor, valid range, missing-value conventions and whether the stored quantity is optical-axis depth, range along a ray or another representation. A grayscale display can look reasonable even when its values are interpreted under the wrong scale. Keep a visualization for human review and the native values for the model.

Do not replace missing measurements with zero unless the convention explicitly says that zero means invalid. Keep a validity mask when the learning or evaluation pipeline needs one. Distinguish sensor saturation, reflective-surface failure, out-of-range measurements and unobserved areas when that information exists. A clean-looking filled map may hide the missingness pattern that matters most for deployment, particularly around glass, edges or thin structures.

Separate raw, filled, predicted and rendered geometry

The NYU source documentation makes the raw-versus-preprocessed distinction explicit. Carry that distinction into the manifest of a proposed collection. Raw depth retains sensor limitations. Filling can improve visual continuity but introduces derived values. Monocular prediction supplies an estimated surface representation, not a new direct measurement. Rendered depth can be exact within a virtual scene while still differing from the behavior of a physical sensor.

Ask for a processing lineage that identifies the input, operation, version and output. If a project uses a depth-completion model, document whether its training material could overlap the evaluation target. Do not compare a system using filled reference depth against one evaluated on raw sparse observations without explaining the difference. Source choice and preprocessing should be designed around the task rather than selected merely to produce the most attractive demo image.

Verify registration and camera geometry together

Color and depth sensors can have different resolutions, viewpoints, optics and timing. “Aligned” should name the target grid, the transform and the policy for occluded or unmapped pixels. Preserve the dimensions and orientation of the exact images delivered; a later crop or rotation changes the relationship. Request an edge-overlay review on representative structures, while retaining the calibration and transformation data needed to reproduce it.

OpenCV’s calibration documentation describes a camera projection in terms of intrinsics, extrinsics and distortion. In a delivery review, check the actual convention: camera-to-world versus world-to-camera, row versus column order, matrix storage and translation units. A file that parses as a 4×4 matrix is not sufficient evidence that it expresses the transformation the buyer assumes. Include a simple reprojection or known-distance check appropriate to the capture setup.

An illustrative calibration record and alignment diagram

The diagram shows the information that must travel with a registered pair: color orientation, depth values and validity, then the transforms connecting them. It is a concept illustration, not an observed sensor output. The associated JSON intentionally marks calibration values as illustrative and the camera-pose state as not provided. Unknown fields should remain unknown rather than being filled with plausible numbers.

Download rgbd-calibration-record-example.json can help a buyer and supplier agree where scale, invalid values, registration and calibration references belong. The example’s 0.001 meters-per-unit value is an explicitly chosen teaching convention, not a measurement of a particular camera or a statement about all RGB-D datasets. Replace it with evidence from the actual source. An automated check should reject records whose declared units, encoding or calibration reference are absent, incompatible or internally inconsistent.

Group rooms, scans and derived views for evaluation

Many frames from one room can contribute coverage while remaining highly related. A random image split may place adjacent or nearly identical views in both training and test sets. Preserve building, room, scan and capture-session relationships when known, and choose held-out groups according to the intended deployment question. State if a collection cannot support room-disjoint or building-disjoint evaluation.

The accepted unit also affects counting. Distinguish a room, a scan, a registered pair, a projected annotation and a mesh surface. If one 3D annotation is projected into many views, those projections are not independent annotation efforts. For object recognition, inspect how the same physical instance appears across viewpoints. For geometry tasks, inspect whether a reference reconstruction incorporates the test views in a way that changes what the evaluation measures.

Design an acceptance pilot around geometric failures

Include examples with edges, reflective objects, dark surfaces, windows, clutter, thin structures and the distances important to the application. Decide which conditions are required coverage and which are explicit exclusions. A pilot should include failure examples, not only complete maps from easy scenes. Record missing-depth fractions and registration issues by relevant region or condition rather than relying on a whole-image average.

A useful review separates file integrity, sensor interpretation, label correctness and permission evidence. Check that pairs resolve, depth values match the declared format, transforms are reproducible and the reference labels have the claimed provenance. If metric accuracy is required, define how it will be measured against a suitable reference. This page does not assign a universal millimeter tolerance or certify a device; those thresholds must follow the use case and the available measurement process.

Prepare a quote that reflects the actual artifacts

Compare prices using a defined accepted unit: a registered pair with specified labels, a complete scan, or a scene package with calibration and geometry. List source access, capture setup, sensor handling, registration, annotation, review and packaging separately. New rooms and difficult materials can alter collection effort, while reusing an existing archive creates a different source-review task. A price per image can conceal all of these distinctions.

No representative commercial average is established here. Start with a pilot that exposes the real calibration and review workload, then estimate the larger delivery with documented assumptions. A Fiund inquiry should identify the task, sensor expectations, distance range, scene coverage, required geometry, held-out groups and intended model use. Source availability and supporting documentation are established for the actual proposed collection, rather than inferred from the benchmark comparisons in this guide.

Questions buyers ask