Compressed DNG brought to PL9, but with a bug

Resurrecting this old tidbit because it seemed incorrect but it took me a while to read enough about the DNG format (including parts of the DNG specification) to be able to offer a correction:

DNG is not a wrapper over a TIFF file. It’s a specialized type of TIFF file. It’s based on the TIFF/EP ISO standard, which itself was a mixture of extensions and restrictions to the TIFF format.

TIFF is a complex beast with a lot of history, but basically TIFF is a container format for image data (called IFD for Image File Directory) and metadata (called Tags, which are typically key-value pairs, where the value might be text or a number code). The metadata can provide additional information about the image (e.g. EXIF tags), or it can be technical information telling software what kind of binary image data is present in a given IFD. The kind of data you can put in a TIFF can vary wildly, some of it is specified in Baseline TIFF (which all supporting software is supposed to implement), some of it is specified in TIFF Extensions (which are optional to implement), and some can be proprietary extensions from specific vendors or software. Apparently a common joke was that TIFF meant Thousands of Incompatible File Formats, though it looks like specifying the Baseline TIFF subset helped make software a bit more compatible and interoperable.

DNG, as an evolution of TIFF/EP, basically specifies new types of binary data that can go in an IFD, restricts what data is possible (e.g. a CYMK image can be a valid TIFF but not a valid DNG), and similarly requires specific tags and specifies their possible values.

A TIFF file can contain several images (as separate IFDs or SubIFDs, see this post for an exploration of different options with graphs), and DNG typically uses that to store a preview image in addition to the main RAW (aka ColorFilterArray) data. DNG 1.4 also supports PhotometricMask data, which are single-channel images stored as separate IFDs, and I believe the iPhone uses that to store depth data (to let users do effects like fake bokeh in post in the Photos app). DNG 1.5 supports “Semantic Masks” as separate IFDs, again I believe it might be used by the iPhone to identify subjects in an image even when saving it in Apple’s ProRaw format (based on DNG). But software like PhotoLab could use that feature to save the result of their own AI subject selections as single-channel 8-bit semantic masks in the DNG files they export.

Another area I was interested in was the compression options available in DNG. In the TIFF format, there are dozens of possible compression algorithms available, though none of them are in Baseline TIFF, and the most used one (LZW compression) is an optional TIFF Extension. Which might be why some software export huge uncompressed TIFF files for maximum compatibility. On the other hand, DNG specifies a smaller set of compression formats but requires their support:

  • DNG 1.0 initially only allowed uncompressed data or LJPEG-1992 compression (aka lossless JPEG, rarely seen in the wild in a .jpg file because many image editors do not support it and only support lossy JPEG). It also allowed lossy JPEG compression but only for 8-bit data (typically for embedded preview images).
  • DNG 1.4 added support for Deflate (aka ZIP) compression and lossy JPEG compression, though when reading the specification as a non-expert I’m a bit unclear if any or both of those can be used for ColorFilterArray data or if they’re reserved for other usage (like PhotometricMask data).
  • DNG 1.7 added support for JPEG XL compression (either lossless or lossy), for most types of image data (including ColorFilterArray, LinearRaw, and image previews). In my tests it looks like most image editing software only supports it well for LinearRaw data, and using JPEG XL compression for image previews or for bayer/X-Trans ColorFilterArray data may cause issues (e.g. it crashes PhotoLab 9.6, haven’t tried in 9.7 yet).

So, to sum up: a Linear DNG is not a wrapper around a TIFF file, it is a specialized type of TIFF file with a more narrow specification than the vastness that TIFF encompasses, and new-ish features that are not always supported by image editing software.

In practical terms, there are differences between a linear DNG (a DNG file where the main IFD contains LinearRaw data) and a typical 16-bit TIFF:

Feature 16-bit TIFF Linear DNG
Pixel values Output-referred Scene-referred
Color space Usually a output-specific color space, such as sRGB, Adobe RGB, ProPhoto or DxO Wide Gamut Camera color space
Compression Uncompressed or LZW Usually lossless JPEG (LJPEG-1992), or more recently JPEG XL
Preview sub-image Technically possible, but not usual and probably not read by software Common (typically generated as a full-size or downscaled size lossy JPEG)

In terms of image quality, the output-referred vs scene-referred data and differences in color space are theoretically the biggest differences, though in my testing both had similar capabilities when it comes to dynamic range at least (color accuracy and clipping in very saturated scenes might be a different story).

Anyway, there are enough differences that it may be worthwile to use linear DNGs rather than 16-bit TIFFs for some use cases; and calling linear DNGs “just a TIFF in a wrapper” obfuscates that.

Though if you’re making edits to a photo while looking at the rendered output on a screen and shooting for a specific look, then what you are doing is maybe closer to what kind of data goes into a 16-bit TIFF, and maybe you should export to TIFF instead of a Linear DNG. With that in mind, I guess the restriction that DxO introduced to their DNG export in PhotoLab 9.6 makes theoretical sense, because TIFF might be the correct format for that use case. But if that’s true, maybe they should explain it, add a “16 bit compressed” option to their TIFF export, and also only make this change in a major release (PhotoLab 10.0) to avoid breaking their users’s workflows.

1 Like