DOwnloaded the trial to check out and have an initial question:
Why are the resulting DNG files almost 3x larger than the originals? When I process NEF files in Lightroom and export as DNGs, those DNG files are roughly the same size as the NEFs.
1 Like
Please note that the same is true for PureRaw.
Output files are âlinear DNGâ files, hence the size difference.
Copied from Adobe: https://helpx.adobe.com/photoshop/using/adobe-dng-converter.html
- Linear (demosaiced)
The image data is stored in an interpolated (âdemosaicedâ) format. This option is useful if a cameraâs particular mosaic pattern is not supported by a DNG reader. The default mosaic format maximizes the extent of data preserved. Mosaic image data can be converted to linear data, but the reverse is not possible.
PureRaw is a useful programme that has a lot of potential. I have also noted that DXO PureRaw produces DNG files three times the size of the original raw files.
I have understood the explanation given above that DXO Pure Raw Converts to linear DNG. However, users need to be given the option to have a smaller raw or Tiff file. IMHO this is a problem that needs to be fixed.
My CR3 files from my 90D are about 45 Mbt. If I convert these with Adobe DNG converter then the raw files with lossless compression are about 50 Mbt which is acceptable. DXO PureRaw produces raw files about 135 Mbt.
For example, DXO RawPrime could export to a tiff with lossless compression (LZW or zip). That would be better than a jpg. Or maybe the file can be exported to tiff and then back to the smaller non linear version of DNG.
Just my 2 cents worth.
Thank you for your hard work in developing this software.
Kind regards
Chris
1 Like
Somebody may correct me.
DNG files are TIFF based files. They both contain RGB values: 3 channels per pixel.
RAW files do contain R,G,B values: 1 channel per pixel.
My Nikon D750 has 6016x4016 pixels. For simplicity the sensor has the same elements.
I shoot 12 bit RAW. The RAW image will be 6016x4016x12=289923072bit or divided by 8 36240384byte. or 34.56MB.
The 16 bit DNG/TIFF takes 2 byte per channel. That means 6016x4016x2x3=144961536byte or 138.25MB. NB: 16 bits.
Lineair DNG means no corrections to the demosaiced image has been done. This doesnât have any influence on the amount of pixels.
From Adobe https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf
Under the tag compression I read that if compression has been used, it will be a JPG compression.
So what I would think that if the DNG file is not much bigger as the raw file, then either 8 bits has been used or a JPG compression has been used or both. If so, I wonder why I had to shoot in RAW??
George
1 Like
Please see recent reply to a post by someone regarding output tif file. Thanks
Thank you George for this explanation. Its clearer after seeing the math.
However it must be theoretically possible for DXO to output to a smaller tiff in a future release. It will be helpful if users have this option. Not everyone wants a massive DNG.
Here is what I have found.
1- Original CR3 File 46 Mbt
2- Converted by Adobe DNG Converter 50 Mbt (Lossless)
3- CR3 converted to DNG by DXO PureRaw 134 Mbt
4- 134 Mbt DNG From DXO opened as 8 bit rgb PSD 85 Mbt
5- Same PSD saved as 8 bit RGB TIFF (LZW Lossless) 39 Mbt
Users could be given the option of choosing which modules of DXO PureRaw to apply. If de-mosaicing is not required, then maybe it will be possible to get a smaller DNG or Tiff.
Kind regards
Chris
Keep in mind that the Adobe DNG converter is creating DNG files with no edits while PureRAW is adding corrections for Chromatic Aberration, Lens Distortion, Vignetting, Lens Sharpness and Noise Reduction. Perhaps an increase in size by 3 times is excessive, but I would expect the output to be much larger than from the DNG Converter.
Mark
Strange things with Adobe DNG converter.
NEF file D750 6016x4016 30.5MB
Adobe DNG converter .dng file 1024x684 27.6MB, 24 bit color depth. When opened in IrfanView it shows 1024x684B. When opened in Photolab it shows 6016x4016.
Chris, have a look at the properties of your file from the Adobe DNG converter.
George
It seems that the properties are referring to the embedded jpg.
To me the only explanation for the smaller file size is that it is stored in the original, not de-mosaiced, structure with the camera parameters in a general structured form.
When I select a embedded jpg in full size, the file size is about the original size.
George
@George, I just checked Adobe DNG Converter output at different settings. This is what I got with a few images of different dimensions and size:
Each of the four blocks of images go along with the following rule:
- Original file (white background vs blue background of DNG files)
- DNG output with DNGC set to default conversion settings
- DNG output with DNGC set to custom (version 1.4) setting, uncompressed
- DNG output with DNGC set to custom (version 1.4) setting, compressed
- DNG output with DNGC set to custom (version 1.4) setting, linear, uncompressed
- DNG output with DNGC set to custom (version 1.4) setting, linear, compressed
DNG output files can be smaller than the original file or considerably larger.
DPL 4 (current release) accepted+processed all DNG files without hesitation.
DPR 1 (current release) accepted+processed all except the linear DNG files (expected)
DPR Output size per file is the same, no matter what type and size the file image had,
the example image 20130220-5160 turns out to have a size of 99 MB, which is close to the 94 MB of the linear compressed file written by Adobe DNG Converter but still about three times the size of the original .cr2 file.
My lessons learned: If drive space matters, donât use DPR - or get more drive space.
What I see that when no linear RGB is involved that then just the CFA from the original RAW is copied and the parameters rearranged in a standardized order. Just a guess.
The difference in size between the original and the âŚ-std.dng is caused by the smaller embedded jpg. When you choice a full size embedded jpg the file sizes are nearly or exactly the same.
The addition ad means adobe.
George
Itâs not going to be me because your calculations are sensible.
We all end up with 3-channel per pixel output no matter what formats we go through in between. When starting with RAW formats there is only one channel per pixel. The magic of DeepPRIME is the conversion between single channel and 3 channel which, at the most basic level, will triple the output size because there is triple the information. This level of information is always used when RAW conversion is done, itâs just that we donât usually write it to a file if the conversion and subsequent processing is done in the same application.
There are nuances to do with how many bits the input data is and how many bits the output data could be said to contain as a result of the input and the conversion process. Thatâs above my head, though.
So if you donât like the size of your linear DNGs, just do what would happen if you did everything in either PhotoLab or Lightroom â delete it as soon as youâre finished. Of course that means you cannot keep your adjustments to return to them.
Although⌠there is a way to proxy the DNG in Lightroom. You could build Smart Previews which will act as a proxy for the DNG and if you need to re-export, you could recreate the DNG using PureRAW.
By example a sensor of 4000x6000 sensels. The camera uses a bit depth of 12. The theoretical size of the raw data will be (4000x6000x12)/8=36Mb.
When converted to an 8 bit RGB raster image that theoretical size will be (4000x6000x3x8)/8=72 Mb. and with a 16 bit color depth 144Mb.
All theoretical sizes. Camera does compress the raw too.
What I wonder about is the use of the DNG files.
When used non linear it has the same size as my NEF files when using the full embedded jpg.
When used linear it stores a linear tiff file. And linear means that no gamma correction has been used. Thatâs what I figured out. So why not using a normal TIFF file??
George
DNG is a container format. Within it there can be different content types, same as with TIFF. Different bit depths, different codecs (e.g. JPEG previews versus compressed but lossless original data), and possibly multiple entities (also previews etc).
From Wikipedia:
As a ânormal TIFF fileâ is quite similar to a ânormal DNG fileâ, the question at hand is what do we want put in it?
Personally I wouldnât see any value in outputting 8-bit data as at the point of demosaicking I have yet to make my exposure and colour adjustments which is why I shoot RAW in the first place â to have the bit depth available to do that to maximum effect.
DNG can be (and generally is) compressed too. If it was not, every frame from a given camera would be identical in file size. I just looked at one (busy) monthâs images* and they range in size from 21.8 MB (a too-dark night exposure with little texture) through to 42.9 MB (with most of the frame containing a fine netting material). The average size of the 774 images is 33.2 MB.
My sensor resolution is 6060 x 4013 at 14 bits, which is approximately 42.56 MB of uncompressed data, suggesting that largest file I found is actually suffering slightly at the hands of compression, but my average frame is compressed by 22%.
When converted to a full RGB frame of 6016 x 4000 at 14 bits, it comes to a theoretical 126.34 MB uncompressed. Put that in a 16 bit file (TIFF seems to only ever come in 8 or 16 bit varieties) and weâre at 144.38 MB.
I took that smallest DNG and converted it with PhotoLab using âDenoise and optical correction onlyâ (equivalent to PureRAW) and it came out at 77.2 MB. The largest became 115.9 MB. When I exported them to 16 bit TIFF they both came out at 144.4 MB. I note that PhotoLabâs choices for TIFF include â8 bitâ and â8 bit compressedâ and â16 bitâ so it stands to reason there is no compression on the 16 bit TIFF, as borne out by the sizes which very closely match my theoretical number.
If the 16 bit TIFF could be compressed, Iâm guessing weâd arrive at the similar numbers to the DNGs, though it would depend on the compression algorithm in use (Iâve seen at least two used in TIFF files).
So I get back to my original point. TIFF or DNG makes little difference. Theoretically storing only 14 bits could be a significant saving and it does seem like the TIFF 6.0 standard would support this (and by extension TIFF/EP and DNG) but I can find no reference to this actually being used and support would likely be almost non-existent in other software.
*Note that my DNG files are camera-native, so they are the original RAW files.
I think weâre on the same line.
I just wonder what is the practical use of DNG. Itâs meant as a universal raw file, replacing all the other different raw formats. It can be created with either the original RAW file or with a TIFF file. PL creates a DNG that includes a 8bit TIFF file. For me thatâs to less for further editing.
When creating a DNG from a NEF file with the Adobe DNG-converter, the filesize is nearly the same when a full jpg is added. When I choose the original NEF file to be embedded, the file size just doubles about.
I just wonder what is the profit of a DNG in general and of a DNG containing linear TIFF special.
George
Why DNG? When I tested DNG vs. TIFF, I found that
- DNG files were roughly half the size of the TIFF
- DNG files allow for temparature/tint WB, which TIFF doesnât offer
- Metadata can be written into both files, which can be a plus compared to sidecar files
DNG containing linear TIFF?
- A DNG file contains image info in raw (r,g,b,g) format or linear (rgb,rgb,rgb,rgb) format
- The term âlinearâ as in linear DNG as used by DxO means that (rgb,rgb,rgb,rgb) is used
Images can look different too:
Upper row: original RAW, linear TIFF and linear DNG, both made from the original RAW
Lower row: Same as above with DxO Smart Lighting
In that case I think you compare 8 bit and 16 bit or RAW DNG with linear DNG. PL uses 8 bit for its linear DNG.
Maybe when you use a RAW DNG. But I donât see why something can be done on linear DNG and not on TIFF. They contain both a RGB raster image. Exposure compensation is als done mostly on the RAW data.
What I learned a linear DNG image is a RGB raster image without gamma correction. Thatâs why linear. But whatever it means, it is a RGB rasterimage.
George
So, DNG files.
From the start, remember that your camera shoots RAW files that have a Bayer matrix in them (ignoring Fuji X-Trans and Foveon and suchâŚ).
So what your camera is shooting, is actually a monochrome file. The first pixel contains red data, the second contains green data, another contains green data, then blue, etc.
So your RAW file is mostly 12bits or 14bits per pixel, stored in a 16bit-per-pixel file, but monochrome.
Letâs take a 24mp sensor, around 6000x4000 pixels.
6000 x 4000 x 2bytes-per-pixel (16bit) = 48000000. So I expect a single uncompressed RAW file to be around 48mb (well, 48000000 / 1024 / 1024 =~ 45.7mb). Of course, there is a preview JPEG and metadata in there, so it differs a bit per photo.
Now, there can be lossy and lossless compressed RAW files of course, so this might be less depending on what compression your camera uses, if any. My Sony A7m2 in its uncompressed form creates RAW files of around 48mb, so it checks out.
Now, if I create a DNG from that RAW file and say it must be uncompressed, I expect the DNG file to be around 48mb as well. It contains the same data basically.
But DNG has its own lossless compression scheme (basically JPEG-lossless internally), so the DNG file will probably end up smaller.
On top of this, Adobe supports a lossy compression scheme, where you can even change the resolution and such. But letâs ignore that for now.
What DxO does is demosaic the RAW data. You now get 3 channels per pixel (each pixel now has an R, G and B value instead of only one of those three). And weâre still writing 16bit per channel, so we basically have three times the amount of data.
6000 x 4000 x 3 channels x 2bytes-per-channel (16 bit) = 144000000.
144000000 / 1024 / 1024 ~= 137mb per photo. If I export my 24mp RAW file to an uncompressed 16bit TIFF, I would expect it to be around 137mb.
So if I write a linear DNG file (from DxO PL / PureRaw for example) I would expect that DNG to be around 137mb (and some metadata / previews).
Once again, DNG has lossless compression which might get applied (or not), so it might be smaller. But it explains why the file size is much larger compared to the original RAW, it contains three times the data now.
If you âexport the DNGâ from Lightroom / ACR and the file is still small, that means you didnât make any changes to the RAW data. What Lightroom exports is your original RAW file, but with the sidecar file with the description of the edits in the metadata. No pixel data is really changed.
What use is a linear DNG then? Well, it allows different demosaicing algorithms to be used and fed into programs that otherwise could not do them (because itâs basically now a regular R, G and B value per pixel). In the case of DxO that means that DxO is the one demosaicing the RAW data, note Lightroom / ACR if you open the DNG in that later.
How does it compare to just writing a 16bit TIFF file? Well, internally, not that much. But the pixel data that is written is still in a somewhat more ârawâ state: It has no camera matrix or calibration applied, itâs still in linear space (no gamma correction) and there is no white balance applied (multiplication of the R and B channels mostly).
So itâs possible to have a file in which the numbers that your camera recorded are still untouched, but now the software âguessedâ the missing R, G and B values that your camera didnât write yet. The rest is still up to interpretation, so a RAW editor still can do a lot with the data.
And DxO PL (I donât know PureRaw) allows changing the data - or not - as you see fit. Denoising, sharpening, lens correction, altering the color response to other bodies, etcâŚ
So compared to Lightroom-DNG-export, itâs really something different. That just writes some metadata about where you left the exposure slider. DxO really changes the data depending on where you set the exposure slider.
That means that in Lightroom / ACR case, your âeditâ isnât really saved (exporting as DNG that is). You still need Lightroom / ACR to get it back, since thatâs the only software that can interpret the edits again like you did the first time.
A RAW converter would normally - sort of - do things like this (I probably have the order wrong somewhere):
- Demosaic (go from one light measurement per pixel to three numbers per pixel)
- Apply white-balance multipliers
- Apply camera profile / color matrix, convert to a âworking profileâ.
- Correct for your cameraâs black-level, white-level, etcâŚ
- Apply a âbase curveâ or some other curve to get the linear data into a more usable presentation
- Any more edits you told it to do
- Convert to the output profile (sRGB for instance)
- Save file into some format (JPEG for instance).
- A normal TIF file - usually - has steps 1 through 7 done to it (and maybe including 8).
- A linear DNG file has only step 1 done to it (or more in DxOâs case if you told it to).
- If you convert your RAW file to DNG through Adobe DNG Converter or export from Lightroom / ACR, you have none of the steps done to it.
And this explains - maybe better - the difference in normal meaning between Linear RAW in a DNG and an exported TIFF. Even if you make no edits and leave everything as default in your RAW converter, the TIF file still has a lot of stuff done to make it so, which you canât simply undo. A Linear DNG has way more room to change it into what you want, just because less stuff has happened to it yet.
Personally I donât keep the DxO DNG exports. I see them as intermediate files (I use DxO PL for little else), and after Iâm done with something, I throw them out. The tech keeps evolving quite fast anyway, so files from a few years back will probably benefit from being recreated.
I archive my original RAW file, and I save my final edit (from Darktable these days). Maybe if I donât want to âlooseâ the edit, I copy the edits over to the original RAW file, so I can keep them there. But since my workflow keeps evolving, and some edit steps are done in other software (Photoshop for example), I donât really see the use to saving all the data and all the steps in between. I keep the source (RAW), I keep the result (JPEG). If I need to somehow, Iâll do it again.
No. Itâs stored just as bytes. That 16 figure is an adressing figure in the pc memory. On disk itâs stored as a list of bytes. Check your camera and set it on 12 bit. Read how many picture you can take. And now do the same with 14 bit. Youâll see a difference that wouldnât be there if these values where stored in 16 bit integers.
The calculation should be 6000x4000x12=288000000 bit for 12 bit and 6000x4000x14=336000000 bit for 14 bit., resp 288000000/8/1028/1028=34.33MB and 336000000/8/10124/1024=40.05MB. That are theoretical sizes.
A DNG can contain a RAW file or a demosaiced file, TIF-format. If it is a RGB file, than it is demosaiced. The difference between ânormalâ demosaicing is that all non-linear calculations are not done.
Donât ask me the practical use of a DNG.
George
Thatâs certainly a good way to keep your archive tidy, but (thereâs always a butâŚ) you might not be able to recreate a jpeg that looks exactly like your first edit, because a) tools have changed and b) you might have forgotten what you did last timeâŚ
The good news is, that every version of an output file is unique - weâre not robots are we.