I am trying to decide if I should convert some of my older raw files to DNG using Adobe’s converter as a means of archiving them. My understanding “was” that DNG was a free to use format, based on tiff, that should enable the image to be opened by any raw converter that can process raw files.
For example before I upgraded to OS Catalina (64bit) I could process DNG files that had been created from Panasonic G9 raw files in Apple Aperture even though Aperture could not open the raw files directly.
So I am curious why DxO faqs state that Photolab will only open DNG files that are based on raw files that it can open directly. (My best guess is that it is something to do with lens corrections)
You may ask why am I interested? Well I have been digging around in raw files looking for jpeg thumbnails and I have noticed that while most are based tiff files some are not. I have some .CRW raws from a Canon G5 circa 2003 which do not look like tiff (despite what the documentation states) and this set me thinking about how long the writers of image editing software will continue to support old formats. For ref the .CRW format is dated 1997 and I believe that canon moved on to their .Cr2 format after the G6 camera.
Would I achieve better future proofing by converting these raws to 16bit tiffs and excepting the gigantic file sizes?
DNG never really took off in the way that it was envisaged and the file-size savings aren’t really important now that storage is so cheap, so I can’t see any good reason for adopting the format. Stick to CRW - it’s still widely supported AFAIK - and keep plenty of backups.
DNG is linear raw, so it’s not a different coding of a rawfile.
Basicly it’s a 16btiff with a floating wb point.
A dngconverter has no optical correction so if you would use dng for backup it’s better to use dxo export dng.
Then you can apply the nessacerry things and then store the DNG.
But the change you can’t open a rawfile in the feature is not likely.
There always be open source raw converters build for that problem.
And as @Mike1 wrote a tiff is more stable to store.
Just open and work the rawfile to a basic and general point and export that as "master-tiff adobe rgb 16bit.) this way you have optical corrected by optical module and noise corrected by PRIME, some shadow and highlight correction “digital negative”
Well I can’t open the .crw files with DxO now! However, I would not be able to open the DNG either if I’ve understood the faqs.
I think I will look into converting to Tiff and possibly jpeg along with some way of robustly storing the two files together.
Digital obsolescence is getting to be quite an old problem and one that the IT industry does not want to do much about. A simple upgrade of OS can and just did render applications useless leaving their data orphaned. I can probably still locate files created on a BBC micro (sprite was the image format) and Starwriter and WP5 word processor files from the late eighties which are now next to useless. At least with WP files we can now use Markdown and / or RTF.
It is a pity that we, the consumers of graphics applications, have not demanded that all graphics applications be capable of reading and writing in a format that is common to all. For example why is DxO incapable of re-editing dng files that it has created? After all the use of tags means that they can store any information they want inside the file. Are the DNGs it creates real dngs or some hybrid?
I suppose that I should be asking what are the negative aspects of converting to sixteen bit tiffs over and above the resulting files size and the time taken to create them?
If the camera isn’t in the supportlist of camera’s the dng’s are also not to open.
Can you open those crw’s with a adobe camera raw?
If so is there optical correction involved?
Try to find a old LR or such which reads those files and export as adobe rgb tiff 16bit.
One reason to convert dxo’s processed raw to tiff is that even when raw’s are readable the dopfiles can be changing setting when you reopen in a much newer dxo pl. Causing to redevelop it all over again.
A 16bit tiff file is pixel fixed so all local corrections and such are baked in.
Yes that’s a know complain. This would be a form of DOP-file inside the image file kind of thing.
The main thing you loose is the ability to re-edit the TIFF with the same flexibility as the raw file - in exporting to another format you’ll have processed the file - made choices about the white point, colour balance, sharpness, etc.
By the way, if you’re thinking of the long term, now would be a great time to get a proper DAM (if you don’t already have one) to make sure that the who-when-why-where-what information is written to the file as metadata, so anyone in the future can understand the photo.
Yes, I am a long term DAM user starting with iView media Pro. At the moment I am using NeoFinder on a mac and have written a small app that uses the DxO Lightroom module to load single or small groups of images into DxO without having DxO list the entire contents of a folder. I shall read your posts with interest.
This doesn’t mesh with my understanding of RAW and DNG.
First, “CRW files” are not a single entity, are they? Certainly in Pentax land, a “PEF file” merely says it is RAW and came from some kind of Pentax camera. The actual format within differs per model, often significantly. This is why lots of camera models are listed in compatibility charts. I recall on one of my older cameras I had to convert to DNG to get my software at the time to open them until about 6 months after I bought the camera when native support was provided.
Second, I wasn’t aware DNG was intended for space saving? My older Pentax camera could write DNG directly to the card but I found it a little slow and the files were larger than the native PEF files. With my current body, that is no longer the case. The file sizes are comparable and the camera can write either at a decent speed.
My understanding of the intent of DNG was that it was specifically to solve compatibility issues. If you want a free converter, you can download and use the Adobe DNG converter. Of any provider, I would trust Adobe most, since they invented the format.
Space saving was often cited as one of the benefits - it offers lossless compression, and when storage was expensive that was a plus. Compatibility and use a long-term archiving format were others, though it’s not been widely adopted as was originally hoped. I’m not convinced that any of reasons are compelling, so would suggest that anyone should make themselves fully aware of both sides of the argument before adopting it.
It’s biggest use now may well be as the native RAW format for Android phones.
My concern over DNG is the fact that it seems that much like Tiffs and RAW files it is a container format meaning that not all DNG files are compatible with all DNG editors. DxO is a case in point as it creates DNGs that are intended for further processing in other apps and that it can not open. This seems perverse to me as it is capable of opening RAW files that it has edited. I believe it does this through the use of its sidecar file so why not do this with DNGs. I also extend the question to Tiffs as there is no reason other than file size why a Tiff should not contain both an edited image and the source RAW file along with JPEG previews and the processing instructions from multiple image editing applications.
Getting back to my original question I am still mystified why DxO PL is unable to open a DNG based on a RAW file when it cannot open the RAW original. Other image editing apps can (e.g. Apple Photos, Aperture) so why not DXO. After all isn’t the whole point of DNG to make the images available in applications that can’t open the raw original?
I confused CRW and CR2. But despite that, my point was that all CR2 files are not equal. I can’t find a canonical (pardon the pun) reference to this fact, but have heard it anecdotally and I know for a fact this is true with Pentax .PEF files because I’ve had software in the past that could not open all .PEF files until additional camera support was included.
I also assumed (though thinking in more modern terms, so likely .CR2) that any proprietary RAW format would also include lossless compression.
As for support… that I know of… macOS, iOS, Apple Photos (macOS and iOS), Aperture (discontinued), Lightroom, Photoshop, Pixelmator Photo (iOS), Luminar, Affinity Photo, and DxO PhotoLab (with some limitations) is a pretty decent list for starters. I only have DNG files (excepting some really old photos in JPEG) and it has been a long time since I’ve been unable to open one.
This is a concern for me, too. As I think I mentioned previously, for a time I would convert my .PEF files to .DNG files specifically so I could open them in Aperture until such time as my camera model was added to the supported camera list. That is one of the key points of DNG…
Canon’s new CR3 file format, on their mirrorless camera range, does offer a C-RAW compression option; it is lossy, but apparently not noticeably so except when shot under particularly poor lighting conditions. Some their cameras that shoot CR2 offer S-RAW or M-RAW compression options, both with a lower pixel count. Maybe someone does offer truly lossless compressed RAW, but I know much less about non-Canon digital cameras.
I get the impression from internet reading that in the past converting a Canon Raw to the DNG format did save some disk space but that Nikon and others already employed lossless compression in their Raw files meaning there was no space saving advantage to converting to Raw.
One “possible” advantage to doing the conversion is that meta data can be safely written to the DNG and updated as necessary, whereas only a few applications are brave enough to write IPTC data back into camera raw files. The advantage depends mostly on how you manage raw files and their sidecar files. I am engaged in a major overhaul of my photo library structure and have discovered a few issues with xmp,
These issues include discovering many xmp orphans that have been left when the parent image has been deleted. These are very small text files so take little space so they just clutter the display in the Finder. I will be writing a utility app to locate and remove these files. A slightly more tricky issue is discovering some raw files that have not been renamed to the format that I have standardised to which incorporates the capture date in sql format as a prefix. The app “A better finder rename” can rename the image files but I have had to write another utility to update the names of the xmp sidecars.
The bumps in the xmp road that I mention above caused me to look at writing iptc data such as keywords directly into the original raw files. There is an urban myth that raw files are never modified and indeed most applications do just read them, mostly because they are complex structures that involve the maintenance of a number of tables of data containing “tag fields” that point to specific data. Adding or inserting meta or other data involves updating these pointers which means there is a chance of errors which would render the raw file unreadable. So most software just writes a simple xml based sidecar. However, PhotoMechanic and ExifTool will update the camera raw file file with iptc data if you ask nicely (all except .raf files).
I experimented with some copies of Olympus raw files adding keywords using ExifTool. No problems occurred and the keywords were visible in “some” applications e.g. Apple’s Preview. DxO PhotoLab would not display them in the UI but it did add them to JPEG derivatives.
So to recap, converting a raw to DNG could avoid messing around with sidecar files.
I found this page which suggests Canon bodies do in fact use lossless compression. (No mention of Pentax, which do use it and have for a long time.)
The intent to “not modify RAW files” is, I think, misplaced. There has long been a rule of thumb when using programs like Photoshop to never touch the original pixel data. The same should hold true with RAW files. The original pixels should never be modified, but adding meta-data is incredibly useful for many reasons including those mentioned by @skids.
@platypus the difference you see in file size may be down to the efficiency of the compression algorithm. I know when I was first experimenting with DNG they were slightly larger than the same image in PEF and that, along with speed advantages, led me to stay with PEF in camera for a while. But both formats were compressed.
One strong indicator that compression is in use is variation in file sizes. If you think about it, without compression, every RAW image (assuming consistent camera settings on quality/dimensions) should have the same file size if compression is not used. Mine vary considerably and I reckon I could pick out shots with a lot of blue sky simply by noting the smaller file size, because large areas of homogenous colour compress very well. Conversely, a picture of a crowd or foliage will have a lot of detail and therefore compress less.
I am tending to agree with you. As I type my computer is renaming files in the folder that stores images taken in 2017. There are 7000 odd images and each one has an xmp file, so the folder contains 14000 files.
Some of the images did not follow my naming convention of ReverseDate_Time_CameraName.Ext e.g.“2014_08_20_202607_P1234560.rw2”. Renaming the images is simple using a tool like “A better Finder Rename” but the .xmp and .dop files take more effort as they are not read by the renaming tools. Most of the sidecars just contain iptc meta data such as keywords and it would be much more convenient to have this data stored inside the raw files themselves.
The often quoted issue of writing iptc data into raw files seems to be the fact that the raw files formats are undocumented along with the often quoted myth that raw files are never modified . However, it seems to me that most raw files are based on the tiff format version 6. This describes how to store a multitude of data including multiple images within a single file. These parts of the file can be thought of as blocks of unrelated data. Both Raw and Tiffs file both use fixed format tables of data that list data tags that point to blocks of data. I suspect that the “undocumented” portion of a raw file refers to the sensor data and the “makers notes”.
In other words it is not to difficult to walk through a tiff or a raw and identify blocks of bytes that are for example a jpeg or the raw data and so on. Adding iptc data is just a case of adding the data in either an already defined block or creating a new block followed by adding a tag pointer to the data and then updating the pointers that point to all the other data blocks that follow the insertion point.
While writing to the raw file is more complex than creating an xmp sidecar file it is no more complex, as far as I can see, than writing iptc data to a tiff file. If true it does beg the question as to why side car files were created in the first place as there is no reason why DxO, Adobe or anyone else could not add their own processing data into the raw files using “private tags”.
I am beginning to suspect that sidecar files were created for the convenience of programmers rather than photographers or computer users.
Biggest problem of exifdata/metadata in rawfiles is that there isn’t a “standard” and it’s not a open data. different developerapplication/rawdiggers can read different amount of data out of the exif.
Why? i think its the manufactorers choice not to reveal all data inside the rawfile.
writing into the rawfile’s exif section can make it even more icky and fail triggerd.
a XMP-file as sidecar can solve a lot of this risk.
I do be with you as in it should be a standard and every reader should read the same info.
DxO chose for DOP-files where they place the image specific DATABASE info as a kind of replaceable “image developing data” so you didn’t needed the DATABASE at all. ( if the cach did grow to big and slowing the app down you could deleted it and reindex done.)
Now with keywords writing only in database this feature got broken.
So they have to figure out if they want to use the DOP-files to store this exif/metadata info( all what’s not in the rawfile’s exif.) or use the XMP-file system for that and basicly write in a same-named sidecar all exif data to company the rawfiles thus get 3 files per “image” in your original DAM.
Older camera’s could give Tiff’s and Jpegs not RAW-files, i supose you talked about those:
Yes, it’s easier to create a sidecar file then make image rawfile coding fully open/disclosed for every programmer who want’s to do something with the files.
Me i started to use XNviewmp (freeware) to create something of a DAM on the frontside of DxOPL, because the search function is quite handy one’s you use it. (with XMP and set in side the company files to move and delete all: raw dop and xml.
We have to wait a bit to let the passthrough of exif/metadata in DxOpl fully controllable (edit inside which effects the front and the rear) and keeps the dxo database just as it was: a cach driven copy which can be easily rebuild. (this wil support the movability of removable SSD’s and you can log in at any pc if you bring your workspace profile with presets and such with you.)
(jpegs and iptc) and raw and source tiff xmp.
professional users will use real DAM applications and would be frustrated using those “half baked simplified DAM systems” but they have far more images to wander through and also at command on the clock.
creating linear DNG’s with extra editted exif/metadata inside would cleanup the list of files but with what cost? (thinking of not be able to read dxo DNG’s back in DXOPL is one of them so you can’t preprocess the rawfiles to a filesystem and go back later to process the image.)