If, along with my JPEG, I were to export a linear DNG with corrections applied, and they save in the same folder at the same time as one export process creating multiple files, would the sidecars or database become moot?
at a later date, I could open the PhotoLab-created DNG in Photolab and pick up editing where I left off? just as if I opened the RAW file and DXO applied the database or sidecar-DOP corrections?
Currently I save a processed image as a JPEG and a TIFF, the RAW file is on an external drive elsewhere, and the processed images may get moved to subfolders without finding and moving the DOP with them.
would continuing edits from the DNG be truly equivalent to continuing edits from RAW plus sidecar/database?
in this approach, my practice would be to move the JPEG and the DNG, together, to a subfolder, later. This would be easy because the DNG and JPEG were created or modified at the same time and sit next to one another in my file system list. Whereas DOPs dont stay next to the JPEG file in the file system listing (MacOS).
And I would have RAW image data, in the DNG format, staying with my processed file with regard to disks and folders.
Exported files, whatever their format, have no connection to either original DOP files or the database. Editing them will create new DOP files for the exported files.
No. exported files are entities in their own right and have no record of the originating file or its editing.
As I just said, there is no connection or knowledge of the originating files or of the editing process that created it.
…which is a bad idea imo - unless you make sure that all files move with the respective .dop and .xmp sidecars. Separating an image file from its sidecar(s) will result in a reset of all edits and metadata upon indexing by a freshly installed PhotoLab.
As of today, PhotoLab has no feature that can e.g. export sidecars to all files in the full photo archive. If there were such a feature, one could update all sidecars in the old PL and import the whole lot in the new PL without risking any loss except for the items that are not written to sidecars (history, projects depending on Win/Mac)
To handle relations I’m utilasing IMatch. The RAW always is the master, all other copies in different formats are just relations. IMatch can handle relation files across more than one share. Keywords are assigned to raw and will be populared to related client files.
Camera JPEGs are used as the preview for all versions, unless I’ve processed the raw NEF and exported it from PL, then the DxO jpeg or tiff will take precedence and be used as preview for all images within the group of versions.
Thanks to all who replied. It became clear that I needed to
A. know where my DOP sidecars are being saved now, and
B. understand DNG files better.
with regard to A, with help on the forum in another thread, I now know the sidecars are saved to the same volume as the RAW files, and I never move them, so no problem there. I keep the volumes with the RAW files, and their DOPs, all organized in folders by date during the original file transfer from SD card to external hard drive.
regarding B,
this helped with DNG files.
were I to export DNG files, at the same time I export the processed JPEGs and TIFFs from DXO PhotoLab, it would provide a safety margin for me should I lose track of, or access to, the original RAW and DOP, on their separate external drive volume.
If you simply export a file from a RAW - as DNG, TIFF och JPG - they will never have a .dop sidecar.
But as soon as you edit one of those exported files from within PL, they will have a .dop sidecar, produced by PL, in the same folder they are placed in.
When you start editing an image, no matter what format, PL creates a DOP, which contains a list of all the changes that have been made. PL expects that DOP file to reside alongside the original image file. If not, PL falls back to the database, which may or may not be in sync with the DOP.
If you want to move an original image file, that has been edited, you must also move the accompanying DOP file to the same folder.
When you export an image, PL creates a file of the appropriate type and places it in the folder that you have chosen on the export dialog. That folder can be the same as where the original file resides. At this point, PL treats that exported file as a totally separate entity and will create a brand new DOP for that file, should you decided to edit it. There is no connection, in either direction, between the original file and its DOP and the exported file or its DOP. There is no continuity in editing between the original and the exported file.
Exporting a DNG file might allow you to work with certain RAW settings, but the DNG doesn’t contain the original RAW or even refer to it.
Rather than creating all these “safety” exports, all you need to do is to ensure that the original files and their DOPs are securely backed up. The exported files can be recreated at any time from those pairs.
I run a Mac and have Time Machine running at all times, which gives me a regular automatic backup of the system disk. I also regularly use SuperDuper to clone the external drive that contains all my images and their DOPs.
In fact, I only keep exported files whist I am preparing a project for printing, as I can easily re-create them at any time from the original file and DOP combination.
In addition, Keywords and other metadata can be written to fies or to sidecars.
In this case, the following applies:
Metadata is written to into JPEG, TIFF and DNG files.
Metadata is written to a .xmp sidecar file per RAW. Keep these next to each other.
Imo, it’s also important to consider the .xmp sidecars if one wants to find out whether to delete the database regularly, occasionally or not at all.
Lose the connection between the RAW file and its sidecars (both .dop and .xmp) and all you have to restore customising and metadata edits - is the database.
my original RAW files, and the DOPs of the RAW files I have processed inPL, are on external drive and backed up to another external drive with carbon copy cloner. The DOP stays with the RAW.
My processed exported JPEG and TIFF files have been on the internal drive, and backed up to a different external drive with Time Machine.
I am changing the latter to keep down the GB of photos on the internal drive.
I am getting all this understanding sorted out as I migrate to a new Mac so I will have a robust system of backups and ability to find original RAW and DOPs.
this forum has helped me alot. Many thanks to all.
My RAW plus sidecar remain together on the external drives containing RAW files from the camera. I am OK there.
The DOPs created from reworking a JPEG or TIFF in DXO, create a DOP, saved in the same folder on my internal drive, but over time, things get complicated there. Sometimes bunches of DOPs have apparently new modification dates, as if the database might have gone out and looked at these(?)leading to a new more recent modification date. ?
Anyway, the good news is that now I know where the DOPs from the RAWs are, and its where they belong, and they stay there.
Thanks for this. This is clearer than other things I have looked up.
I had imagined that a DNG could be a container with the original RAW data plus layers of edits/corrections.
Or linear corrections applied to the pixel by pixel information in the RAW and resaves as pixel by pixel information.
I am still not totally clear how pixel information in a DNG is similar to and different from RAW. but I am making progress in this understanding.
this from OXiDant in another thread was useful: “Well in fact a DNG is a container which holts a TiFF like file.
It’s holds a full colorspace equal to the workingspace you set to convert it to DNG.
It’s RGB based and demosaiced and if you activated denoising denoised. And the optical module is used.”
DNG is based on the TIFF/EPstandard format, and mandates significant use of metadata. The specification of the file format is open and not subject to any intellectual property restrictions or patents.[4]
And this is from Adobe:
Digital Negative (DNG) is a publicly available archival format for raw files…
…
DNG is an extension of the TIFF 6.0 format. For convenience, the TIFF 6.0 and related documents are available below:
Depending on which converter is used, the original RAW data can be embedded in the converted file, which produces some kind of multipage file which is bigger than the original RAW file. Whether PhotoLab will edit the RAW or the converted image, I cannot say though. I suppose that it’s the RAW and judging from file size, the embedded Tiff goes overboard upon export.
DNG files can be RAW or RGB or both.
DNG exported by Adobe converters are usually saved as RAW
DNG exported by Adobe converters can be made to contain both RAW and RGB images - and PhotoLab doesn’t seem to recognise them as RGBs.
DNG exported by PhotoLab reveal themselves as RGB fies in PhotoLab.
I export my processed JPEG and TIFF to a different location from where the RAW, and apparently the corresponding DOP, are originally located.
For my specific situation:
I have vastly more RAW files than I select to process, because most days I am shooting birds with sequential shutter, birds being in nearly constant motion, even when perched, to get that perfect reflection in the eye or tilt of the head or the wing position or moment of movement. In 2024 I think I kept about 3 TB of camera imported files RAW+JPEG. But in 2023 it was more like 5TB.
So over time, the RAW files are on multiple external hard drives that are not all connected to my computer.
Whereas all my processed photos can still be on one drive, which is either the internal drive (current) or an external drive (in the near future).
For files I do process, it would be tedious to manually find and transfer copies of the original RAW file(and its DOP) from the external drive to the other drive location that has the processed files.
Ideal for me would be if the Export box let me check not only a JPEG option and a TIFF option, but also to copy the RAW and DOP to the location where my processed JPEG and TIFF are, all in a single operation for me. Since the option is not available in DXO, I wondered if using the option to export a DNG simultaneously with the JEPG and TIFF would be somewhat equivalent, giving me a RAW or RAW-like DNG alongside the JPEG and TIFF, for convenience and perpetuity. So I dont have to go back and find the right external drive and hook it up and find the original RAW file.
Clearly, from all the feedback, the DNG is not necessarily as much like a RAW file as I thought. Sounds now like at DNG is not that much of an advantage over a TIFF, other than perhaps more color space and dynamic range (?).
I hope that clarifies why I went down this path of inquiry.
In regard to your shear amount of photos and quite a complex need of usage of them. I would sugest that you might be helped a lot by looking into a proper DAM solution, in case you haven’t done so yet.
Many of your challenges would be managed far better through a good asset management system.
PhotoSupreme is a powerful one with very good metadata, versioning and stacking etc and works very well with PL.
Many enjoys iMatch which looks a bit windows-like and then it’s the free digiKam.
