PL5: Moving files outside PL loses corrections and more

RexBlock · March 24, 2022, 6:02pm

I have noticed errors in how corrections are handled in image files that are moved outside the confines of PL5. There appear to be several related problems:

1, not every correction to an image will cause all parameters to be flushed to its sidecar, even if automatic export is on. Presumably, they are written to the DB

2, if an image is moved, its path as known in the database is lost, so PL5 will import what it sees from the associated sidecar

3, PL5 does not recognize “Rating” as a sidecar parameter, even though it writes them, and even though it does recognize “Rank” (which is pre-PL5). This happens even with an explicit sidecar import.

4, due to #1, other corrections are lost. I don’t have a complete list but in a short test I did notice some missing.

This raises some questions aimed directly at product development:

1, are sidecars going to be fully supported going forward? Can a sidecar be expected to have ALL of the corrections made to an image?

2, are DxO expecting users to stop moving files outside PL and use ONLY the interface provided?

In other help posts, I see the occasional answer to “delete the DB and start over.” This is not going to be helpful if sidecars have an incomplete correction list.

Sorry it’s taken so long to narrow this down, but it finally bit me enough times to cause sufficient annoyance. Thanks for listening.

platypus · March 24, 2022, 6:20pm

Moving or renaming a file outside of DPL disconnects that file from the database and a few pieces of information are lost - and DPL does not yet offer any means to “reconnect” such items to the database.

As of now, DPL ingests new files automatically, whether we want it or not; moves files to the trash, whether we want it or not. A more tolerant behaviour can only be attained if DPL replaced automatic ingestion/trashing to user controlled ingestion, plus removal from the DB only, without moving assets to the trash. Such a fundamental change would be welcome in view of more robust asset management, but not necessarily welcome to all those who loathe DPL’s database. IMO, DxO will have to take this decision soon, or do as you (and others in this forum) propose: Make sidecar files more comprehensive, so that they can be used as distributed database backups. Remains to be seen if edit history will be included - not necessary imo, but others might disagree…

Read more about it in the following posts

RexBlock · March 24, 2022, 7:20pm

Thanks for the heads-up. Those are some nice discussions I missed.

Agree strongly about taming the database. I’ve cruised around in there using sqlite and a cursory glance shows a lot of cruft, such as multiple instances of same-named file, which can only result from PL5 seeing the same file in one directory, then another. (In my system, it is impossible to have same-named files).

The part about the sidecars really frustrates me though. I move older images off to external drives all the time to save space and to simply get out of the way. If and when they come back, they aren’t always in the same place, which means they’re “lost.”

So I can see how DxO opened a can of worms when it entered the DAM realm. I would appreciate either a more robust DB and file management interface, or just abandon it altogether – I can handle either one. As it is, we are halfway in both worlds and are trying to build an airplane while it is flying.

Maybe DAM should have been developed and marketed as a wholly separate plug-in, like Film Pack and Nik.

platypus · March 24, 2022, 11:07pm

DAM or no DAM, the database is used and as long as it is (always), it’s best practice to NOT move or rename files outside of DPL, at least not if DPL is the one system you work your files with.

uncoy · March 25, 2022, 9:17pm

DAM or no DAM, the database is used and as long as it is (always), it’s best practice to NOT move or rename files outside of DPL, at least not if DPL is the one system you work your files with.

Almost none of the serious PhotoLab users trust or like the PhotoLab DAM and its database. All we want are robust sidecars:

.dop for picture corrections
.xmp for ratings and metadata

I don’t know when DxO will finally figure out the photographers really interested in PhotoLab are the ones who want a robust system with portable files, where we can store away old sets and be confident that opening them on a new computer with no PhotoLab database we will get all of our picture and metadata changes back seamlessly.

arcane93 · March 26, 2022, 7:17am

This is an issue for me too. My working method for years now has been to edit my photos on my local drive, and then move them off to my NAS for archiving when I’m finished with them. But since my switch to using DxO PhotoLab last fall, I’ve been left with a dilemma… First, that decoupling the photos from the database causes the loss of some correction data, as previously mentioned here, and second, that bringing a photo back into PhotoLab from the archive (or simply moved from any other location) doesn’t show me any of the previous edits in the history, it only shows “Loaded sidecar”. It’s one of the things that I’m most unhappy about with PhotoLab.

Honestly, my preferred fix would be for them to decentralize the database into a catalog file, like Lightroom uses. Make it possible to have multiple catalogs, and to copy the catalog off with the image files to whatever storage location I want to keep it in. And, of course, to reconnect the image files to the catalog in the event that they get moved. Do away with the sidecars entirely.

Otherwise, I want more robust sidecar files, that contain complete information so that a database isn’t even required, and a complete edit history can be accessed.

I’m honestly good with either of those options, but what I don’t want is this thing that we have now, which traps me to a single database which I can’t move or back up easily, and which breaks if I move image files.

Sigi · March 26, 2022, 8:13am

Fully agree

Prem · March 26, 2022, 5:32pm

Fully agree.

gdu90 · March 30, 2022, 11:47am

RexBlock, could you (or someone from DxO) please identify which corrections are not being recorded into the .dop sidecars as you describe. One of the best features of OP/PL for me has always been the belief that all my edits were safely stored in small, efficient .dop files. Is this really not so ? I have not noticed this problem despite completely ignoring the PL5 DB and doing all my DAM directly ‘by hand’ in Windows.

I move my RAW and .dop files off onto NAS after editing and move them back if I want to re-edit. I then maintain my keywords in an SQLite db and embed them into the image files with a batch Exif editor; I can also generate .xmp sidecars for batches of selected images from the SQLite db. This works for me because PL5 is by no means the only editor I use to finish an image, so its DB could never be complete. (I am not a Lr user, BTW.)

With this setup I can easily continue without having a full multi-session PL5 edit history stored for any image, but it is a different matter if I am unwittingly losing some of the corrections I applied to an image that was edited in PL5 because the .dop doesn’t record them. What am I missing ?

platypus · March 30, 2022, 12:44pm

…as far as I see in sidecars written by DPL5 on my Mac, the .dop sidecars seem to contain all settings.
Settings are grouped in two sections called “Base” (settings at default values) and “Overrides” (settings with values changed by the user.

If you want to be sure, export a .dop sidecar and match whatever you find in it to the respective tool.
Here’s a comparison of a sidecar with DxO Standard vs No Correction preset entries:

As you can see, the changes caused by the DxO Standard preset are added in the “Overrides” section. Note that the name of the applied preset is the same in both cases, which means that DPL will be unable to tell which preset was applied after the initial (default = applied) preset.

Looks like most things are in the sidecars…but this does not necessarily mean that DPL will ingest everything on import.

RexBlock · March 30, 2022, 3:15pm

As follows … Conditions - PL5, MacOS 11.6.3 Big Sur

Directory 1
OS: Create new directory dir_1, copy three raws (in this case, Nikon .nef’s) into it. No sidecars.

PL5: Open this directory with the three images in it. Applied the following:
#1: apply standard DxO monochrome setting (right at the top), rating=1
#2: orientation - turn it on its side, rating=2 + “pick”
#3: crop tight, rating=3, “pick”
exit

OS: There are now six files: the three original raws plus the .dop sidecars. Create dir_2, copy all six files into that directory

Directory 2
PL5: Open the second new folder.
#1: image is monochrome, but no rating
#2: orientation is normal (instead of on its side), and no rating and no “pick”
#3: cropping is correct, but no rating and no “pick”
Import sidecars has no effect.

On image #3, apply one of my own presets that I’ve written. Then on all images, apply a relatively harmless change – bump contrast a couple of points, something to cause PL5 to rewrite the sidecars. Exit

OS: Examine sidecars – all three have “Rating = 0”

OS: Create dir_3, copy everything into it. Restart PL5 in this new folder.

Directory 3
PL5: Contrast changes were retained, but the preset I applied to image #3 didn’t take. Import side cars again had no effect. This one is particularly disturbing.

Directory 2 and 1
PL5: Go back through dir_2 and dir_1. Changes – such as ratings, orientation, my own preset – appear to have stuck.

Summary
Ratings lost
Orientation lost
Pick (and probably Reject too) lost
Contrast retained
DxO preset application retained
Personal preset application lost
Import sidecars has no effect

The failure on the sidecar import is the most egregious problem. If that is fixed, the other limitations can be tolerated. I don’t change orientation a lot (ever??), but it does leave one wondering what else is missing.

RexBlock · March 30, 2022, 3:19pm

Yes the info is in the sidecars, but it is not imported back, not even if you specifically request it. Then if you make any changes to the image in PL5, that old data is lost.

RexBlock · March 30, 2022, 3:27pm

To be clear, I don’t think parameters are missing from the sidecars. I have neither tested nor confirmed that.

The only issues I am noticing are when files are copied/moved outside PL5.

In that event, when PL5 opens the copy-to/move-to file, it sees it as a new object, makes a database entry for it, and records some information about it.

It also imports the parameters from the associated sidecar. But apparently it is not importing ALL of the parameters in the sidecar. And that is a problem.

platypus · March 30, 2022, 4:03pm

Yes, this has been covered in other threads too.

gdu90 · March 30, 2022, 8:18pm

Thanks for all this extra info. So it appears that the .dop does contain all edit info and what goes missing is stuff (like rating, orientation, etc) that might otherwise be in Exif.

The failure to re-apply an end-user preset is something I have never noticed myself and every single one of my images has at least one such end-user preset applied to the RAW file (per the camera body I’ve used) and often more than one (eg. if it is a focus-stack or other composite .DNG re-imported to PL5 for finishing). I’m pretty certain I’m not losing any of these settings after moving/copying files around the filesystem outside PL5 as I would have noticed this.

I use scripts to copy/move all pre-existing .dop files into my working folder before opening PL5, so I’ve also never tried ‘importing’ a .dop to an image. Thus every time I open PL5 there is a set of RAW/DNG/TIF files in the working directory and each one either has no .dop (because it is a brand new image) or has had its pre-existing .dop copied back alongside by the script. That seems to work just fine without losing any edits.

OK, I’ve no edit history across sessions and I’ve also no idea what is in my PL5 DB - probably a mess. But I think any change to PL5 that meant the .dop doesn’t include all accumulated edits and which made using the PL5 DB mandatory for editing, rather than it just being optional as a DAM, would fundamentally break DxO’s value proposition. IMO it’s the best RAW converter-editor; I doubt it will ever be much good as a DAM.

platypus · March 30, 2022, 9:43pm

All edits are in the .dop sidecars, the problem is, that DPL does NOT read all things back into the database, which means that a few things are later overwritten by the “absence” of such info in the database. This means that some info can get lost - forever. This has nothing to do with DAM functionality. It’s simply work that has not been done properly.

ianc · March 31, 2022, 7:36am

There’s no way I can keep all my images on the laptop I use for editing. One of the reasons I chose PL was because it didn’t have the issues associated with catalogues.
If I cannot edit an image, move it to my NAS and bring it (and sidecars) back later and be able to produce the same image (don’t care about history) then I’m completely stuffed.

platypus · March 31, 2022, 9:46am

All is well, if you move (copy, then delete) files in Library view in PhotoLab.

ianc · March 31, 2022, 10:12am

Good, thanks!
But I will lose ratings and “picked” status I understand which is a big shame as if, for example, I have three similar images and I picked a particular one to go with, I won’t know which one that is when I return to them.

platypus · March 31, 2022, 11:25am

…not if you drag your images using the Library view of PhotoLab:

Initial Status

Selected all images and dragged them to “Local Set 03” → Images come with rating and traffic lights.

Changed traffic lights (pick/reject) and dragged all images back to Local Set 04"

As we can see, if we drag images in PhotoLab’s Library view,
star ratings and pick/reject come along

Notes:

If Local Set 03 and -04 sit on the same volume
→ images will be moved (standard macOS behaviour)
If Local Set 03 and -04 sit on different volumes
→ images will be copied (standard macOS behaviour)