DxO PL 7 lost a number of sidecar files

DxO PhotoLab
1

I am running DxO PL 7.6.0 (55) on MacOS.
I was editing pictures taken over a series of shoots with one directory per shoot. At some point, I noticed in the browser side that some directories showed up as unrecognized files. I clicked on one that correspond to a directory in which I had just edited all the files (so there was one .dop file per .DNG raw), and it showed that all files where gone. When I check the finder, I see the .DNG files are still there, but not the .dop files. This is the first time I see such a behavior.
Does it mean that DxO PL trashed my work ?
Or can I recover this somewhere else ? I searched all my filesystem for missing .dop files and did not see them anywhere. This looks like quite a bummer.

2

In preferences, the DOP file generation is well checked ?

3

Yes, the automatic export (and import) of sidecars options in Settings->Advanced are checked. I still see a lot of remaining .dop files in all other directories. Also, if I rework one file without .dop file, the .dop gets created.

But, in a couple of directories (at least), all .dop filles seem to have been all wiped out.

I am regularly backing the .dop files, so I could retrieve some from the backups I made. But still it kinds of ruins my confidence in the workflow to see some amount of work gone.

Essentially, my question is twofold.
First, I would like to understand what could have happened and how to avoid that.
Second, I would like to recover the missing info, if at all possible.

4

Hello there.

Sorry to hear about your situation.

Where are these photos located on your Mac?
Do you backup/sync the Desktop and Documents folders in iCloud?

5

Let’s assume, for a moment, that the .dop files were deleted by something else, and it’s not too relevant as all edits, keywords etc. are stored in PhotoLab’s database.

Let’s disable automatic export of .dop files and restart PhotoLab before proceeding…

  • If you now select the folder of missing sidecars and tell PhotoLab to export them, all info should be back as normal.

There can be a few caveats though…

  • if edits etc. are compromised in the database, the .dop files will not contain what you want. In this case, you best restore the database from a backup, be it one you recently created with PhotoLab’s on-board tool, be it from a time machine backup.
  • if the DNG files are “unrecognised” as you write, you’d need to restore the DNG files too, and if the backup contains the .dop files too restore them too and have PhotoLab import the .dop files.
  • once you have done the above, have PhotoLab export the .dop files and set it to do that automatically again.

I’ve seen corrupted databases…but this was usually caused by renaming or moving files with the Finder, Lightroom or other apps. And while I tried to keep the database intact by avoiding anything that could compromise it, I’ve come to the point to regularly delete the database (and cache) files, which reduces the risk of DB corruption greatly, and, as Lightroom is my leading Photo app, doesn’t hurt me at all, even though I’ve set PhotoLab to NOT import and export .dop sidecars automatically. I get ultimate repeatability by exporting to DNG or, even better, 16 bit TIFF (beware the file size).

6

Thanks a lot for your answers.

First, my workflow is as follows:

  • when I download pictures from my camera (my camera produces DNG files), I make multiple copies (so no worry about the pictures), and I put one working copy in a sub-directory of Desktop;
  • directory Desktop is not backed up in the cloud (it is ways too big);
  • I activate the creation of .dop sidecars;
  • I sync the copies with the .dop files produced during edition with PL 7, and also make a (manual) copy of all the dop files after each editing session.

I do not use LR or any other photo software.
I sometimes manually run some scripts to read data in the .dop files, but these scripts only read .dop files and I was not using them when the problem happened.
So I do not see which application would have legitimately accessed the .dop files except PL 7. The OS could be a culprit, but then I would expect issues with not just PL 7 and I expect we would have heard such things elsewhere too…

Also, one thing that makes me suspect PL is that, when the problem happened, it froze for a little while and moved back to the root directory of my account. Then it did show some directories as unreadable files, and all the .dop files in these directories were gone… Could be a symptom of it being impacted by some other program action, but still this seems a bit remote.

I have tried the advice by platypus, but it did not allow me to bring anything back. Either the DB is corrupted, or the info was already wiped from it when I tried. Sigh. I am of course disappointed with the lost work, but also this makes me a bit more wary in the future.

7

Thank you for the details.
Others have reported stories like this when the original raw file is moved to another place outside of PL or processed by iCloud optimize storage space is activated.
This then happens as PL appears to remove/delete orphan dop-files. Although macOS presents a virtual file - PL did not previously parse those properly and the dop was deleted.

Otherwise I don’t see any reason for PL to delete them.

8

Thanks a lot.
This makes sense, but I do not see it happen here. Indeed, I have iCloud optimise storage activated, but the Desktop is not backed up, so I would not expect such a race condition to occur here… If it were inside the iCloud directory, then I would see this as a possible explanation.

9

Yes, it’s almost certain it was something like this that caused the problem Xavier describes (despite not being exactly aware of what happened to change the “state” of the RAW files).

  • I have never experienced PL deleting any of my sidecar/.dop files - unless it was due to a mistake made by me (such as moving files from outside PL’s “awareness”).

“Something” would have caused this behaviour … it would not be spontaneous.

1 Like
10

I see.

I’m quite sure that if you disable the space optimisation you will not experience these problems anymore.

11

Thanks a lot for your replies.
This happened only once. Clearly this is already, once too many in terms of frustration, but I do not see how to investigate more, so I will just hope this does not happen again.
I am turning off the storage optimisation thing, although the Desktop directory is not under iCloud control so that should not be a possible reason. Who knows…
And more importantly, I will make more frequent copies of the .dop files.

1 Like
12

Read that setting carefully, it has some kind of double negative built in.

Optimisation turned on means, that files are only moved to iCloud if the Mac drive doesn’t have “enough” space (whatever that means)…
Switching off optimisation could therefore mean that files can be moved to iCloud, even though there is enough space on your drive…

The text has changed over the last few versions of macOS and one is well advised to verify the settings after each update ( :nauseated_face:) - or to make do without iCloud altogether. If one reads the descriptions if iCloud+, one can also deduct that data in iCloud is encrypted, but not so while it is transferred between the Mac and iCloud.

For reasons of privacy, data protection and collateral issues (this tread), one either has to opt for No iCloud or for iCloud+. The included service has its compromises - if we assume that the vanishing sidecars are a consequence of iCloud settings indeed…but then, the .dop files should be locatable in iCloud. Well ,all in all, something or someone has to be the weakest link and we can never assume that it’s not staring at a screen right now :wink:

13

You’re right! :confused:

Store in iCloud now works like this under Sonoma:

Store files from your Desktop and Documents folders in iCloud Drive, store photos and videos in iCloud Photos, store messages and attachments in iCloud, and optimise storage by keeping only recently opened files on your Mac when space is needed.