Make PhotoLab Robust Against Cloud Optimization

This feature request is based on an issue reported here: PhotoLab lose edits when Optimize Storage is enabled on MacOS

Due to what was shown and tested in the posts of the thread mentioned above, PhotoLab’s reliability can be reduced, which can cause

  • wasted user resources
  • loss of user trust in PhotoLab
    and more

Feature Request

  • Establish PhotoLab tolerance against housekeeping activities of drive space sync’ed between a computer and cloud storage.

As of today, PhotoLab is intolerant to iCloud storage optimization. If image files are kept in iCloud and removed from the local computer, DPL zaps the removed file(s) and setting entries from the database. DPL should be able to flag such changes instead. The resolution of flagged events should then be treated as part of an overall improved database maintenance schema.

Please note that the issue was first documented by @myneur on a Mac. This does not mean, that the Windows computers are safe, they can connect to iCloud too. Maybe other cloud storages exhibit similar optimization routines and we’ve simply not stumbled over those quirks yet.

If the problem reported had occurred on a Windows system, the images would still have gone missing, the database would have been updated at some stage, I believe, and the database entries relating to the images would have been removed! Those entries do contain edit data and metadata, as does the DOP!

But, according to my tests, which I have just repeated, the DOP and Xmp sidecar files would not have been cleaned up by DxPL(Win) and the situation could then have been fixed “simply” by re-uniting the image files from cloud storage with the DOP and Xmp sidecar files still waiting patiently, but alone, on disk!

Instead, DxPL(Mac) is “blamed” for not being able to keep track of the image files as they are “spirited” away into cloud storage, something over which the product has no control whatsoever.

But DxPL(Mac) is guilty of being overly diligent by clearing away the “detritus” left after iCloud swept up and removed the image files.

I believe that the fix that is required is to make DxPL(Mac) as “sloppy” as DxPL(Win), certainly no less, @DxO_Support-Team, and leave the “orphaned” DOP and Xmp sidecar files intact for a potential reunion at some later time. Please do not under any circumstances use DxPL(Mac) as the role model in this particular case!!

@platypus While your proposal may have merit the issue that was detected could be “resolved” simply by aligning DxPL(Mac) with DxPL(Win) in relation to any “tidying” up it undertakes, i.e. leave the orphaned DOP and Xmp sidecar files untouched!!

This should be a relatively simple change @DxO_Support-Team that could be implemented sooner rather than later, waiting for the kind of change that you are proposing might never happen!

My original post:-


  1. Six RAW images (+ 1 VC) with DOPs & Sidecars discovered by PL6.1.1:-

  1. PL6.1.1 navigated away to another directory and RAW files deleted:-

The database still shows the “deleted” image entries and the 3 new ones because DxPL(Win) has not yet “discovered” that the image files are missing!

  1. PL6.1.1 navigated back to directory empty of images:-

The database entries no longer present in database but DOPs and Xmp files still present!

  1. Closing and restarting PL6.1.1 made no difference and the DOPs and Xmp sidecar files remained intact.

  2. Re-uniting the image files with the DOPs and Xmp sidecar files produced

I don’t really care how DxO acts on this, but I feel that they should at least be aware of the tarpit and list it as a known issue.

They’d have to adopt a (not so) new paradigm: Files reside in the cloud and can (but don’t have to) be mirrored to the local drive temporarily.

Until DPL can handle remote photo archives, it is a good idea to not use the Desktop and Documents folders unless iCloud sync is switched off.

DxO could e.g. present a popup (warning against the use of certain folders) when DPL is pointed to such a folder.

@platypus I do care how they act on aligning the two versions such that the fall out from this issue can be “minimised” particularly if it is as easy a fix as believe that it could be @DxO_Support-Team.

That is a good recommendation but the “archive” is not really an archive at all, it is actually the original images that have been moved without their associated elements! It is incomplete both in the cloud and in its original location.

How exactly is DxPL to know that the directory has a “special” status?

DxPL() could leave data relating to “missing” images intact in the database but I seem to remember posts with complaints about “missing” image reports from time to time in the past! Commands could certainly be added to your list of database management/maintenance commands to “clean-up” such data when the user is satisfied that image “recovery” is beyond hope or the deletion was intentional all along!

@platypus please start using the normal ways of notifying me when responding to a post that I have made or that you feel I might be interested in! It is a courtesy I have always tried to extend to you and apologise if I have left you off a list at any point, but my skills as a “clairvoyant” have diminished with age.

I write relatively few posts of late partly as a result of the fall out from a bug I caught from one of our Granddaughters and my stomach’s reaction to the antibiotic, so keeping track is relatively easy but …

@BHAYT , I provide a feature request and leave it to DxO to handle it according to their means.

I had a chat with @Joanna and she’ll come up with a few ideas that also relate to macOS specific interfaces and object properties.

Imo, the important thing is to have come across the issue and reporting it in the first place, credit to @myneur. The next steps should be a) in DxO’s interest and b) taken soon.

1 Like

As @platypus mentions, here are a few thoughts from my POV as a Mac developer.

To start with, we need to understand that iCloud was primarily designed for use with Apple’s concept of Documents, which are essentially a “folder” with various data held within, commonly referred to as a “bundle”. The file system sees that bundle as a single entity, so any changes to anything within that bundle trigger a notification that the bundle has changed, in much the same way as an ordinary file.

PhotoLab works with either two or three separate files - the image, a DOP sidecar and an XMP sidecar.

As @platypus has proven, if you remove an image file, using Finder, whilst PL is looking at the containing folder, PL detects that and deletes any associated sidecar files.

The problem that @myneur raised stems from trying to edit image files that were stored, apparently, on either the Desktop or Documents folders.

Except that, because he had configured, unknowingly, those folders as part of what Apple calls iCloud Optimisation. This means that macOS keeps an eye on how full the primary disk is and, if it is approaching full, files in those folders will be synchronised to the user’s iCloud storage and removed from those two folders.

Which files get transferred depends on things like their size and how frequently or recently they were touched (edited).

If you are editing an image in PL, the image file itself never gets touched because PL is a non-destructive editor, but the DOP and XMP files will get touched as adjustments are made. This means that the image file (which can be quite large) is far more likely to be left in your iCloud storage and deleted from the primary disk when optimisation is triggered. Whereas, the smaller sidecar files are both recently touched and a lot smaller, rendering them much less likely to be removed to save space.

If you look in the ~/Library/Mobile Documents folder on you computer, you will find the locally cached versions of the files you have placed on your iCloud Drive.

If you have chosen to include the Desktop and Documents option in Cloud Optimisation, this is where the system keeps a local copy of everything you see on your Desktop. But only if there is enough room on your disk. And it is these specially managed local copies of Desktop and Documents that are susceptible to appearing to lose files when the disk is too full - your original files are always kept on your iCloud Drive.

So, if you allow PhotoLab to access image files from your Cloud Optimised Desktop folder, you can never be guaranteed to be able to access it after opening it, because it may have been removed from the local cache. If this happens, it is the same as removing an image from a regular folder whilst PhotoLab can see that folder.

Now we all know that you should never manage files in regular folders whilst PL is open - all sorts of things can happen. Well, the same applies to iCloud folders but with the added “danger” of the system removing files to recover disk space as part of Cloud Optimisation.

So, should DxO do something about this?

In conversation with @platypus we feel that selecting a Cloud based folder:

  • should be possible but with a warning that your work is at risk
    … or…
  • should be prevented with the same kind of preference that hides package contents for things like the photos and PhotoBooth libraries.

It is for these reasons that it is highly recommended to never use your Desktop and Documents folders for storing images if you have allowed Cloud Optimisation.

Or, actually, never use your Desktop or Document folders for images, since macOS provides a Pictures folder that is never included in optimisation.

Oh, and never let you primary drive get more than half full. If you need more space, use a USB-C SSD, which will usually react as quickly as the internal drive.

1 Like

iCloud Drive folders are known as Ubiquity Containers and can be detected as such at runtime. But that is a non-trivial, asynchronous request that can take time to execute, thus PL may not react as fast as anticipated.

@platypus I have never sort to minimise the impact of the problem on/to @myneur but rather look for a solution that does not require so much work that it will actually deflect DxO from the myriad of other things that we want done!?

Hence, my suggestion that the problem is “fixed” by employing the strategy that DxPL(Win) appears to use (if I got my tests right), essentially leave things alone albeit the database entries will be removed if/when the user visits the directory but the sidecar files (DOP &/or Xmp) will remain intact and re-unification can then be accomplished later.

In addition, neither solution is mutually exclusive! Making DxPL(Mac) behave the same as DxPL(Win) can be undertaken sooner rather than later and a more “cloud friendly” scheme can then be added later using techniques as outlined by @Joanna!

@Joanna it appears that there are techniques available to limit the risk, now that they have been discovered, but as I stated above not allowing DxPL to add insult to injury would be a good starting point and other techniques can follow as and when!?

@platypus thank you for using my identifier, I did see the notification earlier but we had some sun (we haven’t seen much of that recently) so we went for a walk between the golf courses and I took my camera with me!

@BHAYT , harmonising the Mac and Win versions of DPL is one thing, this issue here is something else all the way. It can obliterate a user’s work and time spent.

@Joanna, how about preventing DPL from accessing image files in the mobile documents folder? Can this be enough to prevent damage?

@StevenL , can you ask someone for a statement regarding this topic? Even though the probability that someone will actually have this issue seems relatively small, the consequences are too bad to ignore imo.

It’s not primary to make PL robust against automated cloud routines.
It’s making it robust and non destructive against any file movement - being automated or manual by intent or mistake.

As I tested, PL simply deletes any orphaned meta files if the main image file is missing from its previous location.

@platypus I am not even vaguely interested in actually harmonising the two products but we currently have two ways that DxPL is reacting to a given situation and one is way worse than the other for the problem identified.

While DxPL(Mac) “tidies” up the sidecar files when it discovers that the actual images are no longer present DxPL(Win) does nothing with the sidecar files (I believe)!

That leaves a situation that can be “recovered” by the addition of the images from wherever they have been moved or the addition of the sidecar files to the images, because both elements are still available just not at the same location!!!

Having an example of a strategy that “works”, if by “works” we mean that the situation is recoverable rather than preventable, is always a better starting point than having nothing.

Hence, if we use the DxPL(Win) as the “role” model in this case then we will have a better outcome than doing nothing or waiting for a preventative strategy to be developed (if it can)!

I am well aware of the rather catastrophic outcome of the current DxPL(Mac) action in this case and have shown in my tests that it is reversible with DxPL(Win), so it seems an obvious first step to make DxPL(Mac) follow the DxPL(Win) example!!

I believe that there needs to be a preferences switch like we have for showing package contents, together with a warning that activating the availability of access to these folders involves a serious risk of data loss.

As I have mentioned, these folders can easily be detected using the Ubiquity Container APIs.

I just ran a couple of tests and found that PL “tidies up” sidecar files, even if you close it before removing an image file. As soon as you reopen PL on the same folder, any unrelated sidecars are removed. So this is not purely an iCloud problem, apart from not being in charge of when the image file gets deleted.

The only way I have found to avoid this behaviour is to delete the database after deleting an image file but before reopening PL.

@Joanna Remove the “excessive compulsive” behaviour (i.e. just be as “sloppy” as DxPL(Win)) and the worst element of the problem is avoided but that is not within the power of the user to control!

It needs DxO to “remove” the code from DxPL(Mac) that “terminates” the “extraneous” sidecar files!

From a users perspective you can never keep too many copies of the sidecar files, sadly the management of swapping them in and out would be left to the user (and the risk will be a Uuid clash at some point and “unexpected items in the bagging area”, sorry, “unwanted Virtual Copies” which also needs some code, inside or outside DxPL, to resolve and we are back to @platypus’s Database maintenance list and complaints/suggestions from you, me and others over the years!!)

Hints like that should come with a warning for the very few DxO PL users trying to organize their images in projects. Database gone = no longer projects visible.

Because a sharp axe can do serious harm, better take a chainsaw, no matter if you wearing flip-flops or leather boots. :roll_eyes: joke aside, an app destroying user’s work will get a rather reducing fan club over time.

1 Like

Indeed. To my mind, there should be a projects database, apart from the adjustments database, apart from the metadata database.

………moreover, deleting the database is no option because it obliterates 100% of the user’s work.

Imo, DxO should really do something about database maintenance in order to make DPL a more robust application than it is now. Removing things automatically seems to be a poor maintenance action in connection with iCloud and its storage management behaviour.

The currently best work around is to NOT put image folders into sync’ed Desktop and Documents folders…

But we still need a solution that effectively prevents productivity loss, and blocking access to sync’ed folders seems to be easy enough to implement really soon.

Note to @StevenL : Accessing image files in Apple might also be good for a few surprises.

@Joanna, @platypus @JoJu when I wrote about keeping multiple copies of DOPs above it is an idea I have had before (not necessarily a good one, just an idea), particularly if I ever continue to code in Python, e.g. a program to swap VC[1] with the [M]aster perhaps, when a backup program to secure the DOP, either as a standalone or as part of the DOP Swop script itself would be rather useful!

However, if someone was “stupid” enough to use that technique there could be a directory full of DOPs which if inadvertently accessed by the user via DxPL(Mac) would result in DxPL(Mac) neatly tidying up the “debris” or would it!?

There may have been “good” reasons behind the DxPL(Mac) coding and it might be DxPL(Win) which is considered to have a “bug” because it leaves things in an “untidy” state …!?