Restoring deleted images

BHAYT · November 18, 2024, 8:49am

The VC issue is a Windows “feature”, it appears!?

What about the mess you were referring to?

From the results of my tests with Win 10 on my 5900X I don’t feel that PL8 (or PL7) does anything particularly peculiar in the situation we are reviewing.

It may not behave exactly how I might like it to, nor how others might want it to, but I think I understand what it is doing and can guess the reason why!

As I described in an earlier post and again below the deletion process follows a pattern that fits with the way the product typically works i.e. whenever a VC is deleted, the DOP will be adjusted to take account of that deletion.

In the case of the deletion of the whole image with a [M]aster and a number of VCs, DxPL seems to be treating that on a copy by copy basis, so as each copy is deleted an updated DOP is written, rather than as a single atomic operation. The outcome of a Restoration is then not what is expected and not what is wanted because the final DOP write only contains the [M]aster’s edits.

@Wolfgang and @SAFC01 I was disappointed that neither of you saw fit to reference my earlier post at Restoring deleted images - #34 by BHAYT where I made the suggestion of the method @Wolfgang tested

and I also tested it!?

As @SAFC01 states, it relies on user initiated exports and users may suffer from many more problems created by omitting to export than from the deletion problem being investigated in this topic.

So my (excessively detailed) summary of what I have learnt with DxPL(Win) goes something like this with respect to PL8 on Win 10

Starting situation for all tests:-

2024-11-15_235713_

and the image contains no ‘Rating’ or ‘Colour Label’ they are “added” during PL8 editing.

Please note that my tests were initially run with these metadata settings, which then affects the results of some of the tests somewhat!

The significant one is the ‘Synchronize metadata with XMP sidecar file’ which does more than that title suggests.

With ‘Synchronise’ selected, the metadata, which I used (Rating and Colour Label) to make the copies “stand out” to demonstrate the test results, will be taken from and written to the image (image or sidecar file) but when this option to synchronise is not selected, the DOP becomes the sole source for that metadata in any new discovery situation, e.g. after the ‘Restore’ from the Recycle Bin.

With Auto Load and Save selected

When deleting a [M]aster image then all the associated VCs will also be deleted, as will the image, the DOP and the xmp sidecar file, if present.
Providing the ‘Remove’ command in the menu is used then the image file, the DOP and the xmp sidecar file (if present|) will be captured in the Recycle Bin and can be restored using the Windows ‘Restore’ command.
But if the user uses the Shift Delete keys then the deleted items will be deleted completely and any form of restoration will be impossible.
However, the DOP that is located in the ‘Recycle Bin’ will contain only the [M]aster edits. DxPL appears to delete from the last VC back to the first from the database and each such delete results in a new DOP to overwrite the previous DOP, each such DOP will have 1 less VC in it. By the time all the VCs are deleted that final DOP only contains the [M]asters details!

Result after Restore from the Recycle Bin:-

Result after Restore from the Recycle Bin with the Synchronise metadata not selected has the same outcome:-

With Auto Save and Auto Load not selected

DOPs can only be loaded and saved on demand.
When deleting the [M]aster none of the intermediate delete activities will be captured in the DOP because auto save (DOP exports) is not enabled and the image and the original DOP (and sidecar) file will be deleted to the Recycle Bin intact.
The DOP in the Recycle Bin will contain all the data that was exported to the DOP the last time the ‘Sidecars’/‘Export’ command was executed which will have updated the DOP (and is the only way the DOP will ever be updated in this scenario)
The Restore process is the same as before except that the DOP should now contain more data, i.e. the [M]aster edits as before plus the edits for all the VCs. However because the auto load is disabled DxPL(Win) does not take any details from the DOP at all when it discovers the ‘Restored’ image after its restoration from the Recycle Bin and therefore creates a new database entry with a new UUID, essentially for the image only, applying the new image presets at the same time.
In order to retrieve all the edits (for the [M]aster and all the VCs) from the DOP the ‘Sidecars’/‘Import’ command must be used and that will result in a clash between the new UUID allocated to the new [M]aster created at step 4 and the old UUID contained in the now imported DOP. The result is a new VC for the [M]aster entry!?

Result after Restore from the Recycle Bin and executing ‘Sidecars’/‘Load’:-

BUT the data retrieved will only be up-to-date if the user exported the DOP after making the edits and before deleting the image!

With metadata synchronisation off and using an image with no previous metadata we have the following after the ‘Restore’ and a manual DOP import

With Auto Save not selected and Auto Load selected

DOPs will be loaded automatically but created/updated only on a user request.
When deleting the [M]aster none of the intermediate delete activities will be captured in the DOP because auto save is not enabled but the image and the DOP (and sidecar) file will be deleted to the Recycle Bin.
The DOP in the Recycle Bin will contain all the data that was exported to the DOP the last time the ‘Sidecars’'Export’ command was executed which will have updated the DOP (and is the only way the DOP will ever be updated in this scenario)
The Recover process is the same as before except that the DOP should contain more data, i.e. the [M]aster edits as before plus the edits for all the VCs and the auto load will recover this automatically, using the UUID from the Restored DOP and avoiding the [M]aster getting an unwanted VC.
With 'Metadata synchronisation off no metadata will have been added to the image during any processing, i.e. before the deletion or after the restoration and (re-)discovery by DxPL but all the metadata added to the image in DxPL will be present in the DOP and will be recovered from the DOP in the auto load after the ‘Restore’.

Result:-

BUT the data retrieved will only be up-to-date if the user exported the DOP after making the edits and before deleting the image!

Careless use (or possibly no use at all) of the ‘Sidecars’/"Export’ command could mean that the above is the result of the restoration but is actually out of date because other image edit or metadata changes have been made in the meantime (since the last ‘Export’) but never made it to the DOP, as described earlier.

For example a change to VC[1]

But with metadata synchronisation off and no explicit export made we still have

after a ‘Restore’.

platypus · November 18, 2024, 10:15am

This was due to messy testing with multiple copies of files in the trash.
The results of re-testing can be found here.

platypus · November 18, 2024, 11:03am

Normally, foolish things happen when photo archives are cleaned up outside of PhotoLab and while it is closed. So, I tested the following:

Test 1 (Restore while DPL is not running)

Opened DPL (which is set to auto I/E .dop files)
Select a folder of images (test folder with 60 images + 60 VCs)
Closed DPL
Checked that the DB contained entries for 60 images + 60 VCs
Moved all images and .dop files to a separate folder
Opened DPL (and the test folder was empty) ___
Closed DPL
Checked the DB and found that the entries for the removed files were gone
Moved all images and .dop files back to their original folder
Opened DPL (and the test folder displayed the originals and the VCs)
Closed DPL
Checked that the DB contained entries for 60 images + 60 VCs

Test 2 (Restore while DPL is running)

Opened DPL (which is set to auto I/E .dop files)
Select a folder of images (test folder with 60 images + 60 VCs)
Closed DPL
Checked that the DB contained entries for 60 images + 60 VCs
Moved all images and .dop files to a separate folder
Opened DPL (and the test folder was empty) ___
Moved all images and .dop files back to their original folder
(and the test folder displayed the originals and the VCs after a few seconds
Closed DPL
Checked that the DB contained entries for 60 images + 60 VCs

Both tests provided the expected restore and nothing else.
Note that steps before the “___” are the same in both tests.

Now, enuff of that stuff.

BHAYT · November 18, 2024, 11:42am

@platypus

I had seen that post and it seemed at odds with the “mess” one, hence my comment.

I also ran into a messy Recycle Bin, mostly with junk from prior to any of my testing for this topic so I cleared it out and started again and since I was Restoring from the Recycle Bin it was emptied and then “filled” with the next (test) deletion.

Doing the tests was easy but writing them up a pain. Hopefully that is the last but one such epic post I produce.

All my tests were conducted with PL8 running and the test directory selected for testing convenience and DxPL(Win) has certain “features” that add (or delete) from the overall expectations.

The “discover a new image” with Auto DOP import not selected and then get a “nice new” VC when the DOP is then imported manually is a documented feature of DxPL(Win), with a nice added twist if metadata auto sync is also not in operation.

As we have both lamented in the past, swapping the [M]aster and VC[1] edits is tedious for 1 image and essentially impossible for a number of images and deleting the [M]aster is what this topic is all about.

It should be possible to differentiate between the[M]aster as an image and the [M]aster as a series of image and metadata edits and/or be able to swap edits (image and metadata) between any two copies and that command should be available for one image or many.

Given the current state of DxO implementing any of the “required” long-standing changes I don’t expect any action whatsoever.

platypus · November 18, 2024, 3:13pm

Originally, PhotoLab made no distinction between Master and Copy. Deleting the master would simply renumber the copies and no further harm came off it. But it was not easy to understand in the beginning, at least it wasn’t for me and I therefore posted a feature request to distinguish master and copy - and it got implemented a few releases afterwards.

If we don’t differentiate between master and copy, we’d need to understand that any image, be it master or copy, was just a meal cooked with the same ingredients (OOC image data) and different recipes. Once we understand this, we need not care about master and copy - because everything is just a copy and that the OOC image is never displayed anyways.

What name we see is just a label that could be changed.
Actual view would be copies derived from their respective recipes from the database/stored in the .dop sidecar file. All images only become real on export.

BTW: In DPL’s database, there are tables for the red (OOC) items and the blue (viewable) items. For DPL 7 on Mac, the tables are “ZDOPSOURCE” and “ZDOPINPUTITEM” and even though the same filename exists in both tables, they have different UUIDs.

Also see Make "master" deletable again / treat all copies the same

Wolfgang · November 18, 2024, 5:22pm

So you’ve noticed that I only read these long-winded posts in detail when I have to.

.

Normally I set the dop-file handling to “automatic” and just because I was interested in the matter, I tried around … with the result already described.

.

After all, PL shows a warning that cannot be overlooked.

While I used to live with the fact that deleted files were gone, I now know that

in I/E auto mode and a separate restore (of pic and dop-file)
“only” the former VCs are lost
and then M appears as VC 1 …

Screen Shot 11-18-24 at 12.21 AM

( and I have no idea if you described that already )

BHAYT · November 18, 2024, 6:47pm

@platypus The DxPL(Win) DB has ‘Items’ and ‘Sources’ where ‘Items’ point to ‘Sources’ and ‘Sources’ point to ‘Folders’. Using the indexes a ‘Folders’ entry can find all the ‘Sources’ entries that are pointing to it (they should then have counterpart disk image entries).

Similarly from ‘Sources’ by using an index on ‘Items’, where ‘Items’ is pointing to ‘Sources’, it is possible to determine all the entries that are related to one ‘Sources’ entry. One entry is the [M]aster and the others (if any) are the VCs but really all should just be copies.

However, the current implementation differentiates between them and then things like copy swapping etc. become impossible.

VCs share the same ‘Sources’ entry as the [M]aster and the UUID at the end of the DOP is the ‘Sources’ UUID and you already know where the ‘Items’ UUID sits.

So using that program of mine we have

[17:58:51] @ line     2  Date = "2024-11-17T18:07:02.7678710Z",
[17:58:51] ----------------------------------------------------------------------------------
[17:58:51] @ line     8  ---->Albums = "",
[17:58:51] @ line     9  CreationDate = "2024-11-17T17:54:20.7350079Z",
[17:58:51] @ line    20  ColorLabel = "Red",
[17:58:51] @ line    21  ModificationDate = "2024-11-17T17:54:29.3178793Z",
[17:58:51] @ line    22  Name = "P1112346.JPG",
[17:58:51] @ line    28  Rating = 1,
[17:58:51] @ line   497  Uuid = "FAD7A34F-1CDA-464A-AE99-EEE43BE677EC",       <----
[17:58:51] Album_count = 1 Uuid_count = 1
[17:58:51] ----------------------------------------------------------------------------------
[17:58:51] @ line   502  Uuid = "B6EE6A6A-7CB4-4C1E-8360-F5C26EE3D199",       <----
[17:58:51] Album_count = 1 Uuid_count = 2
[17:58:51] ----------------------------------------------------------------------------------

The “Album_count” (effectively the copy count, including [M]) represents the total number of copies in the DOP and each will have an entry in ‘Items’ in the database with a UUID (unique id.)

Every DOP will end with an Album-count = X , in the program, and a UUID count = X+1 and that last one is the UUID. of the ‘Sources’ entry and I have no idea how that might be used, might have been used or could be used whatsoever.

With respect to the decoupling of the meat from the recipe that is what is required to allow a number of things to happen that would increase the flexibility of DxPL.

Add a ‘Remove Edits’ command combined with e,g, Alt Delete to remove only the edits. But the way that DxPL is built does not allow for any entity to exist in limbo, although it can exist without any edit values. However, what is actually required is that the [M]aster can have the edits removed but continue to exist.

Data fields that are currently assigned to ‘Items’ should be assigned to ‘Sources’ instead but that would be too radical a change.

If effectively the ‘Sources’ entry is going to be removed (the deletion of the [M]aster has been requested by the user), the current ‘Remove’, command should look at it that way round and treat the deletion as a single atomic transaction, effectively leaving all the edit data intact as at the time of deletion.

@Wolfgang

I never would have guessed!?

In great eye watering detail!

The good news is that I have been trying to reduce my involvement with the Forum and that resolve waned somewhat with the release of PL8.

But the large post above, and you effectively ignored a much smaller one, means that now is the time to find a more fulfilling “hobby” to keep my brain functioning.

Regards

Bryan

platypus · November 18, 2024, 11:29pm

…or at least more complicated. We’d need a temporary image to shuffle settings…like with Tower of Hanoi.

Also, the databases of DPL(Win) and DPL(Mac) are very different, but this does not really matter for restoring deleted images which is buggy in DPL(Win). And the issue has been reported by @SAFC01 as noted here.

SAFC01 · November 19, 2024, 5:23pm

I’ve had an initial response from DxO (beyond the first thank you for your support request) - see below.

Thank you for reaching us,

This issue, as you’ve noted, does not occur on macOS, which can be quite perplexing. Unfortunately, this behavior is present in both PhotoLab 7 and 8 on Windows.

As a workaround, before removing images, consider manually backing up the DOP files containing your Virtual Copies. This way, you can restore them if needed. You might also want to check if there are any updates or patches available for DxO PhotoLab 8 that address this issue, as it’s the latest version and more likely to receive updates.

I hope that at some point the issue is fixed in PL8 (and PL7). As and when I hear anything further I’ll update this thread.

BHAYT · November 19, 2024, 6:14pm

@SAFC01

That response is about as useful as a chocolate teapot for one very simple reason, you can take a backup before your start editing, you can take backups while you are editing but none of those backups will be much use if you haven’t made it just before you make the mistake of deleting a [M]aster!

There is no DOP backup “button” within DxPL to allow a user to instantly backup an image (with DOP and sidecar), or the whole directory, at any chosen point in time.

Even then the user would have to be psychic for that advice to be of any use whatsoever.

Sadly I worked as a developer for 36 years and I am appalled at that “excuse” for a “workable” strategy coming from a developer/support person, although I had guessed that would be the most likely response.

SAFC01 · November 19, 2024, 6:45pm

Yes it’s not a great response but as a stop gap I’m not sure what else better could be said.

It’s certainly no help for the accidental and unintended Remove of an image Master and VCs. The warning you get to confirm the Remove stating the number of VCs (if there are any) might serve as a reminder to export and copy and DOP prior to doing this - but also relies on users being aware of the issue and need in the first place.

Ever the optimist, I haven’t given up on it being “fixed” properly at some point

BHAYT · November 19, 2024, 7:10pm

@SAFC01 I am not querying the fact that no other response could be made, it is using the term “workaround” as if it actually works around the problem being reported.

It typically allows a user to return to the start of the editing session but there might be issues with the backed up DOPs versus the state of the database after the “accidental” deletes, i.e. the safest strategy is to backup the database and the DOPs (and the images and sidecars) before starting any editing session!?

You might like to test what happens if you replace new DOPs created during an edit session with old DOPs copied at the beginning.

The replacement of just the image, DOP and sidecar after an accidental delete is a bit of a struggle to put it politely and you will be back at the start of the editing for that image but at least you will have recovered the image if that was part of the backup.

Sadly this program is just passed the starter gun but nowhere near finished yet and only makes a “crude” process easier to execute, or rather just the backing part up of the “crude” process.

platypus · November 20, 2024, 7:43pm

One issue of accidentally deleting a master (when DPL is turned off)k is, that the DB still has all the entries for the deleted images. There is no way to fix that issue from the menu, but on macOS DPL deletes the entries and therefore looses all memory of these files. This made the restores fairly reliable in my tests, provided I restored the files with the correct versions of .dop sidecars.

This “behind the scenes cleanup” is fairly new and is maybe also coming to DPL on Windows, but I’d certainly not hold my breath. Also, I can’t really be sure that the cleanup works reliably. I just discovered that it existed while testing other things.

Deleting (with DPL) and restoring (with Finder) files (and VCs) in rapid succession a few times while DPL was left on did not throw off the DB list of sources and items though. I suppose that manual DOP export and automatic import would be a safe setting for this test.

BHAYT · November 22, 2024, 3:12pm

@platypus I repeated your test in a directory intended to test Counting DxPL DB contents. Discovered 6 records and got counts of

[15:00:23] Source DB is present = C:\Users\bhtho\AppData\Roaming\DxO\DxO PhotoLab 8\Database\PhotoLab.db DBSize = 74,752 bytes
[15:00:23]          6 Total of all in Sources (original images)
[15:00:23]          6 Total of all in Items (Images including VCs)
[15:00:23]          6 Total of all in Metadatas (1 for each Image)
[15:00:23]          6 Total of all in Iptc (1 for each image)
[15:00:23]          9 Total of all in Folders (1 for every directory level)
[15:00:23]          3 Total of all Drives (Volumes) in Folders
[15:00:23]          6 Total of isRaw in Metadatas (1 for every RAW image)
[15:00:23]          0 Total of Projects
[15:00:23]          0 Total of Items in Projects
[15:00:23]          0 Total of Keywords
[15:00:23]          0 Total of Items in Keywords

I then deleted the images while DxPL was focused on an empty directory and then navigated back to the directory where the 6 images had been and got these counts

[15:03:19] Source DB is present = C:\Users\bhtho\AppData\Roaming\DxO\DxO PhotoLab 8\Database\PhotoLab.db DBSize = 74,752 bytes
[15:03:19]          0 Total of all in Sources (original images)
[15:03:19]          0 Total of all in Items (Images including VCs)
[15:03:19]          0 Total of all in Metadatas (1 for each Image)
[15:03:19]          0 Total of all in Iptc (1 for each image)
[15:03:19]          9 Total of all in Folders (1 for every directory level)
[15:03:19]          3 Total of all Drives (Volumes) in Folders
[15:03:19]          0 Total of isRaw in Metadatas (1 for every RAW image)
[15:03:19]          0 Total of Projects
[15:03:19]          0 Total of Items in Projects
[15:03:19]          0 Total of Keywords
[15:03:19]          0 Total of Items in Keywords

So it appears that DxPL(Win) also cleans up when it discovers changes in already discovered directories.

Thank you for bringing that to our attention.

SAFC01 · December 12, 2024, 1:01pm

This is the latest response to the issue of the VCs being deleted from the DOP before the Image and DOP are Removed to the Recycle Bin.

DxO asked me to install and run a diagnostic that harvested a number of DxO PL and Windows files into a ZIP file and return it., which I’ve done. The further response was:

Our Team are currently checking this now, and will get back with you once I hear an update from them.

We want to manage your expectations by letting you know that the process of checking and investigating may take some time.

However, we will definitely provide you with updates along the way.

I can’t see what new information the collected files will reveal. As has already been described above, with the Windows version of PL you can see each VC being deleted and the DOP updated before the image and DOP are Removed to the Recycle Bin.

Anyway, I’ll update this thread with any further responses.

BHAYT · December 12, 2024, 3:38pm

@SAFC01

Arguably they are verifying for themselves what we discovered empirically and reported above.

We are unable to log the exact event as it happens but we can show the before and after details based on our knowledge of how the product works, but that knowledge does not come from being able to access the code and may be flawed (actually I don’t believe it is flawed but that is always a possibility).

Thanks for the update.

SAFC01 · January 9, 2025, 5:33pm

I’ve just had a further response from DxO support - see below. So they confirm the Windows behaviour is a bug and that it’s being worked on.

This is a follow up message from the Team,

We have checked on the Bug and currently working on it, once fixed we will have it updated on the future versions.

BHAYT · January 9, 2025, 7:50pm

@SAFC01 Thanks for the update