Sorry this turned into a big post but the main element is that the DxPL clean-up routines are “flawed” and I am interested in what other packages do or don’t do!
@Stenis I am glad we have resolved the issue because I was very puzzled.
Exactly, but there are other simple fixes I have suggested over the last few years and they just fall on deaf ears!
Super easy to use but don’t keep it running while you “bounce” DxPL because sometimes it causes problems because it keeps the database open and don’t be tempted to change anything in the database!
However, using it on a “real” database is a nightmare because it is difficult (impossible) to wade through all the data, hence I back-up the “real” database and open an empty one I created some time ago for tests to keep the data down to a minimum so that I can “learn” about whatever it is I am trying to investigate. Returning to the “real” database after the tests.
However, one problem your original issue highlights is the changing nature of data on disk versus the data in a database, and that software can find it difficult to track the changes and any mismatches between the data on disk and the data in the database, i.e.
- What is the software supposed to track, when it is running and when it is not?
- When is it supposed to track it?
- How is it supposed to track it and detect any changes?
- What is it supposed to do when anomalies are detected?
Some image databases may contain very large numbers of entries (images) and setting a reconciliation program loose on such databases is an “expensive” and (potentially) disruptive task.
One of the better programs for this kind of work is iMatch which “insists” on doing almost everything using background worker tasks including with delays to avoid “lock” clashes with other programs, e.g. I just opened iMatch and
I don’t have any experience of when and how other programs detect that either a directory that contained images or an individual image or images that have been “ingested” into a database no longer exist, e.g. deleted or name changed.
However, DxO certainly doesn’t or rather it actually does detect a directory name change or deletion when it is active, i.e. the process that it uses to detect changes in the file system and image metadata provide DxO with an opportunity to recognise changes and act accordingly.
But as tests I did some time ago showed it does nothing with that information or rather what it does has a limited impact on the database, nor does it adjust the database to reflect the fact that a directory has changed its name or been deleted, other than recognise the existence of the new directory.
So the following tests consist of a number of directories and sub-directories holding one image in each sub-directory, i.e.
The directory “Test Sub 04” had its name changed to “Test Sub 04 - New” with DxO running and the new directory was added immediately and the old directory vanished from the display.
But the old image from the changed directory is still in the database (image 4) and the “new” one has been added (as image 5 in ‘Items’)
and the new folder added to the ‘Folders’ table (entry 16 and the old directory is at 15)
The image from the " - new" directory was then drag and dropped on a ‘Project’
and the (sub-)directory was changed back to its original name.
This then happens with the ‘Project’ entry, effectively “orphaned” from the actual image. The thumbnail is still in the cache so it can be displayed but the image itself is …!
Unfortunately the problem then gets worse @Musashi when I do a search on “red” I get the following
The images were “rated” according to their sub-directory from “" to "***” and given “colour” labels according to the directory, with “red” being assigned to “Test Directory 01” where there should be 4 active entries but, because DxPL has not discovered and removed the now missing entry we have 5.
Re-indexing “Test Directory 01” to ensure the correct number of entries in the database we (still) have 5.
@platypus for some time elements that you and then I have complained about have always seemed odd (counter intuitive) to me;
No Command to Delete a Directory:-
There is no command to delete an entire directory.
Its “omission” could be to avoid the risk of a user executing a delete on the whole library of images with a single command but it has always seemed strange that an obvious “directory delete” command is missing, i.e. the user can delete the entire contents of a directory but leave behind an empty directory “header” albeit such a destructive operation is “limited” to the boundaries of a single directory and does not include any other (sub-)directories!?
Or the ‘Folders’ structure is considered “sacrosanct”, hmm…
No “database only” delete capability:-
Possibly even more important than 1 is the fact that there is no command to only delete database entries, the only command on offer is to to delete (‘Remove’) everything, i.e. image, DOP and xmp sidecar file are eradicated!
The most obvious first implementation would have been to provide a less destructive command to delete database entries, i.e. entries within the DxPL domain and leave the images intact.
The ability to also delete the images could then have been added at a later date.
But instead DxPL can only delete everything or nothing, it doesn’t make a lot of sense @Musashi!
With such a database only delete command, the danger of deleting an image and the DOP etc. with a single command is reduced because “only” the database entries are at risk!?
How do other packages manage database to “real” world reconciliation:-
I don’t know how other packages resolve the issue of garbage collection (with the exception of the example from iMatch I included above which is working as I type) when changes are made outside the package.
But having a situation where “phantom” images are located via a search because the database can become littered with the “detritus” caused by the lack of detection and clean-up routines is not good (but I have only tested DxPL for this behaviour).
The (re-)indexing routine needs to be improved to allow its influence to be restricted to refreshing only directories already present in the database, rather than adding images to the database that are potentially of no interest, as I mentioned in an earlier post.
A further option could be provided to embark on a reconciliation process that should be able to clean up the database and remove detritus associated with data that is no longer present on disk.
iMatch is going to take some time reconciling the database with the disk contents (a consequence of moving test directories from F:\ to H:, I believe)
dismissing the main import leaves a progress window
So DxPL has some way to go but I am still interested to know what capabilities Lightroom and Capture 1 have to handle such situations and I believe you have experience of both @platypus (and @JoJu?)
@GrahamB Sorry but I don’t agree, it is certainly flawed but nothing that cannot be fixed if DxO put their minds to it, some fixes don’t require much effort at all (I suppose I must add , in my opinion) but getting DxO to do the work is …
The only database that was corrupted, excluding PL5 losing my database during the upgrade from PL4 and simply creating a new one, was when I was beta testing PL5 and used a trial copy of Lightroom to set metadata to test PL5’s ability to detect the changes.
The third time I tried to use the Catalog (database) Lightroom informed me that it couldn’t open it!
As a “newbie” to Lightroom I may well have made a mistake but there appears be good reason for the backup prompt every time you close Lightroom, and IMatch and Capture One and … but not DxPL!
Not having that prompt makes testing much easier but should there be an option to instruct DxPL to prompt …?
sorry, couldn’t resist. Good luck with further testing, it’s very entertaining.
