Can we please establish some concrete facts amongst the comments that has been written in this Topic. The only reason that I looked at this topic was a conversation I had on another post with @nwboater he become concerned about going near PL5 after he has seen what has been written here in this topic!
The first fact is that I don’t work for DxO, nor am I an “apologist” for DxO (I have written enough critical posts that should be clear by now), neither do I have access to any of the code.
All my opinions and “facts” come from empirical analysis (testing) and it is always possible to get the tests wrong, to interpret the test results incorrectly or to make a final judgement that is either wrong, partly wrong/right or spot on but DxO rarely acknowledge any of that!!
In truth although DxO ask for additional evidence they don’t always acknowledge that a fault has been detected and as for details when the fix is finally available or why a new release of PL5.2.0 has appeared who knows. That may be written down somewhere but I haven’t stumbled across it yet, and there should be no need to stumble across it, it should be writ large whenever any release is made!
But neither do I like gossip which propagates half truths or even truths where the reason has not been investigated, but in a “fact” vacuum rumours start and spread with no real information to quash them.
Lastly even the introduction to one of my posts is long and the post itself is typically really LONG - sorry!
To database or not?:-
I have discussed @John-M’s strategy with him on a number of occasions (What is the best way to move a folder of photos with their respective *.DOP and *.XMP files? - #15 by BHAYT) and it works for him and will work for others who accept the strengths of the approach coupled with the limitations!
Personally I have been “destroying” PL5 databases since I started beta testing PL5 at the beginning of 2021 but would prefer to have a strategy that leaves me free to use all the features of PL5 should I so choose and that includes Projects, keywords, Ratings, Rotation, Tags (except there is a bug in Tag handling which I believe is still present in PL5.2.0) and that requires either the careful use of database and/or careful husbanding of the metadata WHICH FOR THE [M]ASTER COPY OF THE IMAGE IS NO LONGER AVAILABLE FROM ANY DOP BEYOND PL4 (Currently) (Please see the explanation below)!
But @John-M has made one point that others in this topic have hinted is not true, namely the veracity of the data in the DOP. The DOP is reliable, almost, the almost refers to the fact that there is a bug with respect to ‘The Tag’, a “DxO only” piece of metadata, that only resides in the DOP and which is currently not being read back from PL5 DOPs or at least it wasn’t the last time I tested!
Is the PL5 database particularly fragile:-
The database product used by nearly all the DAM software and much other software besides is SQLlite with ACDSee being the most notable exception which uses either DBASE or a variant thereof and that is potentially the reason it can handle huge databases where there might be “horror” stories about the others when the number of images rises!
Having made my living as a Mainframe database specialist for 36 years I can say that my “only” concern about the DxPL database is that some of the structures are a bit “light” and most of the other DAM databases are somewhat “heavier”, i.e. the structures are carrying more data, potentially because they offer more organisational features, perhaps because they have more self checking features built in!
Please see this (rather sternly worded) post which I made in response to a post made by @platypus, whose proposals for PL5 database features I agree with and with whom I have discussed enhanced file management features that should be added to PL5 (sooner rather than later), but not with the comments that @platypus made in this case PL5 Tag Field not read from .dop file - #106 by BHAYT
The Lightroom Catalog falls into this same category (and can be opened with the normal SQLlite utilities after the extension has been changed to .db) and the main thing that these other systems offer that DxPL does not offer is the ability to manage multiple databases (albeit when I started PL5 Beta testing I discovered LightRoom forums like this one full of arguments about one database or many)!?
Sadly that is not an argument for this forum because PL5 does not offer such a capability (and yes I would like DxPL to be multiple database capable).
If you want many databases then use the backup feature naming the database you are backing up appropriately and open an old backup, even one from a previous release (certainly PL5 can open a PL4 database!) Is it as elegant as Lightroom, absolutely not, is it useful, potentially but the user is responsible for the management and DxPL is not going to lend a hand (certainly not at the moment)!
An investigation of the database by one of the posts above seemed to indicate multiple copies of the same data and potentially disused copies with the possible risk of PL5 becoming confused!
I believe the following statement is correct
-
If data is removed via PL5 then the entries in the database will be expunged immediately but sadly that command will also remove the image from disk! There is currently no way of removing items from the database only. One of the features I (March 2021) and others (@platypus - Dec 2020) would most like to be added to DxPL to allow elements of the database to be cleared but the data to be available for re-discovery by PL5!
-
If data is removed from disk “behind PL5’s back” how is PL5 to determine what has actually happened (without what I want added in item 1 above to avoid the need for such deletions!). Having purchased IMatch at Christmas it does appear to be scanning directories in the background for imported directories and warning that ‘Files are offline’. It is a dedicated DAM with more features than I will …, PL5 is a very good raw editor with the possibility to be a good metadata handler, when certain features have been added etc. etc.
The advantage of the current “discovery” mechanism in PL5 is the ease of use, testing other products which need an explicit import process is tedious in the extreme.
So when a database (and they all have one) purports to represent an image of the directory structure and users start renaming files, directories or deleting them wholesale from the actual directory structure then there will be problems.
Singling PL5 out for special criticism is simply cynical and an unfair representation of the issues that all such systems “suffer” and must be built to endure!?
At least PL5 provides the DOP as a backup mechanism, providing it is used correctly and with the right expectations.
However, problems can certainly occur as they did with PL4 and documented in PL4.3.6.32 Ghosting Ratings for Same image in different directories (Win10) generated by BHAYT (me), but I don’t think I have seen this occur again!?
The problem does not necessarily lie with having a database or having a DOP but with the coding that supports the operations that rely on and update those storage mechanisms.
I object to statements like this (but respect the author and his right to make them):-
Sadly the above statements and others like it have been made since DxO first indicated its intention to strengthen its metadata handling and are understandable when it comes to the risk of DxO transferring resources from further developing the editor to the “frivolous” activity of developing a DAM!
But I do object to the part of the statement that makes the somewhat “elitist” suggestion that I might not be a “serious PhotoLab user” because I welcome the improved metadata handling or definitely will when it is finished and all the things I want are in it (PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts)!
Before I continue I want to point out two other posts that I have made which might be useful (with all the standard disclaimers and caveats)!
Synchronizing data between system:-
Do not simply move images and DOPs between DxPL databases on multiple systems or you are likely to encounter ‘Unwanted Virtual Copies’, I developed this procedure to allow this to be accomplished without Virtual copies or using the alternative of removing the database! Synchronise PhotoLab - #26 by BHAYT
In another situation data was being moved from one system to another via an external hard drive and I tested the process and finally found that it could be made to work successfully (when the database and the data are both on the same drive) even when the drive letters on the different system were actually mapped to different drives but in spite of requesting a reason that it worked no-one from DxO has had the courteousy to contact me and actually describe why it works!! Cache file - #21 by John-M and other posts after that one.
What is the cause of the Apparent Data Loss?:-
So what about the loss of data in the DOPs, well I have written about that so many times over the last few months that I am surprised that the issue is not yet better understood.
First please note that the documentation about the changes in this release (PL5) are lamentable, i.e. essentially non-existent!
So here goes:
The PL4 DOP:-
The PL4 DOP can still be used to carry ‘Rating’, ‘Rotation’ and ‘The Tag’ successfully into PL5 and nothing should be lost in that “migration”. This also goes for DOPs from even earlier releases!
This data was stored in and read from the DOP because of the “immutability” constraint that DxO placed upon the image data. Either good, safe practice or an “excuse” for not engaging with all the standards surrounding metadata handling or a bit of both, until PL5!?
What happened in the move to PL5:-
All this changed with PL5 which now stores metadata where it arguably belongs, i.e. in the embedded xmp metadata or in the xmp sidecar data of the image and two of the pieces of “missing data” are now written to the appropriate xmp metadata along with keywords which previously only existed in the DxPL database and would have been lost if the database was destroyed, deliberately, accidentally or corrupted by a bug!
I have written an experimental procedure to recover keywords locked in a PL4 database if anyone is interested. There is currently no post and it is part of a very very very long document that I consider to be the missing PL5 FAQ but it still needs yet more work and I may abandon it anyway. If anyone wants the procedure than I can extract the relevant instructions - all normal discla, etc.etc.
But the data is also in the PL5 DOP!?:-
But the data is also in the PL5 DOP! Indeed it is and the DOP now contains the extended metadata for the [M[aster copy and any Virtual Copies which now includes keyword data as well. BUT the data for the [M]aster copy (often the only copy the user has) is effectively there for consistency with previous releases, consistency between the storage for all copies contained in the DOP (the [M]aster, VC[1] …) and it can be used again as it used to be pre-PL5 iff (if and only if) DxO relents and allows that data to be used routinely or on request (in a recovery situation)!
Currently the [M]aster copy of the metadata held in the PL5 DOP is WOM (Write Only Memory) and PL5 will NOT (currently) read that data back UNDER ANY CIRCUMSTANCES!!!
So you cannot secure the metadata just by securing the DOP!
You must :-
-
Write any metadata set in PL5 back to the image either as embedded xmp data (JPG, TIFF or DNG) or to the xmp sidecar file for RAW.
-
Secure the necessary elements along with the DOP, the image file with updated metadata for JPG, TIFF and DNG or the sidecar files and the image file in the case of RAW. If there is no metadata added in PL5 then it will not be necessary to force any metadata from PL5 to the xmp data!
It is potentially both these steps not being executed that have led to a spate of lost data and ‘Ratings’ returning to 0 etc… This isn’t some weird data loss, this as a result of the the changes DxO deliberately made to the handling of the ‘Rating’ and ‘Rotation’ fields in particular.
Sadly the implications of these changes have never been properly broadcast to the users by DxO, arguably this lack of communication is the real bug!!
So where does PL5 Expect the metadata to exist?:-
So while the metadata for the ‘Virtual Copies’ is still retrieved from the DOP, PL5 expects the metadata for the image to come from the image, i.e. embedded xmp metadata (JPG, TIFF, DNG and RAW if other software has put it there) or sidecar xmp metadata for RAW files if present, along with the IPTC and Exif data, including GPS data!!
But how can the image acquire that metadata set in PL5? :-
The answer to that one is not at all if the appropriate settings and/or operations are not made by the user, e.g. AS(ON) and/or ‘Files’/‘Metadata’/‘Write to Image’ and/or ‘Files’/‘Metadata’/‘Read from Image’ or AS(OFF) and ‘Files’/‘Metadata’/‘Write to Image’ and ‘Files’/‘Metadata’/‘Read from Image’.
If neither of these is set or executed then the metadata will not be read from or written to the image metadata and will remain in the PL5 database, with the original metadata (if any) still intact in the image file. Please note that the initial discovery will perform the equivalent of a ‘Read from Image’ to acquire the image metadata. (Except when a PL4 DOP is encountered and I will write that up in another post!)
But I have been told not to have AS(ON)!:-
By cautious users probably and that potentially includes me!
AS(ON) (Always Synchronize) makes PL5 watch for metadata changes that may occur to an image externally and update the database accordingly, storing certain elements in the DOP. But the other part of the ‘auto’ is that PL5 will also write back any metadata to the image if it changes in PL5 or PL5 decides that it is not in the format that it thinks is correct and changes it!!
The reason for the caution is what happens if PL5 gets things wrong and writes back the metadata and corrupts the image! The only reason for being cautious with PL5 and less so with other DAM software is simply age. PL5 is comparatively new and this particular release has had a way more chequered “career” thus far than previous releases
@uncoy would point to the metadata “distraction” for this “buggy” release and there might be more than an element of truth in that. Personally I would point to DxO simply not publishing the information users needed to verify their current workflow and make amendments (if any) as necessary.
The other reason is that the DAM users in particular do not want PL5 changing their carefully crafted (by their DAM) metadata and writing it back to the image. Even if the image metadata is protected, AS(OFF) and only use ‘Read from Image’ or the new ‘Conflict Resolution’ ‘S’ icon there is still the issue that PL5 will apply its “rules” to the metadata in the images exported from PL5.
But I would rather have the metadata features and suffer the issues that we have suffered than not have the features at all.
But DxO must do better with testing, listening to its users even more and use that userbase to undertake joint testing to improve the standing of DxPL with other metadata handling software and communicate more about the changes that they have made and the implications on the users workflow, which they cannot begin to judge if they haven’t consulted with their users in the first place!!
What AS(ON) does well is effectively “merge” metadata changes from the image (potentially put there by a DAM) or made by the user in PL5. It is not a real merge but PL5 flip flops between reading new data from the image and writing new data to the image whenever there is a change in PL5 or detected in the image!
What are the alternatives if AS(ON) worries me?:-
The answer to that is AS(OFF) but the user is then responsible for instructing PL5 to write to the image file or read from the image file. That is now being helped by the recent ‘Conflict Resolution’ additions by these need to go further.
What problems need resolving;
An option to stop PL5 changing the image metadata format and changing it on the way to an export:-
There needs to be a way that users can insist that PL5 leaves the metadata untouched, i.e. no potential write back and no change to the data output in the export phase.
A Compare option:-
A 'Compare ’ function is really needed because I have encountered situations where PL5 failed to detect an external metadata change! I was unable to isolate the circumstances and could not predictably reproduce the error. This mean that PL5 could miss a change made to the metadata made by the DAM. If all the DAM work is done before presenting to PL5 then there should be no issue but …
The alternative is to execute a ‘Read from Image’ on a selection of images before closing the database etc. but the 'Read from ’ is a “destructive” command, i.e. any (if there are any) changes made in PL5 will be obliterated by the data “coming in” from the image!
Add the missing ‘Conflict Resolution’ icon when changing metadata in PL5 results in a mismatch with the external image data (AS(OFF)):-
Without the missing icon and the icon to show the direction of the mismatch the product is simply incomplete!