What if you go and try it out?
Copy a bunch of pics with their associated dop-files on a stick and put it on your new machine.
With a brandnew database you loose star ratings and keywords.
Otherwise try the same thing, but first copy the old database to your new machine …
You then still can start from scratch.
…yes, or move the old structure + database over and reorganise on the new machine, which might be more powerfull and less impregnated by leftovers…
The Photo Library is to be considered being 1.0. It is first in version 5.x that we got something that looks like an Image Library at all. In fact I was pretty surprised that it’s as good as it is despite it lacks a few things.
This discussion here I think is a little bit unfair too. I don´t think there is one single library software that is comfortable with people moving folders and files outside the application. If you do things like that you have to act to resync and the discussion would be more fruitful if we could have a discussion about the tools in Photolab to resync.
There are a few ways to solve metadata issues: either have a consistent way of storing the edit changes and IPTC-metadata in XMP-sidecars along RAW-files or writing it all into the XMP-headers of XMP-compatible files like with JPEG:s (TIFF:s that doesn´t really have an XMP-header) and DNG:s. Depending only on a proprietary metadata database like in Lightroom is to ask for single point of failure crashes. In PL the problem is the opposit. Here we have XMP-sidecars, DOP-files and a database - and not to be surprised all sorts of syncproblems instead.
For most RAW-converters the RAW-file integrity is holy but not for Sony for example because they write edit data straight into even the RAW-files in their software Imaging Edge. Bad? Well I can see one advantage with that aproache. All the settings made in their cameras like settings in their style sheets e.t.c. are all saved into the files and are read by their converter.
Personally I´m not at all pleased with this mess. Ideally I would prefer that they would be writing both editing data and IPTC and EXIF metadata both to XMP (to sidecars or into the files) and EXIF too for compatibility reasons (because some old software like for example Windows is not compatible with XMP and prefer old fashion EXIF since it doesn´t read the EXIF namespace in XMP). If they had done that and someone had moved a folder outside the application it would have been fine just to just reindex that folder/folders. After a reindex the new search path would be written into the files XMP instead of the old. They could still update the corresponding DOP-file/files in the same reindexing process of pure compatibility reasons but since DOP is as proprietary as the Image Library database the master has to be the XMP-data since that is the only ISO W3C-standard we have to lean on.
In the new version 5.2 there is supposed to be a conflict resolution tool at least.
If they could migrate all this metadata to XMP as a metadata master the only thing we should have to do would be a reindexing of the folder/folders we have moved - even the ones moved outside PL. Even a metadata system like the one in PL has to have one master source and not several as we in practise have today. There also has to be possible to chose if Photo Library in PL or an external application shall own the data because that decides which direction the workflow shall have. It´s never a good idea the use bidirectional metadata workflows. That is also to ask for problems when data might get forked differently in the different directions.
Even if we use PL and Photo Library today we always have to be prepared for the day we have to migrate of some reason and today and for the forseable future XMP is the only standard there is to secure a smoth migration. DXO has not solved these problems yet when we use proprietary RAW but with DNG I think it should work since there all the changes are saved booth into the DNG “RAW”-data and the baked in JPEG that might even be in “full size and quality” and so is the XMP-metadata.
Rexblock, Wolfgang & Platypus. Thanks so much for your thoughtful responses to my dilemma. The last couple of days I’ve been knee deep in taxes. Hopefully i will finish that tomorrow and then can again properly focus on the fun stuff. 
As much as I appreciate all your suggestions I am spooked about the problem of losing PL corrections when I move my file from the old PC to the new one. Thus I sent a request a few days ago to DXO expressing my concern and asking them how they recommend doing it. I sure hope they will give me an answer. Once they do I will share it here.
Rod
Please do let us know what DxO recommend for transfer to different environments. I expect that a simple database export/import across platforms will verify and that your corrections will be intact.
Please do let us know what DxO recommend for transfer to different environments. I expect that a simple database export/import across platforms will verify and that your corrections will be intact.
I sure will Rex, and will be very pleased if it’s as simple as your suggestion.
This works as expected (I do this on a ± regular basis) … unless images are moved to other directories or folders with something that is not PhotoLab, which is to be expected. After all, no inventory system can keep track of goods stolen from a shop…
Unacceptable. All changes to image processing should be in the .dop, all changes to metadata should be in a .xmp file or in the worst case in the .dop (although it would be much better and more compatible to record them in standard form in an .xmp file).
DxO so close, yet so far, from allowing file portability. At this point, I’ll just have to abandon DxO’s metadata features as the information is not portable. I don’t fancy having to redo metadata from scratch or try to extract it from jpegs three or five years from now.
The approach I’ve been using, for a long time now, is to rely only on sidecar/.dop files.
I invoke PhotoLab via a “wrapper” (see below) that starts by deleting the database, before calling the PL executable
A new database is created by PL for the current session, which is then deleted next time I start PL.
This doesn’t cause me any problems at all, because;
I only ever use PL within a “work in progress” folder, containing only new images. Once processed, the RAW file plus its associated sidecar/.dop file is moved off to my structured file-storage system.
The only information stored in the database that’s not also stored in the sidecar files are related to projects (virtual collections of images), ratings & keywords (in which I have no interest at all) and (for the Mac version) multi-session history of corrections (which is no great loss, IMO) … All image corrections are held in the sidecar/.dop files.
That’s actually not the case at all (see above). Provided you generate sidecar/.dop files and keep them with the RAW files to which they’re associated - then ALL your corrections will be safely retained.
In fact, it’s much safer working this way than expecting (hoping!) the database will never be corrupted.
That’s pretty much how I work too.
See above (top of this post).
You’re not losing any of your corrections - They’re ALL safely stored in the sidecar/.dop files.
Yes - I can confirm this too … I have never experienced loss of any correction data whatsoever.
Yes, that’s true for some non-correction type “things” (see above - at top of this post). However, my daily experience with deleting the PL database, and relying only on sidecar/.dop files, proves that absolutely none of any crucial correction details are lost in this process.
All good, Ian; You can safely continue working that way.
That’s a good point, Mr.P … OR when PL is not running; otherwise, PL can get a bit confused.
To completely avoid that potential problem, you might consider deleting the database between sessions (since, like me, you’re obviously not interested in projects, ratings, keywords and correction history).
Which implies you’re still dependant on the database (as it now records the new location of the images). Personally, I prefer to rely only on the sidecar/.dop files - - with caveats outlined at top of this post.
Regards, John M
PS - Clarification: By “multi-session history of corrections” I’m referring to the minutiae of all the step-by-step adjustments made in achieving the final result of your corrections. This does NOT refer to the final result itself (which is always retained in the sidecar/.dop files).
The simplest and safest approach would be to copy your image-storage file structure from the old-PC to the new one. Then open those folders with PL on the new PC … and PL will build a fresh, new database.
Which implies the following assumptions;
You have sidecar/.dop files for all your images (as they will contain all corrections for their images, which PL will use to create the new database).
You don’t care about some non-correction stuff that’s not stored in sidecar files … See the top of my previous post for a list of these.
John M
John-MMelbourne, Australia - Win10OpticsPro EA member
The simplest and safest approach would be to copy your image-storage file structure from the old-PC to the new one. Then open those folders with PL on the new PC … and PL will build a fresh, new database.
Which implies the following assumptions;
John M
John - Thanks very much for posting all you did on this and also for sharing your cmd file. I think this should be most helpful for people like me that don’t want PL to be involved in any metadata or to use PL as a DAM.
My first thoughts:
Thanks again,
Rod
Now that PhotoLab has IPTC data, I’d very much like to use PhotoLab as one-stop finishing of headline/title, caption/description fields as well as keywords and location data.
PhotoMechanic is not yet optimised for M1 architecture (and it may still be awhile) and PhotoMechanic is more than I need for my simple titling and captioning needs. On the other hand, if DxO does not safely store metadata in sidecards (either .dop or .xmp, I’d prefer .xmp), I won’t be able to use the PhotoLab metadata features and more or less no one else who values their time and the metadata enough to enter it, should either.
Very sad. The metadata feature is surprisingly well executed. To be let down on a structural flaw like this is deeply disappointing.
The observations here from @John-M would suggest another test. I find the PL5 wrapper/fresh DB create especially provocative. In this test, I would apply corrections in PL5; quit; make the database somehow unavailable or inaccessible; and then restart PL5.
In theory then, in the absence of a DB object from which to obtain image corrections, PL5 would be forced to ingest the associated sidecar file. This has been the problem all along – PL5 not importing corrections when needed.
I really wish someone from DxO development would speak up on this. I feel like I am describing an elephant to a blind man.
I sure wish they would to!
My request for help to them regarding moving PL5 and my library to a new PC was finally answered. The support person was confused about the issue and asked if I was losing edits. I had linked this thread and my earlier post in it. Hopefully my reply to the support person was more clear and I will get some kind of an answer as to how DXO recommends I do the move. I still haven’t done it.
I can’t help you with this question as I don’t use keywords - or any other DAM attribute.
I only proposed copying your file structure from one PC to the other because that would be the simplest thing to do. Change of drive-letter and/or folder-names would not be a problem. The only proviso is that you keep your RAW/image files AND their related sidecar/.dop files together at all times.
The .cmd file I attached above had only a limited purpose; to delete the database (and cache).
This is the one I actually use (compiled to an executable): Invoke_PL50(ClrDB+Cache).zip (401 Bytes)
You can store and run it from anywhere - it uses a system parameter as reference to your UserName.
Caution: It will delete your database and cache files.
John M
“Forced” is not quite the correct term - but, it would do as you describe …
I can be very confident in asserting this - because that’s how PL works for me every time I use it.
Note, however, that I don’t use ratings, keywords or projects (so, I don’t care that none of these are stored in sidecar files).
HTH - John M
Can we please establish some concrete facts amongst the comments that has been written in this Topic. The only reason that I looked at this topic was a conversation I had on another post with @nwboater he become concerned about going near PL5 after he has seen what has been written here in this topic!
The first fact is that I don’t work for DxO, nor am I an “apologist” for DxO (I have written enough critical posts that should be clear by now), neither do I have access to any of the code.
All my opinions and “facts” come from empirical analysis (testing) and it is always possible to get the tests wrong, to interpret the test results incorrectly or to make a final judgement that is either wrong, partly wrong/right or spot on but DxO rarely acknowledge any of that!!
In truth although DxO ask for additional evidence they don’t always acknowledge that a fault has been detected and as for details when the fix is finally available or why a new release of PL5.2.0 has appeared who knows. That may be written down somewhere but I haven’t stumbled across it yet, and there should be no need to stumble across it, it should be writ large whenever any release is made!
But neither do I like gossip which propagates half truths or even truths where the reason has not been investigated, but in a “fact” vacuum rumours start and spread with no real information to quash them.
Lastly even the introduction to one of my posts is long and the post itself is typically really LONG - sorry!
To database or not?:-
I have discussed @John-M’s strategy with him on a number of occasions (What is the best way to move a folder of photos with their respective *.DOP and *.XMP files? - #15 by BHAYT) and it works for him and will work for others who accept the strengths of the approach coupled with the limitations!
Personally I have been “destroying” PL5 databases since I started beta testing PL5 at the beginning of 2021 but would prefer to have a strategy that leaves me free to use all the features of PL5 should I so choose and that includes Projects, keywords, Ratings, Rotation, Tags (except there is a bug in Tag handling which I believe is still present in PL5.2.0) and that requires either the careful use of database and/or careful husbanding of the metadata WHICH FOR THE [M]ASTER COPY OF THE IMAGE IS NO LONGER AVAILABLE FROM ANY DOP BEYOND PL4 (Currently) (Please see the explanation below)!
But @John-M has made one point that others in this topic have hinted is not true, namely the veracity of the data in the DOP. The DOP is reliable, almost, the almost refers to the fact that there is a bug with respect to ‘The Tag’, a “DxO only” piece of metadata, that only resides in the DOP and which is currently not being read back from PL5 DOPs or at least it wasn’t the last time I tested!
Is the PL5 database particularly fragile:-
The database product used by nearly all the DAM software and much other software besides is SQLlite with ACDSee being the most notable exception which uses either DBASE or a variant thereof and that is potentially the reason it can handle huge databases where there might be “horror” stories about the others when the number of images rises!
Having made my living as a Mainframe database specialist for 36 years I can say that my “only” concern about the DxPL database is that some of the structures are a bit “light” and most of the other DAM databases are somewhat “heavier”, i.e. the structures are carrying more data, potentially because they offer more organisational features, perhaps because they have more self checking features built in!
Please see this (rather sternly worded) post which I made in response to a post made by @platypus, whose proposals for PL5 database features I agree with and with whom I have discussed enhanced file management features that should be added to PL5 (sooner rather than later), but not with the comments that @platypus made in this case PL5 Tag Field not read from .dop file - #106 by BHAYT
The Lightroom Catalog falls into this same category (and can be opened with the normal SQLlite utilities after the extension has been changed to .db) and the main thing that these other systems offer that DxPL does not offer is the ability to manage multiple databases (albeit when I started PL5 Beta testing I discovered LightRoom forums like this one full of arguments about one database or many)!?
Sadly that is not an argument for this forum because PL5 does not offer such a capability (and yes I would like DxPL to be multiple database capable).
If you want many databases then use the backup feature naming the database you are backing up appropriately and open an old backup, even one from a previous release (certainly PL5 can open a PL4 database!) Is it as elegant as Lightroom, absolutely not, is it useful, potentially but the user is responsible for the management and DxPL is not going to lend a hand (certainly not at the moment)!
An investigation of the database by one of the posts above seemed to indicate multiple copies of the same data and potentially disused copies with the possible risk of PL5 becoming confused!
I believe the following statement is correct
If data is removed via PL5 then the entries in the database will be expunged immediately but sadly that command will also remove the image from disk! There is currently no way of removing items from the database only. One of the features I (March 2021) and others (@platypus - Dec 2020) would most like to be added to DxPL to allow elements of the database to be cleared but the data to be available for re-discovery by PL5!
If data is removed from disk “behind PL5’s back” how is PL5 to determine what has actually happened (without what I want added in item 1 above to avoid the need for such deletions!). Having purchased IMatch at Christmas it does appear to be scanning directories in the background for imported directories and warning that ‘Files are offline’. It is a dedicated DAM with more features than I will …, PL5 is a very good raw editor with the possibility to be a good metadata handler, when certain features have been added etc. etc.
The advantage of the current “discovery” mechanism in PL5 is the ease of use, testing other products which need an explicit import process is tedious in the extreme.
So when a database (and they all have one) purports to represent an image of the directory structure and users start renaming files, directories or deleting them wholesale from the actual directory structure then there will be problems.
Singling PL5 out for special criticism is simply cynical and an unfair representation of the issues that all such systems “suffer” and must be built to endure!?
At least PL5 provides the DOP as a backup mechanism, providing it is used correctly and with the right expectations.
However, problems can certainly occur as they did with PL4 and documented in PL4.3.6.32 Ghosting Ratings for Same image in different directories (Win10) generated by BHAYT (me), but I don’t think I have seen this occur again!?
The problem does not necessarily lie with having a database or having a DOP but with the coding that supports the operations that rely on and update those storage mechanisms.
I object to statements like this (but respect the author and his right to make them):-
Sadly the above statements and others like it have been made since DxO first indicated its intention to strengthen its metadata handling and are understandable when it comes to the risk of DxO transferring resources from further developing the editor to the “frivolous” activity of developing a DAM!
But I do object to the part of the statement that makes the somewhat “elitist” suggestion that I might not be a “serious PhotoLab user” because I welcome the improved metadata handling or definitely will when it is finished and all the things I want are in it (PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts)!
Before I continue I want to point out two other posts that I have made which might be useful (with all the standard disclaimers and caveats)!
Synchronizing data between system:-
Do not simply move images and DOPs between DxPL databases on multiple systems or you are likely to encounter ‘Unwanted Virtual Copies’, I developed this procedure to allow this to be accomplished without Virtual copies or using the alternative of removing the database! Synchronise PhotoLab - #26 by BHAYT
In another situation data was being moved from one system to another via an external hard drive and I tested the process and finally found that it could be made to work successfully (when the database and the data are both on the same drive) even when the drive letters on the different system were actually mapped to different drives but in spite of requesting a reason that it worked no-one from DxO has had the courteousy to contact me and actually describe why it works!! Cache file - #21 by John-M and other posts after that one.
What is the cause of the Apparent Data Loss?:-
So what about the loss of data in the DOPs, well I have written about that so many times over the last few months that I am surprised that the issue is not yet better understood.
First please note that the documentation about the changes in this release (PL5) are lamentable, i.e. essentially non-existent!
So here goes:
The PL4 DOP:-
The PL4 DOP can still be used to carry ‘Rating’, ‘Rotation’ and ‘The Tag’ successfully into PL5 and nothing should be lost in that “migration”. This also goes for DOPs from even earlier releases!
This data was stored in and read from the DOP because of the “immutability” constraint that DxO placed upon the image data. Either good, safe practice or an “excuse” for not engaging with all the standards surrounding metadata handling or a bit of both, until PL5!?
What happened in the move to PL5:-
All this changed with PL5 which now stores metadata where it arguably belongs, i.e. in the embedded xmp metadata or in the xmp sidecar data of the image and two of the pieces of “missing data” are now written to the appropriate xmp metadata along with keywords which previously only existed in the DxPL database and would have been lost if the database was destroyed, deliberately, accidentally or corrupted by a bug!
I have written an experimental procedure to recover keywords locked in a PL4 database if anyone is interested. There is currently no post and it is part of a very very very long document that I consider to be the missing PL5 FAQ but it still needs yet more work and I may abandon it anyway. If anyone wants the procedure than I can extract the relevant instructions - all normal discla, etc.etc.
But the data is also in the PL5 DOP!?:-
But the data is also in the PL5 DOP! Indeed it is and the DOP now contains the extended metadata for the [M[aster copy and any Virtual Copies which now includes keyword data as well. BUT the data for the [M]aster copy (often the only copy the user has) is effectively there for consistency with previous releases, consistency between the storage for all copies contained in the DOP (the [M]aster, VC[1] …) and it can be used again as it used to be pre-PL5 iff (if and only if) DxO relents and allows that data to be used routinely or on request (in a recovery situation)!
Currently the [M]aster copy of the metadata held in the PL5 DOP is WOM (Write Only Memory) and PL5 will NOT (currently) read that data back UNDER ANY CIRCUMSTANCES!!!
So you cannot secure the metadata just by securing the DOP!
You must :-
Write any metadata set in PL5 back to the image either as embedded xmp data (JPG, TIFF or DNG) or to the xmp sidecar file for RAW.
Secure the necessary elements along with the DOP, the image file with updated metadata for JPG, TIFF and DNG or the sidecar files and the image file in the case of RAW. If there is no metadata added in PL5 then it will not be necessary to force any metadata from PL5 to the xmp data!
It is potentially both these steps not being executed that have led to a spate of lost data and ‘Ratings’ returning to 0 etc… This isn’t some weird data loss, this as a result of the the changes DxO deliberately made to the handling of the ‘Rating’ and ‘Rotation’ fields in particular.
Sadly the implications of these changes have never been properly broadcast to the users by DxO, arguably this lack of communication is the real bug!!
So where does PL5 Expect the metadata to exist?:-
So while the metadata for the ‘Virtual Copies’ is still retrieved from the DOP, PL5 expects the metadata for the image to come from the image, i.e. embedded xmp metadata (JPG, TIFF, DNG and RAW if other software has put it there) or sidecar xmp metadata for RAW files if present, along with the IPTC and Exif data, including GPS data!!
But how can the image acquire that metadata set in PL5? :-
The answer to that one is not at all if the appropriate settings and/or operations are not made by the user, e.g. AS(ON) and/or ‘Files’/‘Metadata’/‘Write to Image’ and/or ‘Files’/‘Metadata’/‘Read from Image’ or AS(OFF) and ‘Files’/‘Metadata’/‘Write to Image’ and ‘Files’/‘Metadata’/‘Read from Image’.
If neither of these is set or executed then the metadata will not be read from or written to the image metadata and will remain in the PL5 database, with the original metadata (if any) still intact in the image file. Please note that the initial discovery will perform the equivalent of a ‘Read from Image’ to acquire the image metadata. (Except when a PL4 DOP is encountered and I will write that up in another post!)
But I have been told not to have AS(ON)!:-
By cautious users probably and that potentially includes me!
AS(ON) (Always Synchronize) makes PL5 watch for metadata changes that may occur to an image externally and update the database accordingly, storing certain elements in the DOP. But the other part of the ‘auto’ is that PL5 will also write back any metadata to the image if it changes in PL5 or PL5 decides that it is not in the format that it thinks is correct and changes it!!
The reason for the caution is what happens if PL5 gets things wrong and writes back the metadata and corrupts the image! The only reason for being cautious with PL5 and less so with other DAM software is simply age. PL5 is comparatively new and this particular release has had a way more chequered “career” thus far than previous releases
@uncoy would point to the metadata “distraction” for this “buggy” release and there might be more than an element of truth in that. Personally I would point to DxO simply not publishing the information users needed to verify their current workflow and make amendments (if any) as necessary.
The other reason is that the DAM users in particular do not want PL5 changing their carefully crafted (by their DAM) metadata and writing it back to the image. Even if the image metadata is protected, AS(OFF) and only use ‘Read from Image’ or the new ‘Conflict Resolution’ ‘S’ icon there is still the issue that PL5 will apply its “rules” to the metadata in the images exported from PL5.
But I would rather have the metadata features and suffer the issues that we have suffered than not have the features at all.
But DxO must do better with testing, listening to its users even more and use that userbase to undertake joint testing to improve the standing of DxPL with other metadata handling software and communicate more about the changes that they have made and the implications on the users workflow, which they cannot begin to judge if they haven’t consulted with their users in the first place!!
What AS(ON) does well is effectively “merge” metadata changes from the image (potentially put there by a DAM) or made by the user in PL5. It is not a real merge but PL5 flip flops between reading new data from the image and writing new data to the image whenever there is a change in PL5 or detected in the image!
What are the alternatives if AS(ON) worries me?:-
The answer to that is AS(OFF) but the user is then responsible for instructing PL5 to write to the image file or read from the image file. That is now being helped by the recent ‘Conflict Resolution’ additions by these need to go further.
What problems need resolving;
An option to stop PL5 changing the image metadata format and changing it on the way to an export:-
There needs to be a way that users can insist that PL5 leaves the metadata untouched, i.e. no potential write back and no change to the data output in the export phase.
A Compare option:-
A 'Compare ’ function is really needed because I have encountered situations where PL5 failed to detect an external metadata change! I was unable to isolate the circumstances and could not predictably reproduce the error. This mean that PL5 could miss a change made to the metadata made by the DAM. If all the DAM work is done before presenting to PL5 then there should be no issue but …
The alternative is to execute a ‘Read from Image’ on a selection of images before closing the database etc. but the 'Read from ’ is a “destructive” command, i.e. any (if there are any) changes made in PL5 will be obliterated by the data “coming in” from the image!
Add the missing ‘Conflict Resolution’ icon when changing metadata in PL5 results in a mismatch with the external image data (AS(OFF)):-
Without the missing icon and the icon to show the direction of the mismatch the product is simply incomplete!
…longish post indeed. Having managed IT projects for years, I’d say that you need a summary that fits on one slide…
Give it a try: What are the 5 most important things DxO needs to do, to bring asset management quality up to a reasonable level? Two lines max per item, no reasons to be given.
Without going into all your extensive detail - - I can confirm (with confidence from long-time dependence ONLY on sidecar/.dop files - instead of the database) that ALL correction details are held in the sidecar file associated with each RAW/Image file, and that PL correctly reads and applies all of it.
PL does NOT restore metadata (projects, keywords, ratings) from sidecars - because this information is not stored in sidecar files … but, personally, I don’t care about that because I don’t use those features.
John M