@platypus thank you for testing and with respect to the data I would hope that was the case, with DxPL(Win) it copies the database to the current database location and then “finds” it after a restart!
That means that the edits and metadata (also in the DOPs) and projects (if any), essentially the most important elements of the editing data, are preserved.
That just leaves the following, which actually might not be an issue for @bjnelson
When these are available then DxPL will upgrade/update from one release to another but they typically will not be present on a new machine and the upgrade cannot take place!
Finally there is the question of the new SSD.
I have taken my research a step further by cloning an SSD drive to another SSD and setting the option to keep the disk id. but have “discovered” that the field being captured by DxPL(Win) is the “Mount Volume id” assigned by the Windows operating system
i.e. the Uuid field enables DxPL(Win) to keep the data separate even though it is identical between the two SSDs (with respect to the directories discovered in DxPL).
If I detach P: and attach the clone drive and discover in DxPL then there is no change in the ‘Folders’ structure, i.e. the system and DXPL has accepted the drive as it would the original
So I believe a simple copy will result in the new drive being seen as a new drive and DxPL will simply absorb the images and DOP and create new entries, on Windows in particular but a clone might get away with the “subterfuge” and simply be accepted as the original.
@platypus yes on Windows we can locate a database anywhere but I tend to use the sidebar and it will put the database in the location that I have set for the database in ‘Preferences’, and in the above case this is actually on the default drive (SATA SSD).
My concern with what you tried is whether DxPL(Mac) will pick up the old SSD of @bjnelson without “discovering” the images and any associated DOPs afresh, which believe it will!?
So
When an (or a?) USB drive is attached to a Windows machine it will be assigned a drive letter and may also have a label.
When the user opens DxPL(Win) and navigates to a folder of the newly attached USB drive DxPL needs to know if it has seen that drive before or whether the drive is actually new to DxPL, i.e. should everything that is “discovered” be treated as new and the “new discovery” rules that are now associated with the AS(ON) versus AS(OFF) status will come into play!
Because drive letters and labels can change the deciding factor about a new drive versus an “old acquaintance” is controlled by the Uuid in the ‘Folders’ structure shown in my earlier post, not by the drive letter nor by the label but by the Uuid value.
Hence the “fuss” that I keep making about the Drive (Partition) Uuid. If it changes then DxPL believes that it is dealing with new data and starts populating the database with all the new data that is discovered.
But if it recognises the Uuid as being one in the ‘Folders’ structure then it expects to be able to use all the references in the database, e.g. Projects should point to already discovered images and DOP Uuids should match database item Uuids for each and every image that is being re-discovered by DxPL.
Hopefully discovering the old database should work when the old USB drive is attached.
But if those images and DOPs are copied to another larger, faster USB drive then there will be problems. The new drive will have been allocated a Uuid when created and the copied images and DOPs will be treated by DxPL as “new” because the Uuid of the copy will not match anything in the ‘Folders’ Table.
The consequences
Any and all projects might show a thumbnail but will indicate that the image is missing, the pointers are to the old images which are on the old USB drive (with the old Uuid)!
The old image entries in the database will be side-lined waiting for the old drive to re-appear.
DxPL will happily discover the “new” data associated with the new Uuid and all will appear to be perfect, just that the database will be somewhat larger than it needs to be and any projects won’t work until the old drive is restored.
Hence, my experiments with “cloning” using Farstone (not to be confused with FastStone) which appears to be a Windows only product.
It seems (more testing is required) that a cloned drive has the same Uuid as the original drive, i.e. the system and DxPL can’t tell the difference so DxPL will adopt the new drive as if it was the old one even if twice as big and twice as fast!
If there are no projects then you might as well just copy the images to the new drive (with the DOPs etc.), start with a new database and let DxPL discover the data afresh whenever (if) you visit “old” directories.
DPL on Windows allows a few things that do not work with DPL on macOS.
If we could (on Win) drag the photo archive root to the new drive (possibly while pressing a modifier to make a move instead of a copy), DPL would reconnect the folder(s) to the new location (volume).
Not having a Win computer makes all of the above a guess…that might be worth trying though.
@platypus DxPL(Win) barely wants to copy anything I believe! It will copy images (drag and drop) and move images (Shift + drag & drop) but not directories I believe and that would not help with ‘Projects’ because I don’t believe for one moment that DxO would have built in the necessary fix-up code given that there is no way of recovering project data anyway.
@Wolfgang thanks for the input but that moves images not the directories themselves. Interestingly I tried moving 3 images to a newly formatted USB drive and got
@Wolfgang please explain your post a little because I am unclear as to what you mean exactly?
Providing the user has ensured that all edits and metadata are in the DOPs then the simplest process is to “start again” with a sparkling new database but for users who use projects that could be a “disaster” unless someone writes a utility!
I realise that many laptop users process their images from USB drives, with interfaces that are getting faster and faster, albeit on the very latest laptops. So there is the requirement to be able to move to a new and faster machine and/or new and bigger and faster drives.
In fact if I was using my machine seriously for PhotoLab work, I am in the process of walking into the problem myself because I am rejigging my HDDs moving from a 6TB to an 8TB HDD, with the 6TB then replacing a 4TB as the overflow drive.
I am about to walk right into the drive Uuid “problem” as I try to replace the 6TB drive with the 8TB and cloning is not a realistic prospect for a drive that big and I would lose some of the swing space that I need to accomplish the migration!! Hopefully the 8TB drive will arrive for the Test machine in the next few days but that won’t fix this issue, nothing short of the ability to re-assign the Uuid can fix my problem.
However, my database is transient, projects are (currently) transient so …
With a new database on a “new” drive there is no (urgent) need to re-discover data (i.e. trigger an automatic import) until the user is going to re-edit images and the “only” risk is that there is no backup, to the DOP data in particular, some users destroy the database on a routine basis anyway.
The user should always ensure backups that include DOPs and xmp sidecars as a routine matter.
But I truly lament the lack of written procedures, the lack of additional utilities e.g. to export the projects so that they can be re-assimilated/recovered and the lack of training for support personnel which is beyond …
I have cleared another old SSD so will try a clone without keeping the disk id to see what difference, if any, that makes.
How easy is it to clone on the Mac? I used to have mixed success with various Windows utilities, mostly cloning boot drives but Farstone cloner (Free version) has been ultra reliable for me!
Yes, one has to create the new location for the files (outside or inside PL – I’ve done it both)
and then can move pics inside PL, so that PL keeps track.
When I did so, PL also kept the relation from the source file to its output →
I tried and you cannot move enough to be useful only moving images and having to rebuild the directory structure.
However, I did test and I was wrong because no fix-up is required so the project remains intact but potentially a lot of work is involved and I am not sure, yet, what the workflow would look like to transfer ‘projects’ from one drive to another, I’ll leave that to you @platypus!
But it might look like this
Copy the images from old drive to the new (not creating a clone) and with both attached to the machine at the same time (different Uuids is now an advantage)
Consider that the Images and DOPs on the new drive will be the prime source of the data going forward
Taking each ‘Project’ in turn move the images to the new directory (not sure what happens with the duplicate clash - not tested yet or do the moving of the project images before synchronising the two drives using Beyond Compare in my case)
Tidying up the database by deleting the old “stuff” is beyond tedious because that can only be done by deleting all the images in DxPL and then …
So ‘Items’ has a reference to ‘Sources’ which has a reference to ‘Folders’ and all metadata etc. references ‘Items’, @Wolfgang so when you move within DxPL all the associated data remains intact!
‘Sources’ has a pointer to ‘Folders’ and this is a snapshot after the move and ‘FolderId’ has been changed to 10 (I forgot to capture the ‘Sources’ before image and I am way too lazy to run the tests again)
So an additional row has been added for the ‘Target’ directory which is now referenced by ‘Sources’ (Id 10).
Thus the move of images is accomplished by adding the necessary destination path, and the path elements to get to it, to ‘Folders’ and adjusting the ‘Sources’ references to the new ‘Folders’ entry.
Everything with respect to the item remains intact including the references from ‘Projects’ in fact you can actually move the items contained (referenced) in ‘Projects’ and those images will be moved from their original directory to the target directory and the project entries will remain the same!!
Or in pictures
The original data in situ and selected to make a ‘Project’
…and I’ll leave it altogether. I don’t consider myself to be in charge of those details. I’m all in for ideas, whatever they might be, but cannot test that Win-stuff 'cause I use Macs only.
In short: Let the Win-persons do the Win-stuff themselves!
@platypus I have done the Windows stuff as you told me I should
now it would be good to have the Mac side of the puzzle!?
I have laid the “facts” out as I see them and it is not a Windows or Mac issue it is a logic issue!
I welcome any ideas on how we can manoeuvre within the scenario that I have laid out to save Project information during a machine and/or drive upgrade because many users are going to face that, albeit only a small number will need to preserve project data.
Without the need to preserve project data I would just ensure that all database data has been written to the DOPs, backup the old database and start afresh with an empty database on the new machine or new disk!
But the tests that I did can easily be replicated on a Mac i.e.
Create a test directory containing a few images
Edit so that one or more DOPs are created
Create a project with some or all of the images
Move to a target directory from the project and/or from the directory
Check the data at the new location, sorry I hadn’t included that
The results should be the same regardless of DxPL(Win) or DxPL(Mac). The use of “DB Browser for SQLite” should be possible on a Mac but is optional, however, if used it should be possible to closely compare DxPL(Win) db with DxPL(Mac) db.
On my Mac, I can e.g. move image files with the Finder - while DPL is running - and found that DPL tracks these files and updates the DB accordingly (reported this in a post out there already)
I have not tried to move folders or folder hierarchies though.
@platypus I started with moving directories and moving hierarchies which is something that is not available within DxPL. I tested that with DxPL running and I also tried with DxPL(Win) showing the directory I moved externally.
But I also had DB Browser running at the time and I have had problems in the past either on the closedown or restart (I cannot remember which). This time it crashed DxPL with an error report on the first closedown attempt and the crash status was permanent, i.e. it restarted O,K, but crashed again and again and on closedown!
One new database loaded and I managed to do it again with DB Browser running and it closed and restarted O.k. but not a recommended strategy because it can be unpredictable, certainly with another product with the database open!!
What I did discover is that if you delete or move directories (always with the database open in my case) then there is no path for DxPL to discover and the data remains in the database, i.e. no path to the data equals no discovery by DxPL equals detritus in the database.
If you copy the data back to its original location DxPL now has a path to follow and has entries in the database already and resumes using them! If you change the Uuid of a DOP before moving it back you will get the expected “Unwanted Virtual Copy”, e.g. because you started processing it at its new location but changed your mind and renamed/moved the directory back to its original location.
If you ‘Remove’ with DxPL then all data is removed from disk and from the database.
If you delete images from a directory externally, DxPL still has a path to follow and will clean up its database entries.
As we have seen before if you move/delete the image but leave the DOP the DOP will be deleted on the Mac but will be left on Windows.
Any and all external changes will break the links if any of the images are part of a ‘Project’.
Many thanks to all for taking the time to reply. I want to apologize for taking so long to get back to this, but I had some health issues early in the year, and then my father got ill and passed away, so I just haven’t had time until now to deal with this.
I especially want to thank @platypus!!! Wow, I mean, just…wow! I would never in a million years even think of asking you to do what you did, actually test out the scenario. I’m blown away that you did it! Thank you!
My initial reaction was DXO’s support folks ought to read these forums to, you know, SUPPORT their products! Having experienced their support, and you folks, I still think they ought to, but now I think it’s for a different reason: so they can learn what they ought to know, and see how unpaid folks support others in the community.
In any case, I did these steps:
installed DPL 6 on my old laptop
backed up my database to my SSD
ejected SSD
connected SSD to new laptop
restored that database on the new laptop
Fortunately the SSD got the exact same volume name. If it hadn’t, that might have gummed up the works.
I looked at a photo I edited quite a bit on the old laptop on the new laptop, and it looked to me like all the history was there. So I have no reason to believe the history of all the images isn’t there, whew!
I’m now curious about the second part of my query. I have a new SSD that I want to move the images to. As far as I can figure out, there’s no way to move large numbers of images (in folders) to another disk. It only supports copying/moving individual images, correct?
Well, I could do the move outside of DPL 6, of course, and while the ratings and things are kept in the sidecars, the editing history – that I’ve so painstakingly kept intact – is kept exclusively in the db, and moving files outside of DPL 6 will break the connection between the db and the images.
So: DPL 6 & me
I guess my choices are to do the move outside DPL 6 and just lose the history, or I could keep using the old SSD for now hoping a new version of DPL will fix it before I run out of space. If anyone sees another option, I’m all ears.
@bjnelson Not with DxPL(Win) and I suspect not on MacOS as well! The reason why DxPL is not “confused” by such an occurrence is because it relies on another field (the UUID) and it is that which then throws a spanner into the works with respect to the second part of your query
It is essentially impractical (impossible) to move large numbers of files via the DxPL user interface, thereby retaining the link between the database entries and the newly (re-)located files.
Worse the mechanism that avoids issues with the drive letter now means that it is “impossible” to present the old images on a new disk except as just that, new images, i.e. with the final state of the edit from the DOP but minus the ‘Advanced History’, which isn’t available to Windows users beyond the current session anyway!.
In addition, on both platforms all the project references will now point to “orphaned” database entries. Any other issues relating to the Mac platform I simply do not know about and sadly the procedure I am about to outline works on Windows but may or may not be made to work on MacOS?
So the “glue” that holds the database data to the original data is a structure called ‘Folders’ in the database and this is what it looks like in a test I ran yesterday
Drive T: is a USB3 connected SATA SSD and it contains a folder structure represented by ‘Id’ 6 pointing to ‘Id’ 5 etc. which finally points to ‘Id’ 4 which has a UUID of “270b17f5-0000-0000-0000-100000000000” and is designated as EFDOPVolume, i.e. it looks like this
I also set up a number of ‘Projects’ in DxPL pointing to the images from those directories.
I also had another USB drive connected to the machine that had an identical path structure but that was connected as J: and that drive will have its own unique Device (Volume) Identifier and using Powershell I was able to locate the Device Id for all drives on the system with only T: and J: being of interest for this test
So from your perspective T: represents your current USB drive and J: represents you new, faster, bigger etc. USB drive. You were fortunate to have your images on a USB drive because had they been on an internal drive then …
In my case T: represents my internal drive and J: represents my USB connected backup drive which is essentially a copy of my main drive. In the event of failure it would be good to be able to keep any ‘Projects’ I had created intact and that means I need to be able to “marry” the database complete with the ‘Project’ references to the images on the replacement drive.
Please note that if the database is lost then “simply” presenting the images with their DOPs will ensure that no editing data is lost but the DOP does not contain the ‘Advanced History’ (I believe) nor does it contain the ‘Project’ data!
So using DB Browser (SQLite) I need to replace the Device Identifier for the Device entry in ‘Folders’ with the Device Identifier for the “replacement” drive which must never have been introduced to the database!
Backup the database, once or twice or … and close DxPL
Start DB Browser
Open ‘Folders’
Copy new value for J: over the old UUID for ‘T:’ using the value from PowerShell in my case
Editing the database in order to reconnect it to the moved files does not look like something that I’d expect a user to do. Imo, DxO should prestissimo add the necessary functionality to make PhotoLab sustainable.
My approach for such a move at the time being would be to copy everything (image and sidecar files) to the new drive, rename the database (for fallback) and index the new folder structure. This can take some time and you lose history and projects, but it also ensures that the database is void of entries for inexistant files.
@platypus agreed but it depends on how “desperate” the user is to preserve ‘Projects’ and/or ‘Advanced History’ (MacOS only). If I started using ‘Projects’ seriously then I would want to be able to preserve them, they represent an investment that “deserves” to be protected.
For those that have not started using either but want to preserve them going forward, then backup the database and remove it (clear it) and use a ‘Mount Point’ for your principle drive, i.e. last time I tested this successfully I
Set up a Mount Point on Drive E: (a SATA SSD in my case)
Mount Drive F: that holds all my Photos on that Mount Point
Work as normal
In the event of a failure of my main drive then Mount K: (one of my USB Backup drives) on the Mount Point on E:
It should be possible to resume work as if nothing has happened.
But I need to see how to recover if E: fails, which may not be a hackable scenario!!??
But my attempt to use the Mount Point (which I tested a long time ago) just failed so I need to research that a bit more - oops!?
Neither do I but it all depends on how desperate the user is and whether a documented procedure exists!
Of course they should, i.e. add a ‘Replace’ command to the ‘Files’ Menu, preferably actually using the drive letter so the user doesn’t have to locate the Device (Volume) identifier.
One last technique which might work for those wanting to move to a larger drive is to Clone the drive but using software that can preserve the Disk Serial Number but that is not without challenges!
I have used FarStone DriveClone to do just that when backing up my Boot Drive and all had gone well and I bought two new Kingston 480GB SSDs (to add to the two that worked perfectly) and attempted to clone the boot drive but while the process started well is ground to a halt! Then it worked but very slowly but problems emerged on the new machine!
I contacted Kingston and their final conclusion was a change in firmware was responsible for the problem!
So I switched to Silicon Power and the clone started well and then ground to a halt and never finished, so they went straight back and I have reverted to Integral V2 and they seem to work and work faster than any I have tried before!
But operating systems do not like having two drives with the same serial number hence leave the serial numbers alone and hack the database. With database backups you can try it again and again and …!!
Lightroom again: It discovers that the archive can’t be accessed and asks me to locate it. All I then have to do is to navigate to the new folder containing the archive and click okay. Simple as that.
@platypus you are comparing an explicit importation process to an implicit importation process, i.e. in Lightroom you navigate and explicitly choose to import images which then become part of the Catalog. You are then free to browse the Catalog and in doing so Lightroom can inform you of the fact that the related images are now “missing” and offer an opportunity to remedy the situation.
With DxPL the process of navigating is also the implicit process of importing, but if you can’t navigate then you can’t warn (loss of images) and if you discover images that appear to be new (as indicated by the volume identifier) even if the drive letter is the same and the folder structure is identical then you (re-)import the images (potentially orphaning all the old entries).
This latter is the possible reason why some users never encountered the Unwanted Virtual Image situation.
But now we have a problem with when and how can DxO recognise a potential loss or the desire of the user to replace one drive with a new drive.
With the current implicit importation mechanism the answer is never!
Hence, there needs to be a way for the user to indicate that one Volume replaces another, with all the appropriate checks and balances to avoid “accidents”!
PS:-
Unless you add a ‘Reconcile’ function that has been discussed in the past.
