…and I’ll leave it altogether. I don’t consider myself to be in charge of those details. I’m all in for ideas, whatever they might be, but cannot test that Win-stuff 'cause I use Macs only.
In short: Let the Win-persons do the Win-stuff themselves!
Bryan, you are very funny!
@platypus I have done the Windows stuff as you told me I should
now it would be good to have the Mac side of the puzzle!?
I have laid the “facts” out as I see them and it is not a Windows or Mac issue it is a logic issue!
I welcome any ideas on how we can manoeuvre within the scenario that I have laid out to save Project information during a machine and/or drive upgrade because many users are going to face that, albeit only a small number will need to preserve project data.
Without the need to preserve project data I would just ensure that all database data has been written to the DOPs, backup the old database and start afresh with an empty database on the new machine or new disk!
But the tests that I did can easily be replicated on a Mac i.e.
- Create a test directory containing a few images
- Edit so that one or more DOPs are created
- Create a project with some or all of the images
- Move to a target directory from the project and/or from the directory
- Check the data at the new location, sorry I hadn’t included that
The results should be the same regardless of DxPL(Win) or DxPL(Mac). The use of “DB Browser for SQLite” should be possible on a Mac but is optional, however, if used it should be possible to closely compare DxPL(Win) db with DxPL(Mac) db.
Hmm?
On my Mac, I can e.g. move image files with the Finder - while DPL is running - and found that DPL tracks these files and updates the DB accordingly (reported this in a post out there already)
I have not tried to move folders or folder hierarchies though.
@platypus I started with moving directories and moving hierarchies which is something that is not available within DxPL. I tested that with DxPL running and I also tried with DxPL(Win) showing the directory I moved externally.
But I also had DB Browser running at the time and I have had problems in the past either on the closedown or restart (I cannot remember which). This time it crashed DxPL with an error report on the first closedown attempt and the crash status was permanent, i.e. it restarted O,K, but crashed again and again and on closedown!
One new database loaded and I managed to do it again with DB Browser running and it closed and restarted O.k. but not a recommended strategy because it can be unpredictable, certainly with another product with the database open!!
What I did discover is that if you delete or move directories (always with the database open in my case) then there is no path for DxPL to discover and the data remains in the database, i.e. no path to the data equals no discovery by DxPL equals detritus in the database.
If you copy the data back to its original location DxPL now has a path to follow and has entries in the database already and resumes using them! If you change the Uuid of a DOP before moving it back you will get the expected “Unwanted Virtual Copy”, e.g. because you started processing it at its new location but changed your mind and renamed/moved the directory back to its original location.
If you ‘Remove’ with DxPL then all data is removed from disk and from the database.
If you delete images from a directory externally, DxPL still has a path to follow and will clean up its database entries.
As we have seen before if you move/delete the image but leave the DOP the DOP will be deleted on the Mac but will be left on Windows.
Any and all external changes will break the links if any of the images are part of a ‘Project’.
Many thanks to all for taking the time to reply. I want to apologize for taking so long to get back to this, but I had some health issues early in the year, and then my father got ill and passed away, so I just haven’t had time until now to deal with this.
I especially want to thank @platypus!!! Wow, I mean, just…wow! I would never in a million years even think of asking you to do what you did, actually test out the scenario. I’m blown away that you did it! Thank you!
My initial reaction was DXO’s support folks ought to read these forums to, you know, SUPPORT their products! Having experienced their support, and you folks, I still think they ought to, but now I think it’s for a different reason: so they can learn what they ought to know, and see how unpaid folks support others in the community.
In any case, I did these steps:
- installed DPL 6 on my old laptop
- backed up my database to my SSD
- ejected SSD
- connected SSD to new laptop
- restored that database on the new laptop
Fortunately the SSD got the exact same volume name. If it hadn’t, that might have gummed up the works.
I looked at a photo I edited quite a bit on the old laptop on the new laptop, and it looked to me like all the history was there. So I have no reason to believe the history of all the images isn’t there, whew!
I’m now curious about the second part of my query. I have a new SSD that I want to move the images to. As far as I can figure out, there’s no way to move large numbers of images (in folders) to another disk. It only supports copying/moving individual images, correct?
Well, I could do the move outside of DPL 6, of course, and while the ratings and things are kept in the sidecars, the editing history – that I’ve so painstakingly kept intact – is kept exclusively in the db, and moving files outside of DPL 6 will break the connection between the db and the images.
So: DPL 6 & me
I guess my choices are to do the move outside DPL 6 and just lose the history, or I could keep using the old SSD for now hoping a new version of DPL will fix it before I run out of space. If anyone sees another option, I’m all ears.
Thanks again for all the replies,
Brian
@bjnelson Not with DxPL(Win) and I suspect not on MacOS as well! The reason why DxPL is not “confused” by such an occurrence is because it relies on another field (the UUID) and it is that which then throws a spanner into the works with respect to the second part of your query
It is essentially impractical (impossible) to move large numbers of files via the DxPL user interface, thereby retaining the link between the database entries and the newly (re-)located files.
Worse the mechanism that avoids issues with the drive letter now means that it is “impossible” to present the old images on a new disk except as just that, new images, i.e. with the final state of the edit from the DOP but minus the ‘Advanced History’, which isn’t available to Windows users beyond the current session anyway!.
In addition, on both platforms all the project references will now point to “orphaned” database entries. Any other issues relating to the Mac platform I simply do not know about and sadly the procedure I am about to outline works on Windows but may or may not be made to work on MacOS?
So the “glue” that holds the database data to the original data is a structure called ‘Folders’ in the database and this is what it looks like in a test I ran yesterday
Drive T: is a USB3 connected SATA SSD and it contains a folder structure represented by ‘Id’ 6 pointing to ‘Id’ 5 etc. which finally points to ‘Id’ 4 which has a UUID of “270b17f5-0000-0000-0000-100000000000” and is designated as EFDOPVolume, i.e. it looks like this
I also set up a number of ‘Projects’ in DxPL pointing to the images from those directories.
I also had another USB drive connected to the machine that had an identical path structure but that was connected as J: and that drive will have its own unique Device (Volume) Identifier and using Powershell I was able to locate the Device Id for all drives on the system with only T: and J: being of interest for this test
So from your perspective T: represents your current USB drive and J: represents you new, faster, bigger etc. USB drive. You were fortunate to have your images on a USB drive because had they been on an internal drive then …
In my case T: represents my internal drive and J: represents my USB connected backup drive which is essentially a copy of my main drive. In the event of failure it would be good to be able to keep any ‘Projects’ I had created intact and that means I need to be able to “marry” the database complete with the ‘Project’ references to the images on the replacement drive.
Please note that if the database is lost then “simply” presenting the images with their DOPs will ensure that no editing data is lost but the DOP does not contain the ‘Advanced History’ (I believe) nor does it contain the ‘Project’ data!
So using DB Browser (SQLite) I need to replace the Device Identifier for the Device entry in ‘Folders’ with the Device Identifier for the “replacement” drive which must never have been introduced to the database!
- Backup the database, once or twice or … and close DxPL
- Start DB Browser
- Open ‘Folders’
- Copy new value for J: over the old UUID for ‘T:’ using the value from PowerShell in my case
- Paste that entry over the old value (for T:)
- Save changes and we have
- Close DB Browser and start DxPL
- Test for valid ‘Project’ references which all worked fine!
As shown the actual work is not really difficult but as for DxO doing anything about it …!?
Sorry this is the procedure for DxPL(Win).
Regards
Bryan
Editing the database in order to reconnect it to the moved files does not look like something that I’d expect a user to do. Imo, DxO should prestissimo add the necessary functionality to make PhotoLab sustainable.
My approach for such a move at the time being would be to copy everything (image and sidecar files) to the new drive, rename the database (for fallback) and index the new folder structure. This can take some time and you lose history and projects, but it also ensures that the database is void of entries for inexistant files.
@platypus agreed but it depends on how “desperate” the user is to preserve ‘Projects’ and/or ‘Advanced History’ (MacOS only). If I started using ‘Projects’ seriously then I would want to be able to preserve them, they represent an investment that “deserves” to be protected.
For those that have not started using either but want to preserve them going forward, then backup the database and remove it (clear it) and use a ‘Mount Point’ for your principle drive, i.e. last time I tested this successfully I
- Set up a Mount Point on Drive E: (a SATA SSD in my case)
- Mount Drive F: that holds all my Photos on that Mount Point
- Work as normal
- In the event of a failure of my main drive then Mount K: (one of my USB Backup drives) on the Mount Point on E:
- It should be possible to resume work as if nothing has happened.
- But I need to see how to recover if E: fails, which may not be a hackable scenario!!??
But my attempt to use the Mount Point (which I tested a long time ago) just failed so I need to research that a bit more - oops!?
Neither do I but it all depends on how desperate the user is and whether a documented procedure exists!
Of course they should, i.e. add a ‘Replace’ command to the ‘Files’ Menu, preferably actually using the drive letter so the user doesn’t have to locate the Device (Volume) identifier.
One last technique which might work for those wanting to move to a larger drive is to Clone the drive but using software that can preserve the Disk Serial Number but that is not without challenges!
I have used FarStone DriveClone to do just that when backing up my Boot Drive and all had gone well and I bought two new Kingston 480GB SSDs (to add to the two that worked perfectly) and attempted to clone the boot drive but while the process started well is ground to a halt! Then it worked but very slowly but problems emerged on the new machine!
I contacted Kingston and their final conclusion was a change in firmware was responsible for the problem!
So I switched to Silicon Power and the clone started well and then ground to a halt and never finished, so they went straight back and I have reverted to Integral V2 and they seem to work and work faster than any I have tried before!
But operating systems do not like having two drives with the same serial number hence leave the serial numbers alone and hack the database. With database backups you can try it again and again and …!!
Lightroom again: It discovers that the archive can’t be accessed and asks me to locate it. All I then have to do is to navigate to the new folder containing the archive and click okay. Simple as that.
@platypus you are comparing an explicit importation process to an implicit importation process, i.e. in Lightroom you navigate and explicitly choose to import images which then become part of the Catalog. You are then free to browse the Catalog and in doing so Lightroom can inform you of the fact that the related images are now “missing” and offer an opportunity to remedy the situation.
With DxPL the process of navigating is also the implicit process of importing, but if you can’t navigate then you can’t warn (loss of images) and if you discover images that appear to be new (as indicated by the volume identifier) even if the drive letter is the same and the folder structure is identical then you (re-)import the images (potentially orphaning all the old entries).
This latter is the possible reason why some users never encountered the Unwanted Virtual Image situation.
But now we have a problem with when and how can DxO recognise a potential loss or the desire of the user to replace one drive with a new drive.
With the current implicit importation mechanism the answer is never!
Hence, there needs to be a way for the user to indicate that one Volume replaces another, with all the appropriate checks and balances to avoid “accidents”!
PS:-
Unless you add a ‘Reconcile’ function that has been discussed in the past.
How the app puts things in a catalog does not matter. All it takes is something like this:
- User points DPL to a folder (any folder that contains image files or folders with image files)
- DPL checks if something is in the DB and if it’s also on the stored location
→ if yes. do nothing
→ if no, pop up a dialog asking for directions
I don’t care about the number of lines of code this will take. Imo, such operations are necessary to make DPL sustainable and hence move it to a professional level. As of now, we need things that can reliably manage our assets (Lr, C1, PM etc.) … and DPL can’t be the tool of choice.
Imagine working in financial industries and your IT “deletes the database”!
1 Like
It is essentially impractical (impossible) to move large numbers of files via the DxPL user interface, >thereby retaining the link between the database entries and the newly (re-)located files.
Well, so one would think. I thought so too, at first. But then I realized I was making some assumptions about the scope of the problem. So I resolved to figure that out first. What follows will work on OS/X only, although if you’re willing to install Cygwin (which I have in the past) on a Windows system, they MAY work there, too:
To find all the DOP files on my SSD (cd to top level folder of SSD, first):
find . -name ".dop" -exec dirname {} ; | sed 's//.///’
- Ugh. Not good.
Removing duplicate folder names:
find . -name ".dop" -exec dirname {} ; | sed 's//.///’ | uniq
But then I realized that the number of FILES is less important than the number of FOLDERS containing those DOP files. Why? Because it’s the folder that represents ONE copy (or move) operation.
How many folders are there with DOP files:
find . -name ".dop" -exec dirname {} ; | sed 's//.///’ | uniq | wc -l
18 folders. Okay, now THAT I can handle!
Now find all DOP files and print the full path to the parent folder (makes it easier to find each one, as there are some duplicates if I shot images on the same day with 2 different cameras):
find . -name “*.dop” -exec dirname {} ; | uniq
I hope some of this might help someone else. It turns out it WAS reasonable in my case; although I can easily imagine the next time I’ll want to do this, it won’t be.
I had a number of folders where there was just one DOP file, or maybe a couple or a few out of dozens or hundreds of images. I could have sped things up by just copying those images, but it was simpler and less error prone to just do everything in each folder.
The first step was to copy EVERYTHING on the old SSD to the new one. And I mean everything.
So here’s what I did for each folder from the last “find” command above:
- Pointed Finder at new SSD, DxO PhotoLab 6 at old SSD.
- Removed all DPL assets from corresponding new folder in Finder.
- Clicked on old folder on old SSD in DPL; then clicked on first image that appeared.
- Hit CMD/A to select all images.
- Grabbed first selected image and dragged it (and all other images) to new (now empty) folder in DPL.
In my case, I wanted to do a COPY, and that’s what this does. The documentation is either so badly written, or just completely wrong on how to do this. Probably a bit of both. Not sure how to do a MOVE, maybe holding down a key (shift, Option, Command, etc). Usually not holding down a key results in a move operation, so whatever it is, it’s non-standard.
The Advanced editing history is still there. I did notice the Processed checkmark was lost, but that’s not a big deal to me.
Now I have all my images on my new computer, stored on my new SSD. I’m relatively happy. But VERY unhappy with PhotoLab. This is a ridiculous amount of time/effort/energy.
I don’t mind that PhotoLab isn’t a DAM; in fact I’d prefer if they never went there. Luminar was a good editor until they put a DAM in, and then it became unusable. BUT, if a program creates data or metadata, then it HAS to provide a simple, easy, intuitive way to manage those things. I spent more than 35 years building software, very often on customer facing GUIs, and this was one of the mantras I learned. Another was, if the user has to go outside your app to deal with YOUR data, YOU failed.
I’ve downloaded DB Browser and managed to connect to the db. I’m not seeing the same tables as @BHAYT, though. I see a bunch of “ZDOP*” tables, then some “Z_*” tables. I’ll have to spend some more time looking into this, it honestly never occurred to me to dive into someone else’s db.
I agree with the response that said that this is probably not a good idea for most people, though. Even someone who is VERY experienced, knowledgeable, and careful can muck up a db when poking around, especially a db you’re not familiar with. I would probably make TWO backup copies. 
I do appreciate all the replies, though! I had no idea how much I had to learn when I started down this road, all those months ago…
Brian
@bjnelson I am glad you finally resolved the issue but it should have been a simple operation if DxO had bothered to provide the facilities.
However, I am puzzled by the procedure you used? My suggestion had been to copy the old SSD to a new SSD and “hack” the ‘Folders’ structure to make the database/DxPL think that the new SSD was always the SSD it had been working with.
You chose to introduce the new SSD alongside the old and copied the folders contents from one location on the old SSD to another location on the new SSD in DxPL, which proved a little tedious in your case but your analysis gave you a good idea of the size of the problem.
That should work as you found providing the task is not too huge but will it work for ‘Projects’ and I believe that the answer is no because the ‘Projects’ entries will still point to the images on the old SSD. For it wo work then you would need to use a Move and that is a destructive process with some data on the old SSD and some on the new SSD at any given point in time.
The advantage of the “hack” is that it is essentially a non-destructive and can be repeated as many times as necessary if it fails, providing you don’t lose a copy of the original database, or lose your patience/sanity!
The reason for my comments about never allowing the database to “see” the new SSD is that once that is in its ‘Folders’ table it seemed to cause problems when I first tried the technique. I introduced the new SSD to give me an easy route for copying from one UUid to another but that won’t work because you would have two entries in the table. Attempts to avoid the problem on a re-run failed for one reason or another (?) but the technique worked perfectly in the test that I documented in this topic.
I might try some more tests because there is currently no way for a user to preserve ‘Projects’ when using a backup drive to replace a “lost” drive and that is like “adding insult to injury” and a situation I may find myself in!?
From my perspective it is one of the most important changes that DxO should make as a matter of urgency!
The code to fix it is trivial but it needs a bit of UI to introduce the feature, either to detect a potential replacement occurring and offering to use the new drive as a replacement, even DigiKam managed to detect that situation and offered me the opportunity to take the new drive as a replacement (I had replaced a 6TB drive with an 8TB drive since I last used DigiKam).
Alternatively as a new ‘File’ option e.g.
On Windows dragging does a COPY and Shift dragging does a MOVE and neither has a menu option so traversing “long” distances on screen dragging with the mouse is a nightmare.
I am seriously concerned about how many in DxO actually understand the image husbandry side of the product and have ever actually used it in anger?
I wonder sometimes if the reluctance to provide simple enhancements is not because they are focussing on the image editing elements but because they are afraid to touch the image husbandry part of the product!
@bjnelson the Z prefix appears to be a Mac “thing” which with other products seems to be carried over to the Windows database as well but not with DxPL!
I recommend at least two copies somewhere in the forum however DxPL does not use the backup copy directly at all! It copies that database to the appropriate location and starts using that after a restart (arguably it continues with the original database until the actual restart but any updates to that database will be lost after the switch anyway, verified empirically but the old database is not “destroyed” until the actual DxPL restart so there is a window of opportunity available)!
I would never recommend changing the database other than in its default name and in its default location, thereby preserving your original backups.
But I would also recommend copying (saving) the amended database to re-instate if a test almost works and the database needs some more finessing, plus a notepad and pen to make notes of which version contains which “fixes”.
YOU CAN NEVER HAVE TOO MANY BACKUPS, except when you forget which backup contains what!?
There are some things that you must not change with DxPL active and I used to have DxPL dumps when DxPL (re-)started with DB Browser still active but those situations have been diminishing.
The database is actually pretty simple, the keyword and keyword items structures are the most complex but I have an excruciating long description of how that is processed if you ever need it.
I worked with a non SQL database product for 36 years (after a 4 years degree course, 2 years research and 3 years lecturing all in a college that ran a Computing degree course, there were only two such degrees when I started in 1964).
One of my customers had 33 million household records in their database and others had dozens of structures and the DxPL database is not very complicated but the complication comes from having no access to the code and no access to anyone that does have that access. Once upon a time a little insight was provided when DxO were supporting the forum but since DxPL6 there has been none!
All the knowledge about how the system works has to be derived empirically and that still does not provide an insight into the order in which the code is actually executed!
However, just make sure you can always return to a “safe” starting point even if that is a database copy where DxPL has done something stupid or nasty so that you can then try again!
Perhaps but “my” customers had me and those that backed me up so they never went near the code or the database but we taught them enough that they were able to undertake acceptance testing alongside us!
If I wait for DxO to improve the image husbandry side of the product I will be spinning in my grave instead of ranting in this forum!
Take care
Regards
Bryan
@BHAYT , have you tried the following:
Edit the database in order to relate the root of the photo archive to a different drive that has a copy of the photo archive? I suppose that the paths must be identical, which should be no problem, 'cause folder entries in the DB have no UUID (on Mac).
@platypus that is exactly what the above posts Moving Photolab to a new computer - #36 by BHAYT and other posts I have been making for some time are all about, i.e. replacing
with the an identifier from a table like this one
I first encountered the problem here This image cannot be processed since the source file could not be found ( After I cloned the Drive ) and found that for that user ‘Mount Points’ could avoid their particular problem but that does not address the problem of losing a disk and trying to use a replacement backup.
The technique I tested above at Moving Photolab to a new computer - #36 by BHAYT does.
You suddenly seem to have “discovered” this as if it is something new which puzzles me!?
No puzzle. Your posts are simply too long and and carry too much ballast for someone who is not technical. For me, your posts need a summary (without screenshots) 
1 Like
@platypus although not particularly complicated the procedure should “arguably” not be tackled by someone without a little technical understanding and/or a very desperate need.
It is not a particularly difficult task but there are always risks.
With respect to my posts they are long because there is a lot to explain and they are intended to inform a user before they attempt anything potentially “risky”.
I understand the language issue and that may be compounded by snapshots which are “alien” being both in English and possibly for a different OS (and a different user layout etc.) and I have similar problems when they are for the Mac and not in English!
But you did respond to my post in this topic as follows
so you did appear to understand at least some of what was a long post.
Summary of alternatives
-
@bjnelson DxPL Copy between the drives, ‘Advanced History’ appears to have been preserved but ‘Projects’ will be lost (I believe).
-
@bjnelson DxPL Move between drives, ‘Advanced History’ should be preserved and ‘Projects’ should be preserved. No empirical evidence for either at this time and I cannot test ‘Advanced History’ anyway because that is a Mac only feature!
-
The drive designation hack which requires “courage” and DB Browser for SQLite (other products may be available). With careful database backups and drive management the least invasive procedure, the “patient” can be restored to its original state (providing the reason it is being done is not because a drive has “collapsed”) at any time and empirical evidence exists.
-
Just use an empty database and the DOPs if you are not worried about 'Projects; or ‘Advanced History’ (Mac only). This is always the fall pack position for 1 to 3.
I will try to write the step by step summary that you consider will improve understanding of my posts and we can see if that actually works but that will be later because the weather is currently good (maybe a little too hot where I need to work) and some outside woodwork/paintwork needs some attention, when the sun moves off that part of the bungalow.
These are created because the database is actually managed by an ORM (object relational mapping) engine, which generates the database structure from an object model, rather than just defining the tables and columns.
This makes it quite dangerous to tamper with since things like one-to-many relationships are handled inversely.
e.g.
RDBMS
Invoice
Invoice Number
Customer ID
Total
Invoice Line
Invoice ID
Product ID
Quantity
Unit Price
Total
OODB
Invoice
ID
Invoice Number
Customer ID
Invoices Lines (list of Invoice Line IDs)
Total
Invoice Line
ID
Product ID
Quantity
Unit Price
Total
@Joanna thank you for the information.
Typically I use the database as a window into what DxPL is doing and rarely tamper with any data contained in the database.
With respect to the problem at hand the only issue is that the field in question is typically unique amongst the disks on a system, even to the point that the OS may/will object if it has two disks with the same id. active on the system.
Hence, playing with the database is way safer than trying to play with the disks, which I didn’t even include in my list of options in the post above.
Before I recommend or even suggest a strategy I typically test and normally manage to turn up even more problems with DxPL as well as some with my logic, typically by getting a step or two in the wrong order.
The strategy proposed appears to work on DxPL(Win) just fine and there is no logical reason why it shouldn’t (it is pointing to a disk outside the database entirely rather than within the database) but it does require “tinkering” with the database and I would only suggest its use
-
If a disk has been replaced with one of another size (been there but didn’t bother to fix the mismatch)
-
A disk has failed and a backup copy is available.
In all cases the procedure is only required to preserve ‘Projects’ and/or ‘Advanced History’, the edits are preserved by the DOP and can be used to reconstruct parts of the database if and when required.
However, if the technique works then it can bring an exact backup disk into operation very quickly and what worked before in DxPL should work instantly with the copy. The only “minus” is that all the old detritus is still present!