Moving Photolab to a new computer

DxO PhotoLab
41

How the app puts things in a catalog does not matter. All it takes is something like this:

  1. User points DPL to a folder (any folder that contains image files or folders with image files)
  2. DPL checks if something is in the DB and if it’s also on the stored location
    → if yes. do nothing
    → if no, pop up a dialog asking for directions

I don’t care about the number of lines of code this will take. Imo, such operations are necessary to make DPL sustainable and hence move it to a professional level. As of now, we need things that can reliably manage our assets (Lr, C1, PM etc.) … and DPL can’t be the tool of choice.

Imagine working in financial industries and your IT “deletes the database”!

1 Like
42

It is essentially impractical (impossible) to move large numbers of files via the DxPL user interface, >thereby retaining the link between the database entries and the newly (re-)located files.

Well, so one would think. I thought so too, at first. But then I realized I was making some assumptions about the scope of the problem. So I resolved to figure that out first. What follows will work on OS/X only, although if you’re willing to install Cygwin (which I have in the past) on a Windows system, they MAY work there, too:

To find all the DOP files on my SSD (cd to top level folder of SSD, first):

find . -name ".dop" -exec dirname {} ; | sed 's//.///’

  1. Ugh. Not good.

Removing duplicate folder names:

find . -name ".dop" -exec dirname {} ; | sed 's//.///’ | uniq

But then I realized that the number of FILES is less important than the number of FOLDERS containing those DOP files. Why? Because it’s the folder that represents ONE copy (or move) operation.

How many folders are there with DOP files:

find . -name ".dop" -exec dirname {} ; | sed 's//.///’ | uniq | wc -l

18 folders. Okay, now THAT I can handle!

Now find all DOP files and print the full path to the parent folder (makes it easier to find each one, as there are some duplicates if I shot images on the same day with 2 different cameras):

find . -name “*.dop” -exec dirname {} ; | uniq

I hope some of this might help someone else. It turns out it WAS reasonable in my case; although I can easily imagine the next time I’ll want to do this, it won’t be.

I had a number of folders where there was just one DOP file, or maybe a couple or a few out of dozens or hundreds of images. I could have sped things up by just copying those images, but it was simpler and less error prone to just do everything in each folder.

The first step was to copy EVERYTHING on the old SSD to the new one. And I mean everything.

So here’s what I did for each folder from the last “find” command above:

  1. Pointed Finder at new SSD, DxO PhotoLab 6 at old SSD.
  2. Removed all DPL assets from corresponding new folder in Finder.
  3. Clicked on old folder on old SSD in DPL; then clicked on first image that appeared.
  4. Hit CMD/A to select all images.
  5. Grabbed first selected image and dragged it (and all other images) to new (now empty) folder in DPL.

In my case, I wanted to do a COPY, and that’s what this does. The documentation is either so badly written, or just completely wrong on how to do this. Probably a bit of both. Not sure how to do a MOVE, maybe holding down a key (shift, Option, Command, etc). Usually not holding down a key results in a move operation, so whatever it is, it’s non-standard.

The Advanced editing history is still there. I did notice the Processed checkmark was lost, but that’s not a big deal to me.

Now I have all my images on my new computer, stored on my new SSD. I’m relatively happy. But VERY unhappy with PhotoLab. This is a ridiculous amount of time/effort/energy.

I don’t mind that PhotoLab isn’t a DAM; in fact I’d prefer if they never went there. Luminar was a good editor until they put a DAM in, and then it became unusable. BUT, if a program creates data or metadata, then it HAS to provide a simple, easy, intuitive way to manage those things. I spent more than 35 years building software, very often on customer facing GUIs, and this was one of the mantras I learned. Another was, if the user has to go outside your app to deal with YOUR data, YOU failed.

I’ve downloaded DB Browser and managed to connect to the db. I’m not seeing the same tables as @BHAYT, though. I see a bunch of “ZDOP*” tables, then some “Z_*” tables. I’ll have to spend some more time looking into this, it honestly never occurred to me to dive into someone else’s db.

I agree with the response that said that this is probably not a good idea for most people, though. Even someone who is VERY experienced, knowledgeable, and careful can muck up a db when poking around, especially a db you’re not familiar with. I would probably make TWO backup copies. :slight_smile:

I do appreciate all the replies, though! I had no idea how much I had to learn when I started down this road, all those months ago…

Brian

43

@bjnelson I am glad you finally resolved the issue but it should have been a simple operation if DxO had bothered to provide the facilities.

However, I am puzzled by the procedure you used? My suggestion had been to copy the old SSD to a new SSD and “hack” the ‘Folders’ structure to make the database/DxPL think that the new SSD was always the SSD it had been working with.

You chose to introduce the new SSD alongside the old and copied the folders contents from one location on the old SSD to another location on the new SSD in DxPL, which proved a little tedious in your case but your analysis gave you a good idea of the size of the problem.

That should work as you found providing the task is not too huge but will it work for ‘Projects’ and I believe that the answer is no because the ‘Projects’ entries will still point to the images on the old SSD. For it wo work then you would need to use a Move and that is a destructive process with some data on the old SSD and some on the new SSD at any given point in time.

The advantage of the “hack” is that it is essentially a non-destructive and can be repeated as many times as necessary if it fails, providing you don’t lose a copy of the original database, or lose your patience/sanity!

The reason for my comments about never allowing the database to “see” the new SSD is that once that is in its ‘Folders’ table it seemed to cause problems when I first tried the technique. I introduced the new SSD to give me an easy route for copying from one UUid to another but that won’t work because you would have two entries in the table. Attempts to avoid the problem on a re-run failed for one reason or another (?) but the technique worked perfectly in the test that I documented in this topic.

I might try some more tests because there is currently no way for a user to preserve ‘Projects’ when using a backup drive to replace a “lost” drive and that is like “adding insult to injury” and a situation I may find myself in!?

From my perspective it is one of the most important changes that DxO should make as a matter of urgency!

The code to fix it is trivial but it needs a bit of UI to introduce the feature, either to detect a potential replacement occurring and offering to use the new drive as a replacement, even DigiKam managed to detect that situation and offered me the opportunity to take the new drive as a replacement (I had replaced a 6TB drive with an 8TB drive since I last used DigiKam).

Alternatively as a new ‘File’ option e.g.

image
image227×530 16.2 KB

On Windows dragging does a COPY and Shift dragging does a MOVE and neither has a menu option so traversing “long” distances on screen dragging with the mouse is a nightmare.

I am seriously concerned about how many in DxO actually understand the image husbandry side of the product and have ever actually used it in anger?

I wonder sometimes if the reluctance to provide simple enhancements is not because they are focussing on the image editing elements but because they are afraid to touch the image husbandry part of the product!

@bjnelson the Z prefix appears to be a Mac “thing” which with other products seems to be carried over to the Windows database as well but not with DxPL!

I recommend at least two copies somewhere in the forum however DxPL does not use the backup copy directly at all! It copies that database to the appropriate location and starts using that after a restart (arguably it continues with the original database until the actual restart but any updates to that database will be lost after the switch anyway, verified empirically but the old database is not “destroyed” until the actual DxPL restart so there is a window of opportunity available)!

I would never recommend changing the database other than in its default name and in its default location, thereby preserving your original backups.

But I would also recommend copying (saving) the amended database to re-instate if a test almost works and the database needs some more finessing, plus a notepad and pen to make notes of which version contains which “fixes”.

YOU CAN NEVER HAVE TOO MANY BACKUPS, except when you forget which backup contains what!?

There are some things that you must not change with DxPL active and I used to have DxPL dumps when DxPL (re-)started with DB Browser still active but those situations have been diminishing.

The database is actually pretty simple, the keyword and keyword items structures are the most complex but I have an excruciating long description of how that is processed if you ever need it.

I worked with a non SQL database product for 36 years (after a 4 years degree course, 2 years research and 3 years lecturing all in a college that ran a Computing degree course, there were only two such degrees when I started in 1964).

One of my customers had 33 million household records in their database and others had dozens of structures and the DxPL database is not very complicated but the complication comes from having no access to the code and no access to anyone that does have that access. Once upon a time a little insight was provided when DxO were supporting the forum but since DxPL6 there has been none!

All the knowledge about how the system works has to be derived empirically and that still does not provide an insight into the order in which the code is actually executed!

However, just make sure you can always return to a “safe” starting point even if that is a database copy where DxPL has done something stupid or nasty so that you can then try again!

Perhaps but “my” customers had me and those that backed me up so they never went near the code or the database but we taught them enough that they were able to undertake acceptance testing alongside us!

If I wait for DxO to improve the image husbandry side of the product I will be spinning in my grave instead of ranting in this forum!

Take care

Regards

Bryan

44

@BHAYT , have you tried the following:

Edit the database in order to relate the root of the photo archive to a different drive that has a copy of the photo archive? I suppose that the paths must be identical, which should be no problem, 'cause folder entries in the DB have no UUID (on Mac).

45

@platypus that is exactly what the above posts Moving Photolab to a new computer - #36 by BHAYT and other posts I have been making for some time are all about, i.e. replacing

image
image1689×200 20.9 KB

with the an identifier from a table like this one

image
image713×657 110 KB

I first encountered the problem here This image cannot be processed since the source file could not be found ( After I cloned the Drive ) and found that for that user ‘Mount Points’ could avoid their particular problem but that does not address the problem of losing a disk and trying to use a replacement backup.

The technique I tested above at Moving Photolab to a new computer - #36 by BHAYT does.

You suddenly seem to have “discovered” this as if it is something new which puzzles me!?

46

No puzzle. Your posts are simply too long and and carry too much ballast for someone who is not technical. For me, your posts need a summary (without screenshots) :wink:

1 Like
47

@platypus although not particularly complicated the procedure should “arguably” not be tackled by someone without a little technical understanding and/or a very desperate need.

It is not a particularly difficult task but there are always risks.

With respect to my posts they are long because there is a lot to explain and they are intended to inform a user before they attempt anything potentially “risky”.

I understand the language issue and that may be compounded by snapshots which are “alien” being both in English and possibly for a different OS (and a different user layout etc.) and I have similar problems when they are for the Mac and not in English!

But you did respond to my post in this topic as follows

so you did appear to understand at least some of what was a long post.

Summary of alternatives

  1. @bjnelson DxPL Copy between the drives, ‘Advanced History’ appears to have been preserved but ‘Projects’ will be lost (I believe).

  2. @bjnelson DxPL Move between drives, ‘Advanced History’ should be preserved and ‘Projects’ should be preserved. No empirical evidence for either at this time and I cannot test ‘Advanced History’ anyway because that is a Mac only feature!

  3. The drive designation hack which requires “courage” and DB Browser for SQLite (other products may be available). With careful database backups and drive management the least invasive procedure, the “patient” can be restored to its original state (providing the reason it is being done is not because a drive has “collapsed”) at any time and empirical evidence exists.

  4. Just use an empty database and the DOPs if you are not worried about 'Projects; or ‘Advanced History’ (Mac only). This is always the fall pack position for 1 to 3.

I will try to write the step by step summary that you consider will improve understanding of my posts and we can see if that actually works but that will be later because the weather is currently good (maybe a little too hot where I need to work) and some outside woodwork/paintwork needs some attention, when the sun moves off that part of the bungalow.

48

These are created because the database is actually managed by an ORM (object relational mapping) engine, which generates the database structure from an object model, rather than just defining the tables and columns.

This makes it quite dangerous to tamper with since things like one-to-many relationships are handled inversely.

e.g.

RDBMS

Invoice

Invoice Number
Customer ID
Total

Invoice Line

Invoice ID
Product ID
Quantity
Unit Price
Total

OODB

Invoice

ID
Invoice Number
Customer ID
Invoices Lines (list of Invoice Line IDs)
Total

Invoice Line

ID
Product ID
Quantity
Unit Price
Total
49

@Joanna thank you for the information.

Typically I use the database as a window into what DxPL is doing and rarely tamper with any data contained in the database.

With respect to the problem at hand the only issue is that the field in question is typically unique amongst the disks on a system, even to the point that the OS may/will object if it has two disks with the same id. active on the system.

Hence, playing with the database is way safer than trying to play with the disks, which I didn’t even include in my list of options in the post above.

Before I recommend or even suggest a strategy I typically test and normally manage to turn up even more problems with DxPL as well as some with my logic, typically by getting a step or two in the wrong order.

The strategy proposed appears to work on DxPL(Win) just fine and there is no logical reason why it shouldn’t (it is pointing to a disk outside the database entirely rather than within the database) but it does require “tinkering” with the database and I would only suggest its use

  1. If a disk has been replaced with one of another size (been there but didn’t bother to fix the mismatch)

  2. A disk has failed and a backup copy is available.

In all cases the procedure is only required to preserve ‘Projects’ and/or ‘Advanced History’, the edits are preserved by the DOP and can be used to reconstruct parts of the database if and when required.

However, if the technique works then it can bring an exact backup disk into operation very quickly and what worked before in DxPL should work instantly with the copy. The only “minus” is that all the old detritus is still present!