Why can I not just delete the [M]aster copy and leave any other VCs (Virtual Copies) intact!?

BHAYT · August 12, 2024, 4:13pm

@platypus the answer is yes and yes but actually the basic coding, i.e. minus all the good checks, proper timestamps etc. etc. is simple enough and currently I have the following for a DOP from another user

So ahead of the first “Albums” entry is the header and the trailer is between the last but one “Uuid” entry (Line 1105) and the end of file and includes the last “Uuid” shown at line 1110

As far as I am concerned the lines between the “Album” and associated “Uuid” entry are the details for the copy, [M] or VC or … and it should be possible to do a number of things with that data, allegedly!!

@Joanna As do I and many others but I am trying to see how close I can get to a working “Unwanted VC” fix as possible. My ideal fix is an intact database with intact ‘Projects’ and both copies but reversed and I have managed that manually a number of times until I didn’t for some reason(s)!?

Plus I want to avoid going anywhere near the database if possible, other than automatically backing it up, along with the original DOPs and sidecar files etc.

The task of doing the job by hand is error prone, hence the coding which is easy to fix and try again and again and …

Its a hot day here in the South of the U.K., I still have 4 rack bolts to fix but only mad dogs and Englishmen go out in the midday sun, I have returned to some coding!

RexBlock · August 15, 2024, 7:21pm

@BHAYT What are you using to get that schema analysis?

BHAYT · August 15, 2024, 8:35pm

@RexBlock Basically my eyes, i.e. countless tests and inspections of the resulting DOPs.

The analysis may or may not be completely accurate, for Macs in particular. with respect to the ordering of the copies in the DOP image, i.e. I have assumed that the order in the DOP was the same as the order in the database, i.e. [M]aster first, then VC[1} then etc etc, but this might not be the case for Macs and possibly not for Windows!?

I have written a program that shows the following for a Mac DOP is particular

and looking for patterns

Does this help!?

PS:- The program makes no assumptions about what it will find, it reads each line of each DOP and reports on “lines of interest” as they are encountered, i.e. a line with “Albums”, a line with “Dates”, then split into “Creation Date” and “Modification date”, and a line with “Uuid”.

As I wrote, a copy (exactly which copy may be in doubt?) exists starting with “Albums” and ending with the “Uuid” for the copy, and then repeats again and again with 1 entry for [M] and then entries for VC[1], VC[2] etc.

An additional “Uuid” is part of the “Trailer” as I called it and I need to see if that points to the “Uuid” of the “Sources” entry, which I once thought it did but haven’t bothered to confirm recently. Regardless of what it refers to it exists and I believe that DxPL will expect such an entry in the “Trailer” of any DOP.

However, there are issues with my analysis because it is purely empirical and while I have succeeded with my tests getting the desired results from time to time I have also failed with my hand created DOPs.

I am close to being able to create a new DOP from an existing DOP, e.g. a [M]aster only DOP, a VC[1] only DOP programmatically, i.e. the errors will be made in a program which can be “easily” rectified and tested again and again quickly instead of putting the DOP together by hand and finding it doesn’t work because of some stupid mistake!

PPS:- the database schema is available via “DB Browser for SQLite” available for Mac and Windows and opening the database gives

on Windows .

RexBlock · August 16, 2024, 2:08am

It does, but I’m on holiday and I need to return home and clear the holiday brain before digging into this more deeply. This is too much to entertain while listening to the crickets and owls.

My curiosity was piqued earlier in this thread with a couple of screen shots that came out of a DB analysis tool. That was my paying job back in the day and it reminded me of a tool I once used called Toad. It was for Oracle RDBMSs but I’m sure there are similar tools for MySQL DBs.

I was wondering if you managed to open the DxO MySQL DB with the tool, and what tool it was. Thanks, your work is quite enlightening.

BHAYT · August 16, 2024, 7:11am

@RexBlock Enjoy the rest of your holiday.

Having started as a Database Consultant for Burroughs (which became part of Unisys) and working for them for 36 years with various job titles, mostly involving a product called DMS II, I should have realised that you were referring to a database schema rather than the DOP “schema”.

So the product of choice for the PhotoLab database is “DB Browser for SQLite” available for Windows and Mac (and Linux etc.) and free (or Patreon sponsorship). It provides access to PhotoLab DB and gives an insight into what DxO are doing with the database.

I decided to get back into programming a couple of years ago and started to learn Python but stopped for a bit and then developed my own Watchdog program (built around someone else’s core code, of course), before discovering PureBasic (PB) which I am using to further investigate the “ways” of the DOP.

I have written the basis for a program in PB to “swap” drives which allowed me to start amending the PhotoLab database, albeit only the ‘Folders’ structure in Windows PhotoLab.db,. There appear to be fairly substantial differences between the DxPL(Win) database and the DxPL(Mac) databases.

That program needs finishing before it can be used for an “emergency” disk replacement.

The program that I used to investigate the contents of the DOP is useful and I have started to modify that to provide the ability to create new DOPs prior to investigating swapping the [M]aster entry with VC[1], or rather extracting the DOP element for the [M]aster and VC [1] etc. and rebuilding a “replacement” DOP with the data reversed.

I need to verify that the order of DOP data placement is as I originally thought, on Windows at least, with the order of Copy data within the DOP being the same order as we see in the display or by determining what strategy DxO uses for extracting the data from a DOP to reflect the order in the display.

platypus · August 16, 2024, 7:24am

PhotoLab for macOS creates dop files in which the section describing the master’s customising is in any seemingly random position in relation to the virtual copies.

The master and the copies cannot be identified directly by a special entry, but by the creation timestamp. only. We could also use manually added data like colour tags or changed file names, but I consider this to be less practical and reliable in comparison to a timestamp written by PhotoLab.

Deleting the section of the master from a dop file will “elevate” the remaining oldest entry to “master”.

Note that some of the above was discovered and verified in cooperation, via private messages, with @BHAYT .

gserim · August 16, 2024, 8:01am

Interestingly, delete and remove are used in the same context. Delete rather means “put in the trash” while remove implies complete elimination. Also, the dialogs says that only virtual copies are deleted from projects . Mhhh, all rather inconsistent.

BHAYT · August 16, 2024, 8:04am

@platypus Thank you for the update and the technique of colour Tags + Rating + IPTC field would help to verify that the timestamp, which requires no special action by any user, is doing what you believe it is and logically the [M]aster exists before VC[1] which exists before VC[2] etc. so the ‘Creation timestamp’ should be correct.

I need to verify that works for Windows and understand the implications for any DOP Copy swap strategy.

But that needs to wait a while, we are off to Hastings to see one of our granddaughters taking part in “Charlie and the Chocolate Factory”, a bit part but you have to start somewhere.