PL7.0.2(Win) Editing one collection of images from 2 computers each with their own DxPL Database

BHAYT · December 6, 2023, 7:06pm

The collection of images referred to in the title could be on a NAS accessed by the two computers or being stored on one computer and accessed by the other via a LAN. This testing was done with PL7.0.2 but some tests were repeated today with PL7.1.0.

The summary of the tests and the outcome are as follows:-

Two computers accessing one directory of images but only one computer at a time, this seems to work fine with DxPL(Win), according to my tests, providing only one computer has the directory open at any one time.
Two computers accessing the same directory of images at the same time, this rapidly turns into chaos because of the way the two copies of DxPL interact when a change is made by the other computer. If the file is a JPG and a keyword is added then a keyword added to a Virtual Copy could wind up being written to the image, according to tests I ran some time ago and reported to DxO Support and reran here.

Editing a single directory of files on two different systems:-

I started editing images on my i7-4790K, equipped with a 1050Ti GPU and then decided to export the images using a 5600G with an RTX 3060 to improve (reduce) the export times.

Although I could have copied the images between machines I chose to access the images in situ and accessed them via the LAN, similar to using a NAS drive.

So on the i7 I left PL7.0.2 running but “parked” on an empty directory and navigated to the images using PL7.0.2 on the 5600G and I exported the images successfully with no issues.

But I then noticed “pumpkins” in certain images (part of the Halloween pumpkin trail at Highdown Gardens) and decided to use DxPL to remove them. I started on the 5600G using VCs to “preserve” the original images and did the edits on the 5600G.

I then “parked” the 5600G DxPL on an empty directory and continued to edit the images on the i7.

The VC edits were in place on the i7 from the 5600G editing (there needs to be a delay of 20 seconds or so between making edits on the second system to ensure that the DOPs have been written to disk) and I continued further ‘Retouching’ on the i7, navigating away from the directory to a “neutral” directory (preferably empty of images) before re-discovering the directory on the 5600G.

Providing care is taken to ensure that only one machine has a directory open at any one time there appears to be no problem with “flip-flopping” between the machines with respect to image editing.

So for edits it appears to be possible to have two machines working with and updating image edits and metadata at the same time providing each machine is accessing a different directory or, if updating the same directory, only one machine has the directory open at any one time.

Both machines must never be accessing the same directory at the same time.

During the tests I monitored the directory using “Folder Monitor”, a free product available for Windows at https://www.nodesoft.com/foldermonitorand which uses .NET.

Folder Monitor submits a request to the Operating System to be informed of any file events for a designated directory (it is possible to watch more than one directory with FM, including subdirectories), the program then waits on an event that occurs for any of the watched directories.

In the case of FM the action taken when an event occurs is to report it.

DxPL almost certainly uses essentially the same mechanism to be informed that an event has occurred to the images in the directory of images on view in the product. DxPL then examines the event and the state of the files involved in that event and takes appropriate action which will be reflected in the application, database and the DOP.

One primary use for this mechanism is to detect when another program has made a metadata change to an image in view, changed ‘Rating’, added a keyword etc…

With files mounted on a NAS or accessed via a LAN the events FM monitors are frequently duplicated in the logs, i.e. an edit update made in DxPL that causes a DOP update in a local directory looks like this in the FM Log

Edit on local directory:-
2023-12-05 19:41:40.006: F:\___Beta DXO PL5 - Tests\Test 17 - Directory Name change (small directories)\Test 17 - Run 01\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-05 19:41:40.183: F:\___Beta DXO PL5 - Tests\Test 17 - Directory Name change (small directories)\Test 17 - Run 01\Test 17 - 01\P1107710.JPG.dop created
2023-12-05 19:41:40.241: F:\___Beta DXO PL5 - Tests\Test 17 - Directory Name change (small directories)\Test 17 - Run 01\Test 17 - 01\P1107710.JPG.dop modified

but like this when the file is accessed via a LAN

Edit in NAS Directory:-

2023-11-27 18:00:14.521: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 05\P1107801.JPG.dop deleted
2023-11-27 18:00:14.615: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 05\P1107801.JPG.dop deleted
2023-11-27 18:00:14.616: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 05\P1107801.JPG.dop created
2023-11-27 18:00:14.616: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 05\P1107801.JPG.dop modified

The key “giveaway” is two DOP delete events when the file is mounted on a NAS or accessed via a LAN. Sometimes there are also two file create events, i.e. how can you delete a file more than once or create a file more than once?

However, I created a Python script to do similar monitoring using a Python WatchDog library and got the same issues, as shown below!?

Python File Monitor (PFM):-


Edit in NAS Directory:-
2023-11-27 18:00:14.529424 - FILE - Y:\TEST\PHOTOS\TEST 17 - DIRECTORY NAME CHANGE (SMALL DIRECTORIES)\TEST 17 - RUN 01 - PL670\Test 17 - 05\P1107801.JPG.dop - FILE - deleted
2023-11-27 18:00:14.547417 - FILE - Y:\TEST\PHOTOS\TEST 17 - DIRECTORY NAME CHANGE (SMALL DIRECTORIES)\TEST 17 - RUN 01 - PL670\Test 17 - 05\P1107801.JPG.dop - FILE - deleted
2023-11-27 18:00:14.565411 - FILE - Y:\TEST\PHOTOS\TEST 17 - DIRECTORY NAME CHANGE (SMALL DIRECTORIES)\TEST 17 - RUN 01 - PL670\Test 17 - 05\P1107801.JPG.dop - FILE - created
2023-11-27 18:00:14.577408 - FILE - Y:\TEST\PHOTOS\TEST 17 - DIRECTORY NAME CHANGE (SMALL DIRECTORIES)\TEST 17 - RUN 01 - PL670\Test 17 - 05\P1107801.JPG.dop - FILE - modified

I also ran tests updating two adjacent directories on the NAS which worked fine until I deliberately moved DxPL on one of the machines to the same directory as the other machine and when it came time to write a DOP all hell broke loose.

I will add details of the update tests and the problems when two machines are accessing the same directory with DxPL in a subsequent post.

uncoy · December 7, 2023, 12:20am

Fascinating test, Bryan. DxO should be sponsoring your stress tests. Good warning to DxO users. PhotoLab is basically a single user system. I didn’t expect anything different but it’s good to see it tested.

BHAYT · December 7, 2023, 12:48am

@uncoy

Actually its a multi-user system, albeit they will tell you not to do that, but if you avoid using the same directory at the same time then I believe that nothing nasty will happen.

When I complete the second post (properly) you will see what happens if you break that “rule” and I believe it could be avoided but what can and should be avoided is the situation whereby a VC keyword ended up in the JPG image as happened to me, I reported it and managed to reproduce it.

With respect to the main problem, effectively the change made by one system causes a new DOP and the other system reacts and changes its database and writes a new DOP which causes the first system to react and change its database and writes a new DOP which causes … and so it goes on.

The fact is that the DOPs are arguably not changing materially (in this particular case), probably just the ‘Modified timestamp’ of the file and possibly the modification date in the DOP. With a bit of clever coding one system might be able to decide that the DOP update is not a material change and ignore the event and the endless chain is broken.

One slight problem with that with DxPL(Win) is the random order of the DOP which might make actually determining that it is essentially the same DOP as last time difficult to determine.

But if that could be done then any number of users could pitch in, even using the same directory without causing a runaway chain of events, but there certainly could be chaos with real edit updates clashing!

However, whether anyone wants to use the product that way is an entirely different question but I like to know the limits of what is possible and define real boundaries.

platypus · December 7, 2023, 3:33am

What happens when both computers access the same folder simultaneously? Imo, it should work too because OSs and file systems have evolved and should be able to handle near synchronous access.

BHAYT · December 7, 2023, 8:24am

@platypus

This is what happens, the annotations to the log are put there by me, Folder Monitor releases the log after each event log so it can then be edited and gaps entered between events. My “own” Python Folder Monitor (using “borrowed” code for the event capture) had to be modified to close the log after each event so that I could put the annotations into the log.

Switching i7 to the same directory as the 5600G:-

Nothing until a DOP update was done on one of the machines (probably the i7) and then this:-

2023-12-06 10:21:57.573: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:21:57.680: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:21:57.680: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop created
2023-12-06 10:21:57.680: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.216: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG modified
2023-12-06 10:22:01.237: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG modified
2023-12-06 10:22:01.279: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:22:01.338: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:22:01.338: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop created
2023-12-06 10:22:01.363: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop created
2023-12-06 10:22:01.363: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.364: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.365: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.369: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.369: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.369: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.400: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:22:01.400: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:22:01.402: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:22:01.412: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop created
2023-12-06 10:22:01.413: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop created
2023-12-06 10:22:01.413: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.416: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.449: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:01.450: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:05.616: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted
2023-12-06 10:22:05.642: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted

and it just continues for as long as the two machines remain “fishing in the same pond”!

But looking at the entries in the “Folder Monitor” log I found this

2023-12-06 10:22:13.172: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:13.173: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop modified
2023-12-06 10:22:16.548: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG modified
2023-12-06 10:22:16.569: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG modified
2023-12-06 10:22:16.610: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted

and arguably even worse this

2023-12-06 10:22:28.417: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 02\P1107754.JPG modified
2023-12-06 10:22:28.484: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 02\P1107754.JPG modified
2023-12-06 10:22:28.870: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 02\P1107754.xmp modified
2023-12-06 10:22:28.900: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 02\P1107754.xmp modified
2023-12-06 10:22:29.031: Y:\Test\Photos\Test 17 - Directory Name change (small directories)\Test 17 - Run 01 - PL670\Test 17 - 01\P1107710.JPG.dop deleted

There was this

Run 01 - PL670\Test 17 - 01\P1107710.JPG modified
Run 01 - PL670\Test 17 - 01\P1107710.JPG modified

in the early part of the first log section I showed and that might be legitimate (?), i.e. because the i7 was then seeing the directory for the first time.

But the second update occurs from one system or the other after a “flurry” of DOP activity and why the xmp sidecar of the RAW image should be touched at all indicates that there is some “suspect” coding in the DxPL(Win).

I believe that “rogue” updates should be tracked down and eradicated @DxO_Support-Team but that still leaves the issue of breaking the game of “ping-pong” between the two machines that needs to be detected and stopped!

Plus my comments about the risks of two users attempting to edit the same image are unlikely, it is more likely that one user uses two pieces of hardware to edit same pool of images and that user should then know what they are doing, or do I!?

PS:-

I need to do more investigating because the images in the second incident, P1107754.JPG, were being used by the 5600G and I now think that I switched the 5600G to use the directory containing P1107710.JPG which started the run-away so why is P1107754.JPG even in the log at all!!

Sorry I will retest and see what I get but the runaway always occurs whenever two machines “share” the same directory.

platypus · December 7, 2023, 9:03am

Sorry @BHAYT , I don’t speak log. You might have recited from the Mahayana in Sanskrit with similar success.

BHAYT · December 7, 2023, 9:04am

@platypus I apologise for my confusion so a repeat of the tests produced this

The i7 start directory

The 5600G start directory

I switched the 5600G to the same directory as the i7 and the runaway shown in the attached log ensued.

It contains JPG modifications (!), xmp sidecar modifications (!) amongst others but all for images from the same directory that both machines are now accessing i.e. Test 17 - 04.

FolderMonitor 2023-12-07_01.txt (38.2 KB)

PS:- the log is simply a list of events that Folder Monitor is watching, it is set to watch the folder on the NAS drive (mapped as the Y: drive on my machines) that contains

PL7.1.0 will have a similar “watch” set up and will be seeing the same events to which it may or may not respond based on its programming.

George · December 7, 2023, 9:05am

Like with every other use of a nas with an editing software, including databases, the edited subject must be locked, not accessible by another program/user.

George

platypus · December 7, 2023, 9:07am

An image can say more than a thousand words…please try words…and I suppose that less than a thousand should suffice.

BHAYT · December 7, 2023, 9:13am

The NAS is simply a collection of files, the main use for a NAS is accessibility not impeded by “locks”. If a set of files was locked then other programs that change metadata etc. could not access them.

Each copy of DxPL in the tests is using own database but they happen to be accessing a common source of images which will include DOP sidecar files and xmp sidecar files.

BHAYT · December 7, 2023, 9:17am

@platypus this is the consequences of me setting the Tag in DxPL 7.1

DxPL “deletes” the existing DOP, flagged twice because of the vagaries of monitoring a shared resource.

DxPL then creates a new DOP (you sometimes get two of those as well with NAS)

DxPL then modifies the DOP, which it couldn’t have done all the work before creating the DOP you would need to ask DxO!

platypus · December 7, 2023, 9:17am

This should be no problem with RAW files, they are only read by DPL. There could be a problem with one DPL overwriting the other’s sidecar file(s) though.

Anyways, editing the same files with two apps doesn’t make much sense to me, except when two different apps or app versions are used. Well, I think I’ll pass with this topic.

BHAYT · December 7, 2023, 9:19am

As when another program changes the metadata which DxPL will detect using the same mechanism.

You are welcome!

platypus · December 7, 2023, 9:31am

I often use DPL and LRC with the same set of local images on my Mac(s) and never had issues with that. BUT: DPL and LrC are used for different things and neither of the two apps understands the other’s settings - and I don’t auto-sync sidecars. I suppose that this setup is okay.

Using DPL in different versions works. Tried it with images on iCloud too, and again, no issues, I suppose that it has to do with the local copies that are actually used.

BHAYT · December 7, 2023, 9:43am

@platypus The reason I went down this path was because of a post by you at PhotoLab 6 on 2 Computers and files on a NAS

When I was testing that I accidentally focussed two copies of DxPL at the same NAS directory and wound up with data from a VC being written to a JPG image.

I reported that to DxO Support and was informed that I shouldn’t be accessing the same directory with two systems and that it was “expected” behaviour !

While the runaway interaction might be expected behaviour the “corruption” of data is not and should not occur!

Which is why you won’t necessarily run into the same problem unless or until you do sync if the other system is still “watching” the directory at that time, which is probably unlikely. It is the occurrence of a new or updated DOP which triggers the runaway.

As my tests have shown me it is possible to access from two systems, successfully, providing sufficient time is allowed for DOPs to be written to disk and providing both systems are not trying to access the same directory at the same time accessing or DOPs are not being written automatically!

George · December 7, 2023, 10:53am

for a NAS is accessibility not impeded by “locks”. If a set of files was locked then other programs that change metadata etc. could not access them.

That is a must when more then one user can edit.

george

BHAYT · December 7, 2023, 11:24am

@George

I deliberately left out the notion of two users using DxPL at the same time because that might be a violation of the licence, albeit I then discussed the problems that might ensue if users had an unrestricted access and made edits to the same image at the same time!

I tested this not in that context but rather from the perspective that when I started this my i7 was equipped with a GTX 1050Ti versus my 5600G with an RTX 3060 and moving from one machine to another to continue edits and particularly to export on the 5600G was a real prospect.

Since then I have changed the 1050Ti for an Ebay purchased RTX 2060 and the need for switching has reduced somewhat. I actually don’t use the NAS for my image collection because it is not particularly fast, I use LAN access with mapped drives between the systems.

Nearly all the products I have tested for keywording lock the image for as little time as possible to allow for other products picking up that data as quickly as possible!

So while you might be right about the need for locking when multiple users are accessing a data store mine are definitely not configured that way and neither are the products I use.

Regards

Bryan

George · December 7, 2023, 1:30pm

That’s exactly why it should be locked. Only one can do editing. As far as I know every database is working like that. Even my text editor Open Office. If you just want to view the records there is no problem with multi users.

George

The former post was send from my phone. I forgot that.

BHAYT · December 7, 2023, 2:41pm

@George

DxPL’s database is the same as most other products, ACDSee uses a DBase variant, but the rest all use SQLite and the databases are essentially single user but in my test case there are two databases in use with just one set of images and it is the images and their “hangers on”, sidecar files, that are being “shared”.

So DxPL is not better nor worse than every other product which uses SQLite.

I did not mark this topic as a “bug” deliberately, I consider the way is works to be fine for casual use from more than one computer providing the same directory is not accessed by more than one machine.

I feel that the runaway that happens when two machines access the same directory might be fixable but that might prove too complicated.

However, I am concerned about the corruption that occurred during my original test and repeated more recently, plus the JPG and xmp sidecar modifications that “popped” up during the runaway, to be an error, regardless of how it is caused, that needs attention.

Joanna · December 7, 2023, 4:11pm

Don’t forget that much code is run on multiple threads, asynchronously. Multi-user software has to be explicitly designed with locks and conflict resolution dialogs. Something that DxO doesn’t do because the design and maintenance can be an absolute nightmare.

This also involves having to have monitoring threads active on any folder, to determine when a file is touched, or is being accessed, which tends to slow down CPU activity.