Disable virtual copies at all!

[Kit]

@Joanna

Which prompts me to ask why on earth you are using Windows instead of Unix-based Mac?

It’s easy to answer, i make my money as a Windows Admin & System Engineer, so i have to work on the system that pays me. I know about the issues of Windows, but hey…they pay me the most.

And…my photos are on a 70TB Synology NAS, so they’re on Linux/Unix though. And i guess exactly this causes the problem as on that NAS for whatever reasons file date gets changed on some folders regularly. PL runs on my Windows workstation and access my pics via network on the NAS.

@Joanna

So, buy a bigger disk - they’re cheap enough

Nope, i won’t spend money on unlimited backups bcs an app is buggy and the vendor refuses to do changes. I’ll rather change the app ! It would be so easy to give users the option to disable that mostly unwanted VC-“feature”…

@Joanna

That works out at around 274 images per day, every day, 365 days per year, for 20 years.

It’s not on you to consider if my pics are important or perfect or not to me. That was also not my question and i surely didn’t want a comment on this.

@Joanna

The system and software doesn’t delete anything without your say so. It is well documented that deleting the ‘M’ copy is destructive. Avoid that and you will never lose photos.

I didn’t say that. I did delete some pics myself accidently bcs it was sorted by VC number, i marked the numbers and there were still some “M” in between which i have overseen.

And i recognized it only when i tried to find exactly those pics the next time and it was after far more then my 2 weeks backups period, so they were lost.

The fact is: I MYSELF DID NEVER CREATE ANY VC AND EVEN NEVER WANTED THEM TO BE CREATED ! So it was the app with that for me useless feature that brought me to delete files which i never had done without…exactly…the unwanted VCs…

So i don’t need any advices how to set up my IT or, how long i should keep my backups (2 weeks is a kinda standard btw.) or if i consider 200k pics important or not, but i (and it seems many others too!) need simply a solution which prevents that “feature” to do it’s confusing and destructive work…

And that’s what i kindly ask here. And if the answer is no, PL6.x was the last i bought. Quite easy…

[noname (Captain Obvious)]

@Kit

i use it but it’s anyway deleted after 2 weeks.

you need to change your versioning habits

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@asvensson Then try the following which “simulates” the behaviour that was occurring a lot when users took edited images from a laptop to a desktop and then back to the laptop (round tripping and image).

I have done more recent tests on that scenario and the DOP made the journey with the same UUID as it was originally allocated and no “Unwanted VC” was created.

But the following was how I determined exactly what was causing “Unwanted VCs”.

1. Create an edit for a test image in DxPL
2. ensure that the DOP has been created (not least because you are going to edit the file)
3. Open in a text editor of you choice, go to the end of the DOP and change the last number in this UUID (Uuid) field and save the DOP

So before the change we have

After the change we have

But in this case the image containing the “changed” DOP becomes VC[1], but typically containing the edits the user wants to preserve!

The situation I am looking at here is a clash between two versions of DxPL, on the same machine in this case.

@Kit

It might be the reason for PL to create VCs without asking.

The typical reason in the past has been the round tripping I described above but I believe that has become less of a problem with later versions of the code.

However, have you been testing PL7 with images from a previous release?

So the database has a UUID field that identifies each image, it is allocated when the image is first encountered and arguably stays with the image throughout its “life”. It also exists in the DOP, while the database is updated with edit details immediately the DOP can be written immediately but generally in batches some 20+ seconds later or the user can force it to be written.

The UUID remains with the DOP if it is taken to another system and it now appears (according to my tests) that the database will take that UUID from the DOP and use it in any new entry into the database, unless it happens that the UUID is already in use when DxPL will assign a new UUID.

However, if at any stage DxPL detects that an image being presented is already in the database then a VC (typically referred to as an “Unwanted VC” by users) is created. This protects the original edits in the [M]aster and the new edits go into VC[1] but it gets way more complicated if the image contains user VCs.

So with two databases we have two units potentially allocating UUIDs!

Providing the same images in the same directories on the same drives are presented (discovered) by a database (they are actually discovered by a copy of DxPL accessing a database) that already contains an entry for that image where the UUIDs match then there is no problem.

But if the UUIDs do not match then "Unwanted VCs will be the result!

@Kit the VC process is an essential part of editing to allow a user to evaluate different edit scenarios.

What is actually needed is for DxPL to provide a mechanism that allows a VC to replace the [M]aster and to be able to do that for an entire directory, and/or permit a [M]aster to be deleted but without actually deleting the image itself, essentially a database removal not the current removal which deleted everything in sight for that image!!

All of these options have been requested time and time again but …

@Kit Consider the use of the following to isolate the VCs

But there is no ‘Filter’ option (there should be) so you need to select all images in the directory and ‘Sort’ on ‘Virtual copy number’ and be careful not to go beyond the last VC because you will then stray into the [M]aster copies!

@Kit

I didn’t say that. I did delete some pics myself accidently bcs it was sorted by VC number, i marked the numbers and there were still some “M” in between which i have overseen.

@Kit that shouldn’t happen!? All the VCs should be sorted to the front and the [M]asters should be bringing up the rear!?

With respect to the NAS playing with the dates I am not convinced about dates causing VCs but I haven’t tested it and DxPL is useless with dates anyway (another story). It is absolutely committed to UUIDs matching however!

@Kit

And…my photos are on a 70TB Synology NAS,

Your NAS is somewhat larger than my 5TB “baby” NAS but that is Synology and I haven’t noticed any issues with dates whatsoever?

[rsp]

What we need is the possibility to sort, not by VC number, but by the PL version number that created the VC.
Else, it’s a mess.

[anon78744791]

How much extra disk space is actually created for each virtual copy? And does a virtual copy use up different space other than increasing the size of the DOP file and database?

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@anon78744791 It gets another database entry and basically doubles the size of a DOP, the image remains unchanged. The size of a DOP depends on the nature of the edits and will differ somewhat from users to user.

I do find this a pain

PS:_ This is a DOP with a very basic edit

and the database is hardly bigger than an empty one

This is with 30 (i.e. [M]aster + 30 VCs) VCs created

and the database has jumped a bit in size but essentially the size of the DOPs plus additional image data!

PPS:-

The ‘Items’ Table with one entry for every image “imported” or created (VC)

The ‘Settings’ field contains the contents of the edit part of the DOP from ‘Settings’ onwards!

The ‘Metadata’ table contains camera details

The ‘IPTC’ Table also gets one entry per image & VC since they can all have independent IPTC settings

The software that provides that data

[Lucabeer (Luca Chichizola)]

I have been using Photolab for now 4 years, and I have never seen the creation of an unwanted Virtual Copy ONCE.

And I use Windows.

Probably it’s because I work on single folders at a time, and I avoid the indexing of my whole photo catalogue.

[anon78744791]

Thanks for the clarification and examples @BHAYT

Ok, so with lot of edits and 30 VCs, the DOP file could be 288kB for one RAW file. Is that correct? So one VC adds around 10kB.

EDIT: I was trying to understand how much extra disk space that is for OP.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Lucabeer

Probably it’s because I work on single folders at a time, and I avoid the indexing of my whole photo catalogue.

@Lucabeer indexing is immaterial, the issue is an “alien” DOP UUID versus a database UUID. The UUID is allocated when a new image is “imported” (discovered) without an existing database entry and without a DOP in tow.

If the image is not present in the database then a UUID is assigned and will be included in the DOP when created.

If the image is present in the database but has no DOP then all is O.K. and the existing database details will be used to create a new (replacement) DOP

If the image is present in the database but has a DOP then there will be a comparison of the database UUID to the DOP, if they agree then I believe date/timestamps will be used to determine if the edits in the DOP supersede those in the database (not entirely sure about that one)

If the image is present in the database and has a DOP and the two UUIDs do not agree then database = [M]aster, DOP=VC[1] etc. to protect both edits!

Please note that present in database means drive/directories/image id. and not just the image id…

Hence, the image can be in the database many times as long as the different copies are on different drives or different directories.

@anon78744791 for very basic edits 10k is the correct starting point.

[Lucabeer (Luca Chichizola)]

@BHAYT

@Lucabeer indexing is immaterial, the issue is an “alien” DOP UUID versus a database UUID. The UUID is allocated when a new image is “imported” (discovered) without an existing database entry and without a DOP in tow.

Yes! I have duplicates of my raw files (two copies on local hard disks, and one in a remote one used for backups), but ALL of the copies have a copy of the corresponding DOP too in the same folder. All DOPs are exact copies, of course.

This seems to be enough to avoid the creation of unwanted virtual copies, and I would consider it good standard practice.

Simple rule: wherever you have a RAW file that you already processed, have the DOP in the same path too. And be sure it’s the SAME DOP. Otherwise it’s asking for trouble. It’s NOT a bug of Photolab.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Lucabeer

but ALL of the copies have a copy of the corresponding DOP too in the same folder

@Lucabeer As I do with my five copies, but no remote because our internet connection simply can’handle it (0.5 - 1.0Mb/s does’tr get you much).

But I have a couple of small portable drives (4GB and 5GB) and should be copying to them and leaving them is the Garage, shed etc. so that they are “off-site”.

@Lucabeer

All DOPs are exact copies, of course.

A file without a DOP is “naked”.

However, if you don’t regularly trash the database or suffer a disaster with the database then a “naked” image can find “clothes” in the database!

[Joanna (Joanna Carter)]

@Kit

I know about the issues of Windows, but hey…they pay me the most.

Yup. I know what you mean. Even when I got my first MacBook Pro, I straightaway installed Parallels and transferred my Windows XP (yup ) into a virtual machine, because I was in the middle of one of the best paying consultancy/programming jobs of my career - a massive Windows app. It paid the bills for a few years.

The great thing was, when my development environment (Visual Studio) trashed the system, all I had to do was overwrite the VM with the previous night’s backup and continue working, in less than half an hour. All the source code was checked in on a version manager server, so I only lost about ten minutes work.

Listen, I agree that this should not be happening but, as others have said, it appears to be a Windows only problem, which should have been fixed a long time ago. Unless it is one of those bugs that ducked as soon as you look for it.

I don’t know for sure but I get an inkling that, somehow, date/time changes on your NAS may be triggering false change notifications to the database/DOP synchronisation mechanism.

Rather than complaining in this mainly user forum, might I suggest you make loud noises on DxO’s official bug reporting system, preferably with a watertight, reproducible test case.

Please don’t get the idea that I am belittling your abilities but, as a software engineering consultant, I am used to asking stupid questions that can sometimes get overlooked.

I repeat - I too think this is a bug, and a serious one. But that doesn’t mean that DxO should insert conditional code to “hide” it. Rather it needs tracking down and killing.

[platypus (Canon & Mac (Intel and M1))]

@Joanna

Rather it needs tracking down and killing.

The idea of not losing edits (as described in DxO’s article) makes sense imo. It provides some consistency in what DPL does when an image is edited in more than one app.

Considering that DPL is not really built for such a scenario, creating a VC is the cheapest way out.

What we’d need is a reconciliation of edits, managed by the user, at least with an option that allows to a) stick to the original edit, b) adopt the newer edit or c) for manual decision on an image to image basis.

Anyways, new features can’t be handled by older versions of DPL which leads to some loss, no matter what we do, but we could still keep those edits in the sidecar for future use.

[stuck (Canon, PL7 on Win 10 + GTX 1050ti)]

@Joanna

Which prompts me to ask why on earth you are using Windows instead of Unix-based Mac?

Oh dear, that’s not the usual helpful style of response I’ve come to expect from @joanna. Why on earth did you ask that?

[Kit]

Of course not ! 2 weeks is standard and with working apps absolutely enough. What i need is to change the app if i cannot turn of that unwanted “feature”…

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Kit You will not be able to turn off that feature any time soon (if ever).

Given the usefulness of VCs for editing they are not going to vanish and even if a flag was provided how does DxPL resolve the dilemma that it two sources of edits that it needs to reconciled or one ignored or …!

Features to fix the problem after it has occurred have been proposed by others and me but the chances of getting any resolution any time soon is unlikely!

I still do not believe that Date/timestamps of the DOP file rather than in the DOP are causing the problems, the Synology NAS can only change the former and not the latter.

I conducted an experiment yesterday where I allocated an edit and saved the DOP (DOP1) and after a few minutes I made an additional change and saved the DOP (DOP2). The database edits at this time will reflect contents of DOP2.

I then navigated PL6.10 away from the test directory and replaced the current DOP with DOP1 and the DOP was simply ignored (to be expected) I then changed the modified date to later than DOP2 and the DOP was ignored. This would be better tested by going straight for the later date modification before rediscovering in DxO.

Theer are a lot of alternatives to this test but I never asked you the following

1. What version of DxPL are you using?

2. Exactly what were you doing when the Unwanted VCs appeared?

3. Have you any idea when you edited those images before the VC occurrence?

4. Exactly what did you do that resulted in the loss of the data, i.e. as you tried to resolve the unwanted VC situation?

5. How frequently does this problem occur?

6. Which date field do you believe that the Synology NAS is adjusting.

7. Is it individual images that are getting VCs or an entire directory.

8. How long has the database been in use and do you use ‘Projects’ within DxPL? If you don’t have ‘Projects’ with DxPL(Win) then you could use a strategy of “trashing” the database before an editing/re-editing session, when there will be reason for a VC.

9. Is there any compelling reason that makes it worth further inquisition from me and other users, excluding criticisms of you backup strategy, and further investigation of the problem looking for an avoidance strategy?

I can make guesses at all the answers but it is the facts that I need to know or you can decide it is simply not worth the effort!

PS:- One quick last test

1. Navigate away from test directory
2. Copy and paste DOP
3. Change the UUID of the copy by one digit
4. Change the Date time stamps back to yesterday Mofified = Last accessed
5. Replace the DOP (DOP3) with the hacked DOP
6. Navigate back to the test directory and we get

No real surprise there then

[asvensson]

@BHAYT

Given the usefulness of VCs for editing they are not going to vanish and even if a flag was provided how does DxPL resolve the dilemma that it two sources of edits that it needs to reconciled or one ignored or …!

I would say it can at least resolve the dilemma the same way on Windows and Mac.

This is a single user program, there aren’t multiple users editing images at the same time, so any resolution is only aimed at saving the user from themselves. Not that a user can’t accidentally shoot themselves in the foot, but there are certainly scenarios in which the automatic virtual copies are just a nuisance.

For example, my images are on a server which PL sees from both Mac and Windows, with read/write of dop enabled. It’s always the last host I edit on that writes the dop I intend to carry on with. Discovering new virtual copies on Windows following a previous editing session on Mac just isn’t helpful in my case. I have dop read enabled precisely because I want PL to read the changes written by another PL. I have dop write enabled precisely because I want to communicate my latest edits to the common dop. That’s my resolution, so I’d very much like an option to disable the automatic creation of virtual copies.

DxO can also easily shoot themselves in the foot with this approach. Say you have two instances of PL looking at same directory. If both are creating virtual copies and immediately writing them to disk then they’ll loop on it. In reality the loop would probably stop when one of the two instances gets a read or write error, and I don’t think that automatic virtual copies are immediately written to dop today, but it would be a fairly easy bug for DxO to introduce.

Maybe DxO just thinks that Windows users need more saving than Mac users.

[Joanna (Joanna Carter)]

@asvensson

Maybe DxO just thinks that Windows users need more saving than Mac users.

It’s not just DxO that thinks that

[platypus (Canon & Mac (Intel and M1))]

There are no votes yet for this feature, but the discussion is quite busy…

[asvensson]

I think we know how much votes are worth.

Anyway, I think the original statement is too imprecise. I’m all for an option to disable the current behaviour on Windows (not just changing it, as DxO is wont to do, since someone out there might actually like it), but I also don’t think this is the kind of feature people should have to vote on: DxO should be interested in making the behaviour of PL consistent on both platforms without continually being prodded, and giving the users choice when it’s obviously not one size fits all.

As is, I had a long discussion with support on this back in PL5, and DxO was as amenable to addressing the issue as they are to most. That is, not at all.