Exact rules for the creation and use of sidecar files

OXiDant · December 27, 2023, 10:04am

Ohw no i don’t.
I have a flowchart which could be shorter.
Ive done some ingestion of images i took this year in my importfolder.
A few i already processed halfway others where new.
Lot of doubles (copied from camera earlier) and mixed from phone and camera.
I skipped FRV step because they where mostly snapshots of things no real important stuff. So raw previewing wasn’t needed.
Opened Bridge 2024.
Reconnected faststone viewer in the open with list for bigscreen viewing

And some how it’s a nice easy working application.
1 select all. => template peter stamp on iptc.
2 cull and tag modes.
3 color and stars can be skipped in this case.
4 migrate, select and drop, images in the folders which are misplaced in the intake process., make new folder if needed.
5 open dxopl tag geodata if needed.(manual export updated iptcdata)
6 start editing in PL.
7 use bridge for extra input in iptc and such. ((That’s why i would like to have a masterslave sync option in dxopl. Now i need to remember to manual sync)

I started this behaviour in the early stages of pl’s metadata library management because of the quirks pl 's DAM had and mentioned in this forum.

Probably a few (or a lot) of this bugs/ quirks/hicups are resolved but i know how much work it is to restore a database with allways a to old backup.
So im glad your the guy who’s testing the status of dxo’s DAM.

be51 · December 27, 2023, 10:23am

As usual, this thread and posts are extremely long: it’s impossible for me to read everything, especially with the translation, even though I speak English; mainly because the software’s technical vocabulary is only loosely translated.
To come to the point, I don’t see how you can rely 100% on one tool to manage your photos, and if you want to switch from DxO to one or more other programs like Lightroom, it’s even more complicated.
The only reasonable solution is based on :

all necessary data at the level of the image file and its sidecars, dop and xmp
all photos identified by a unique identifier and not by a path, folder + file.

I’m trying to put this into practice :

the identifier is that given by the camera DSC12345, in which I replace DSC with a custom code (AIV for my A7RIV) and extend the sequence beyond the 9999 limit (mass renaming in the macOS finder on import)
manual organization of files in macOS by year and major themes.

I just wish DxO has the same concept and explain it clearly.
I have made some tests :

they show that if I modify xmp by adding keywords, DxO doesn’t read them and read database information
if I move a photo, DxO reads the dop file but not the xmp .

Happy end of year for all !

BHAYT · December 27, 2023, 11:06am

@OXiDant As you may be able to see from my above post with software I trust, i.e. Folder Monitor, I have a different log outcome between it showing 1 event and “my” script showing and logging 3 “Raw events” (not to be confused with RAW images).

Plus when testing changing the RAW image I have different reports from different software with respect to which keyword fields are actually being set.

This doesn’t stop analysing situations but it can throw doubt on exactly what you are seeing!

@be51 I understand the issue of the length of posts and potentially the issues of “language”, English versus ?, Win versus Mac and I am sorry if my posts confuse rather than clarify.

Nevertheless, your comment warrants investigation even if I am working on a Win10 system and you on a Mac.

I have shown in the post above that under some circumstances (but only some) DxPL misses changes made to JPGs but it typically doesn’t miss such changes in RAW when the xmp data is in the sidecar files whose timestamps are always (?) being updated.

I don’t work for DxO nor have any special allegiance other than to determine when DXPL works and when it doesn’t!

My system has a strict hierarchical organisation of files which has nothing to do with DxPL whatsoever, I don’t change the filenames but my folder structure identifies which camera and which lens was used (the image metadata will identify that for each image) i.e.

So please identify the what, where, when and how with respect to the above comment and also this comment

that DxPL failed to handle the situation properly.

But please remember that if AS(OFF) i.e. the ‘Preference’ setting is

then DXPL will not automatically read the xmp metadata ever!

With that setting you need to use one of these commands to read the xmp metadata and to write the xmp metadata

With that ‘Preference’ selected, i.e. AS(ON), then DxPL will read metadata from the image when a change is detected and/but also write the metadata to the image and the format of the hierarchical keywords in particular (if you have any) may not conform to the format you expect.

That write will be to embedded xmp metadata for JPG, TIF and DNG and only ever to an xmp sidecar file for RAW (never to the RAW image file itself).

If we understand the actions you took exactly we might get to the bottom of the problem and/or recreate the situation and/or be able to raise a support request.

The longer the doubts and confusion exist the longer the “pain” will continue and the software will fail to meet the potential that it could and should meet.

Regards

Bryan

PS:-

That statement is only accurate after the initial read has placed the data in the database BUT

With AS(OFF) if there is a DOP present when DxPL encounters an image which is not already in the database then the xmp metadata will be taken from the DOP and not from the image.

With no DOP present the first discovery will take the metadata from the image (embedded or sidecar) but no changes will be looked for or detected automatically thereafter, the user must use the ‘Read from image’ command.

Derek_K · December 27, 2023, 12:11pm

Hi, thanks all for your contributions. I have learnt a lot, I also RTFMed it up!.

DxO is pretty clear about how metadata is dealt with. Please refer to the following web page as it provides a lot of detail regarding the rules around how DxPL deals with metadata via the Database, image, XML and DOP files. and especially the section titled Useful information about how DxO PhotoLab works near the bottom of the page.

Link to Relevant page of the DxO Website…
https://userguides.dxo.com/photolab/en/menus-preferences-and-functions/

And yes - I spent almost a whole day testing and my results were congruent with the advice on the above page.

My key takeouts and next steps

I am committed to the DxO ecosystem - nothing compares in my view (and it is just a view).
I choose to work with the strengths of the ecosystem and work around its weaknesses
It is clear that DxPL does not read XML files for non-Raw Files.
It is also very clear that DOP and XML files are subordinate to the Database
It is ridiculously clear that no single standard protocol for xml files exists and we are all pretty screwed as a result.
I will find a good software package for updating metadata in non-raw image headers - a few suggested in this and other threads and I am currently trying out both digiKam and a VERY expensive option Excire Foto. FYI I am aware of Exiftool but it is a big nope for me (“skill issue” as my youngest daughter would say - not the software’s fault!).
I will simplify my use of keywords

All the best for the Festive Season and HNY. I wish you all and those you care about a wonderful and fulfilling 2024.

BHAYT · December 27, 2023, 1:03pm

@Derek_K

I have just updated digiKam and it can use ExifTool on the users behalf so it might be worth a look.

Responses to your points:

I share that view
As do I but that doesn’t mean we should give up the futile task of trying to get DxO to finish what they started!
No they don’t and I believe, but cannot verify, that means issues with Capture One if it is used for JPG etc.
Absolutely
Agreed
Bridge might be worth considering except “Big brother” (Adobe) then starts keeping an eye on you but if it suits then it is a “cheap” starter.
That is your decision I feel there is a place for both simple and hierarchical keywords.

@Derek_K you may or may not find this useful, I have posted it in the forum in the past and not done any work on it since some time in 2022.

I was looking for patterns and comparing what the various packages I could get access to actually did with the keywords entered.

I should try to bring it up to date but …

The “Hierarchical Component” refers to the simple keys that go to make up an hierarchical keyword so “a|b|c” is made up of the HC elements “a”, “b” and “c” some or all of which might find there way into the ‘DC’ keywords!

OXiDant · December 27, 2023, 3:11pm

wel i made a featurerequest for some improvements. (hopefully we stire up the taskforce of DAM.)
Please add of correct my point of view if needed.

thanks

Peter

(@BHAYT , Bryan, your deep analises of the way things happen are accurately and bring up the bugs and other “deepfish” to the surface. Every change DxO staff makes must be checked for purpose and action. So we all can activate dxo sync checkbox in the future without fear of destruction. keep up the good work. ( i can’t level your skills in this)

At this moment i think once in a wile activating the sync function to provoke .force, a readout and overwrite DxO DB value’s and deactivate before working on files could be helpfull to keep the manual master slave situation in sync.

i believe “indexing” command doesn’t force DB to overwrite existing internal metadata with external.

BHAYT · December 27, 2023, 5:29pm

@OXiDant Why do people not believe what I say?

Test with directory of mixed images JPGs, TIF, TIFF, RW2
Allocated simple keywords to images in PL7, PL-01 etc.
Find via Search
Move DxPL to a neutral directory
In Bridge delete all PL7 assigned keywords and add Bridge keywords, BR01, etc.
Move directory selection to a directory that contains the directory with the images and ‘Index’
Look for PL- keywords. none found!
Look for BR keywords all found!!

Keywords in database including misspelling:-

Bridge reviewing keywords before changing:-

Bridge after keyword changes:-

Please note I missed putting BR02 into the correct image!

Search before re-indexing:-

Re-Index operation:-

Search after re-indexing:-

Search after re-indexing for BR keywords:-

The BR0 entries are being found but not all and there appears to be no way of scrolling the search table? BR02 is missing because I neglected to add it to the appropriate image but BR07 and BR08 are also missing from the search display but present in the database!?

Database after re-indexing:-

The database still contains the old keywords but also contains the new keywords, minus BR02 which I forgot to add to image and there is a count of 2 for BR01 2 because I used it in another test and didn’t load an empty database before running the test!!

The directory after re-discovery:-

freixas · December 27, 2023, 6:01pm

Hey, I believe you.

Let me ask you about the following situations. Assume that a RAW file has already been indexed and DOP and XMP files exist. In PL7, keyword PL-01 is added to it.

Does the keyword get added to the DOP file?
Does the keyword get added to the XMP file?

PL is closed.

Assuming the DOP file contains keyword PL-01, manually edit it to PL-02.
Assuming the XMP file contains keyword PL-01, manually edit it to PL-03.

Re-index the folder and select the RAW file. What PL keyword(s) does it contain?

I think the XMP file contains some modification date field. Does it make a difference if in manually editing the file, the modification date field is not changed? (I assume Bridge would update all fields properly so a Bridge change and a manual change might have different results).

Yes, I could test all this myself, but it sounds like you already know the answers so I’m being lazy!

George · December 27, 2023, 6:22pm

It’s hard to follow you. You use different programs and files.
I just checked here with a jpg, when adding keywords no xmp file is created, only a dop. When using nef a xmp file is created.

I don’t know what you mean with this.

I don’t know Bridge, only by name. What does it do? I assume it’s deleting the keywords in a RGB image. But in PL you only have an updated RGB image after exporting the image. When you don’t you’ve only a dop file and the original image.

I assume you index only the directory containing the test images.

George

BHAYT · December 27, 2023, 6:24pm

@freixas I know a number of things but not the answer to every bizarre question possible.

The truth is I actually believe the answer is yes because your hacking will change the ‘Date Modified’ timestamp of the DOP and/or the xmp sidecar file which makes them candidates for re-reading! But I will test and include the answer here as a PS!

PS:- I am not sure what you are hoping to do with a hacked DOP but that can only be reloaded with an ‘Import’ request and if AS(OFF) the amended xmp would be loaded using a ‘Read from image’ command.

Sidecar xmp Hacked with AS(ON):-

Hacking the xmp sidecar is easy!??

Hacking the DOP is O.K. unless you get a DOP write before you can actually import the hacked DOP. I managed it O.K.

BHAYT · December 27, 2023, 6:35pm

@George simply the one you are not going to “play” with since we are testing “indexing” not normal discovery. Typically I choose a directory with no images whatsoever e.g.

Adobe Bridge provides free image management and metadata management courteously of Adobe.

You will unless you tell DxPL not to include some parts e.g.

You will have an updated RGB image file if AS(ON) is set or you use

which does what it says on the label!

In this case yes given I have around 400,00 images in my “Photos-Taken” directory I would be waiting for a long time.

In fact the PL7 indexing is a step backwards from PL6, you issue an index request on a directory now so I picked a directory up a level otherwise I would simply be discovering the directory and that is not what we are trying to test!

freixas · December 27, 2023, 6:37pm

Well, I changed computers and I found out I had just tested this! How is that for a senior moment?

The answer…

The PL database will contain a PL-01 keyword with a count of 0.
The RAW file, when selected, will contain the keyword PL-03.
The keyword PL-02 will not appear.

I edited the XMP manually and only changed the keyword. PL either detected the keyword change due to the system timestamp or because it may always read the sidecars when an image is selected. Darn! Now I have to test to tell which it is! I’ll let you know…

BHAYT · December 27, 2023, 7:22pm

@freixas I found my renaming post that never got posted, principally even by my excessive standards it is huge and very complicated but very pertinent to what you have been trying to do!

It started because someone wanted to change “Birds” to “Bird” so I started testing with “Ivies” and wanted to change to “Ivy” and ‘Plants’ to ‘Plant’ etc. However, I found what I considered a minefield and my post is way more complicated than anything I have written here!

So

I need to write a more concise summary than it has already to inform the “casual” reader.
The current “Concise” intro needs to be reworked and re-checked against PL7
The “detailed” section also needs to be reworked but will always be heavy.
But the first thing I need to do is understand exactly what I discovered and then verify my logic.

But where I believe some “issues” with keywording are user misunderstandings (although some may also be errors) if I am right about the keyword renaming issue, when keywords are already in use in DxPL, there is a fundamental problem either with the (database) design/implementation or my understanding, the latter would actually be the better situation!

I will return to that tomorrow.

freixas · December 27, 2023, 7:43pm

That was me! The keyword and sidecar issues are intertwined.

The ideal would be that, with sidecars synchronized (AS(ON)), I should be able to delete my database, re-index, and find my original hierarchy restored. The reality was that it was not.

At this point, I don’t have any small, reproducible cases that might shed clarity on what happened. I look forward to your research.

platypus · December 27, 2023, 7:55pm

Duplicate a few files in a new folder and try whatever comes to your mind. Fairly chaotic to begin with…but patterns will turn up as you proceed.

OXiDant · December 27, 2023, 8:15pm

I do believe what you say. (sometimes i have trouble to follow the exact context. )

Move dxopl to a neutral directory means highlight a neutral folderby clicking on it right?
You don’t move the folder itself because then it becomes “new” right?
Point 7 shows that ones keywords are read and stored in the DB of dxo they stay inside dop or xmp? So indexing ADD’s new keywords but not all.
Right or wrong?
According to this image it does right?

This is exactly why i don’t trust the sync option and certainly not as master DAM.
After a few years it’s dxo db cluttered with “old news” and hidden old keywords.

If i recall correct keywords and iptc are spread over xmp and dopfile due the write to image action?
If so dual sync isn’t possible then because in my case Bridge can’t read the dop file.
Ive too less free time to fully test a database(bridge) to database(dxo) and back for multiple times. In order to mimic months and years of use of both.
If so that things got shattered in xmp and dop (if read your comment correct) that would be my no go. what’s in the xmp Should Not Migrate To The Dopfile!

100%, that’s why dxo must be crystal clear in what inflics what.
Which action provokes what.
In retroprospect dxopl and therefore the dxo DAM will never be a leading factor. So it never will be a Master DAM for all people who uses dxopl.
Therefore ddopl’s DAM must be able to be set as Slave DAM. Aka it’s has to be able to read and write xmp data only back in xmp file. And acording the same rules as for instants adobe’s rules. And photomechanic’s rules. And name an other leading DAM provider.
It has to be “play nice” with others so to speak.
Aslong i am not sure of this i like to be able to delete the dxoDB and re-index to get a clean DB and not be lost any metadata which i should placed in the xmp.
I know for sure geodata is written in xmp.
Iptc fields arn’t the same in name because of language difference between bridge(Netherlands/dutch) and dxopl so blanc fields can be causing a surprise when filled.

BHAYT · December 27, 2023, 8:50pm

@OXiDant just a quick response, there should be no spread across. DOP contains what the database contains, xmp contains what it is automatically allowed to contain as a result of AS(ON) and AS(OFF), plus what the user “forces” out to the image via the ‘Write to image’ command.

There should be no mix and match between the two, rather in an AS(ON) system they should both be showing the equivalent metadata (but you may not want DxPL keywords written to the image), in an AS(OFF) situation the same thing applies with respect the keyword format but at least you are in control of when it is written.

Sadly the ‘copy selected metadata’ and the selection settings available for export are not available for the ‘Write to Image’, i.e. we would like a ‘Write to Image - selected’ command please @Musashi.

be51 · December 27, 2023, 9:31pm

Please don’t be offended, it’s not an attack, just an observation of how difficult it is to read the essentials in a language you don’t fully master.
In French, I can diagonal read.

A view of my folder organisation

I have made my own software to manage my pictures

In the explorer, I show only jpeg picture and hide associated files (raw, dop, xmp, json) so it is much easier to manage.
The unique identifier is the 8 char code at the beginning of the file name.
I use it to rename, copy, tag and resize pictures, generate html pages and maps for publication on the internet (escaich.com)

Useful reminder but I don’t see DxO updating keywords from xmp file…

I hope that DxO uses exiftool, the best tool for IPTC/XMP tags !
I use it in my own soft and it works pretty well.

freixas · December 27, 2023, 10:31pm

Correction: for RGB files, keywords should be written only to the file. For RGB files, there are no XMPs. I just checked this with Adobe Bridge 2024.

And a note: Somehow, during testing, I managed to get a DOP file with my latest keyword, but with an XMP file that included a keyword I had tagged the image with a while back. Unfortunately, I don’t yet have the minimum steps required to reproduce and file a bug report.

I had a similar problem with an RGB file and its DOP—they had two different sets of keywords—but I don’t know how this happened.

A proposal:

For RAW files, with XMP synch enabled, the database, DOP and XMP keywords should always match.
For RGB files, the database, image file and the DOP file keywords should match.
When differences are detected, they should be fixed.
For interoperability, precedence should go to XMP (RAW) or image file (RGB) over DOP. If the DOP file has a later timestamp than the XMP or image file, then it could be used if the user approves.
In any clash, the database should have the least precedence, at least with XMP synch enabled.

The XMP synch setting is confusing. For RGB files, there is no XMP file, but for RGB files, the image plays the same role as the XMP file at least with respect to storing keywords and probably other metadata. How does XMP synch affect RGB files? How should it affect RGB files?

I feel close to making a feature request here, but some of you have looked at this longer than I have and might want to counter-propose.

OXiDant · December 27, 2023, 10:41pm

And what about this written by

Oh, and not forgetting that DxO have not maintained the idea of SPOD (single point of definition), by allowing a keyword to be written to…
the database
the DOP file
an XMP file

?
We can add a fourt a rgb file’s properties

Dopfiles shouldend be a part of this storage group.

She knows alot about this matter so if she writes this i take this as true. Atleast until someone else proves otherwise.