Exact rules for the creation and use of sidecar files

@Derek_K

I have just updated digiKam and it can use ExifTool on the users behalf so it might be worth a look.

Responses to your points:

  1. I share that view
  2. As do I but that doesn’t mean we should give up the futile task of trying to get DxO to finish what they started!
  3. No they don’t and I believe, but cannot verify, that means issues with Capture One if it is used for JPG etc.
  4. Absolutely
  5. Agreed
  6. Bridge might be worth considering except “Big brother” (Adobe) then starts keeping an eye on you but if it suits then it is a “cheap” starter.
  7. That is your decision I feel there is a place for both simple and hierarchical keywords.

@Derek_K you may or may not find this useful, I have posted it in the forum in the past and not done any work on it since some time in 2022.

I was looking for patterns and comparing what the various packages I could get access to actually did with the keywords entered.

I should try to bring it up to date but …

The “Hierarchical Component” refers to the simple keys that go to make up an hierarchical keyword so “a|b|c” is made up of the HC elements “a”, “b” and “c” some or all of which might find there way into the ‘DC’ keywords!

wel i made a featurerequest for some improvements. (hopefully we stire up the taskforce of DAM.)
Please add of correct my point of view if needed.

thanks

Peter

(@BHAYT , Bryan, your deep analises of the way things happen are accurately and bring up the bugs and other “deepfish” to the surface. Every change DxO staff makes must be checked for purpose and action. So we all can activate dxo sync checkbox in the future without fear of destruction. keep up the good work. ( i can’t level your skills in this)

At this moment i think once in a wile activating the sync function to provoke .force, a readout and overwrite DxO DB value’s and deactivate before working on files could be helpfull to keep the manual master slave situation in sync.

i believe “indexing” command doesn’t force DB to overwrite existing internal metadata with external.

@OXiDant Why do people not believe what I say?

  1. Test with directory of mixed images JPGs, TIF, TIFF, RW2
  2. Allocated simple keywords to images in PL7, PL-01 etc.
  3. Find via Search
  4. Move DxPL to a neutral directory
  5. In Bridge delete all PL7 assigned keywords and add Bridge keywords, BR01, etc.
  6. Move directory selection to a directory that contains the directory with the images and ‘Index’
  7. Look for PL- keywords. none found!
  8. Look for BR keywords all found!!

Keywords in database including misspelling:-

Bridge reviewing keywords before changing:-

Bridge after keyword changes:-

Please note I missed putting BR02 into the correct image!

Search before re-indexing:-

image

Re-Index operation:-

image

Search after re-indexing:-

image

Search after re-indexing for BR keywords:-

The BR0 entries are being found but not all and there appears to be no way of scrolling the search table? BR02 is missing because I neglected to add it to the appropriate image but BR07 and BR08 are also missing from the search display but present in the database!?

image

Database after re-indexing:-

The database still contains the old keywords but also contains the new keywords, minus BR02 which I forgot to add to image and there is a count of 2 for BR01 2 because I used it in another test and didn’t load an empty database before running the test!!

The directory after re-discovery:-

1 Like

Hey, I believe you.

Let me ask you about the following situations. Assume that a RAW file has already been indexed and DOP and XMP files exist. In PL7, keyword PL-01 is added to it.

  • Does the keyword get added to the DOP file?
  • Does the keyword get added to the XMP file?

PL is closed.

Assuming the DOP file contains keyword PL-01, manually edit it to PL-02.
Assuming the XMP file contains keyword PL-01, manually edit it to PL-03.

Re-index the folder and select the RAW file. What PL keyword(s) does it contain?

I think the XMP file contains some modification date field. Does it make a difference if in manually editing the file, the modification date field is not changed? (I assume Bridge would update all fields properly so a Bridge change and a manual change might have different results).

Yes, I could test all this myself, but it sounds like you already know the answers so I’m being lazy!

It’s hard to follow you. You use different programs and files.
I just checked here with a jpg, when adding keywords no xmp file is created, only a dop. When using nef a xmp file is created.

I don’t know what you mean with this.

I don’t know Bridge, only by name. What does it do? I assume it’s deleting the keywords in a RGB image. But in PL you only have an updated RGB image after exporting the image. When you don’t you’ve only a dop file and the original image.

I assume you index only the directory containing the test images.

George

@freixas I know a number of things but not the answer to every bizarre question possible.

The truth is I actually believe the answer is yes because your hacking will change the ‘Date Modified’ timestamp of the DOP and/or the xmp sidecar file which makes them candidates for re-reading! But I will test and include the answer here as a PS!

PS:- I am not sure what you are hoping to do with a hacked DOP but that can only be reloaded with an ‘Import’ request and if AS(OFF) the amended xmp would be loaded using a ‘Read from image’ command.

Sidecar xmp Hacked with AS(ON):-

Hacking the xmp sidecar is easy!??

Hacking the DOP is O.K. unless you get a DOP write before you can actually import the hacked DOP. I managed it O.K.

@George simply the one you are not going to “play” with since we are testing “indexing” not normal discovery. Typically I choose a directory with no images whatsoever e.g.

Adobe Bridge provides free image management and metadata management courteously of Adobe.

You will unless you tell DxPL not to include some parts e.g.

You will have an updated RGB image file if AS(ON) is set or you use

image

which does what it says on the label!

In this case yes given I have around 400,00 images in my “Photos-Taken” directory I would be waiting for a long time.

In fact the PL7 indexing is a step backwards from PL6, you issue an index request on a directory now so I picked a directory up a level otherwise I would simply be discovering the directory and that is not what we are trying to test!

Well, I changed computers and I found out I had just tested this! How is that for a senior moment? :slight_smile:

The answer…

  • The PL database will contain a PL-01 keyword with a count of 0.
  • The RAW file, when selected, will contain the keyword PL-03.
  • The keyword PL-02 will not appear.

I edited the XMP manually and only changed the keyword. PL either detected the keyword change due to the system timestamp or because it may always read the sidecars when an image is selected. Darn! Now I have to test to tell which it is! I’ll let you know…

2 Likes

@freixas I found my renaming post that never got posted, principally even by my excessive standards it is huge and very complicated but very pertinent to what you have been trying to do!

It started because someone wanted to change “Birds” to “Bird” so I started testing with “Ivies” and wanted to change to “Ivy” and ‘Plants’ to ‘Plant’ etc. However, I found what I considered a minefield and my post is way more complicated than anything I have written here!

So

  1. I need to write a more concise summary than it has already to inform the “casual” reader.
  2. The current “Concise” intro needs to be reworked and re-checked against PL7
  3. The “detailed” section also needs to be reworked but will always be heavy.
  4. But the first thing I need to do is understand exactly what I discovered and then verify my logic.

But where I believe some “issues” with keywording are user misunderstandings (although some may also be errors) if I am right about the keyword renaming issue, when keywords are already in use in DxPL, there is a fundamental problem either with the (database) design/implementation or my understanding, the latter would actually be the better situation!

I will return to that tomorrow.

1 Like

That was me! The keyword and sidecar issues are intertwined.

The ideal would be that, with sidecars synchronized (AS(ON)), I should be able to delete my database, re-index, and find my original hierarchy restored. The reality was that it was not.

At this point, I don’t have any small, reproducible cases that might shed clarity on what happened. I look forward to your research.

Duplicate a few files in a new folder and try whatever comes to your mind. Fairly chaotic to begin with…but patterns will turn up as you proceed.

I do believe what you say. (sometimes i have trouble to follow the exact context. )

Move dxopl to a neutral directory means highlight a neutral folderby clicking on it right?
You don’t move the folder itself because then it becomes “new” right?
Point 7 shows that ones keywords are read and stored in the DB of dxo they stay inside dop or xmp? So indexing ADD’s new keywords but not all.
Right or wrong?
According to this image it does right?

This is exactly why i don’t trust the sync option and certainly not as master DAM.
After a few years it’s dxo db cluttered with “old news” and hidden old keywords.

If i recall correct keywords and iptc are spread over xmp and dopfile due the write to image action?
If so dual sync isn’t possible then because in my case Bridge can’t read the dop file.
Ive too less free time to fully test a database(bridge) to database(dxo) and back for multiple times. In order to mimic months and years of use of both.
If so that things got shattered in xmp and dop (if read your comment correct) that would be my no go. what’s in the xmp Should Not Migrate To The Dopfile!

100%, that’s why dxo must be crystal clear in what inflics what.
Which action provokes what.
In retroprospect dxopl and therefore the dxo DAM will never be a leading factor. So it never will be a Master DAM for all people who uses dxopl.
Therefore ddopl’s DAM must be able to be set as Slave DAM. Aka it’s has to be able to read and write xmp data only back in xmp file. And acording the same rules as for instants adobe’s rules. And photomechanic’s rules. And name an other leading DAM provider.
It has to be “play nice” with others so to speak.
Aslong i am not sure of this i like to be able to delete the dxoDB and re-index to get a clean DB and not be lost any metadata which i should placed in the xmp.
I know for sure geodata is written in xmp.
Iptc fields arn’t the same in name because of language difference between bridge(Netherlands/dutch) and dxopl so blanc fields can be causing a surprise when filled.

@OXiDant just a quick response, there should be no spread across. DOP contains what the database contains, xmp contains what it is automatically allowed to contain as a result of AS(ON) and AS(OFF), plus what the user “forces” out to the image via the ‘Write to image’ command.

There should be no mix and match between the two, rather in an AS(ON) system they should both be showing the equivalent metadata (but you may not want DxPL keywords written to the image), in an AS(OFF) situation the same thing applies with respect the keyword format but at least you are in control of when it is written.

Sadly the ‘copy selected metadata’ and the selection settings available for export are not available for the ‘Write to Image’, i.e. we would like a ‘Write to Image - selected’ command please @Musashi.

Please don’t be offended, it’s not an attack, just an observation of how difficult it is to read the essentials in a language you don’t fully master.
In French, I can diagonal read.

A view of my folder organisation

I have made my own software to manage my pictures


In the explorer, I show only jpeg picture and hide associated files (raw, dop, xmp, json) so it is much easier to manage.
The unique identifier is the 8 char code at the beginning of the file name.
I use it to rename, copy, tag and resize pictures, generate html pages and maps for publication on the internet (escaich.com)

Useful reminder but I don’t see DxO updating keywords from xmp file…

I hope that DxO uses exiftool, the best tool for IPTC/XMP tags !
I use it in my own soft and it works pretty well.

Correction: for RGB files, keywords should be written only to the file. For RGB files, there are no XMPs. I just checked this with Adobe Bridge 2024.

And a note: Somehow, during testing, I managed to get a DOP file with my latest keyword, but with an XMP file that included a keyword I had tagged the image with a while back. Unfortunately, I don’t yet have the minimum steps required to reproduce and file a bug report.

I had a similar problem with an RGB file and its DOP—they had two different sets of keywords—but I don’t know how this happened.

A proposal:

  • For RAW files, with XMP synch enabled, the database, DOP and XMP keywords should always match.
  • For RGB files, the database, image file and the DOP file keywords should match.
  • When differences are detected, they should be fixed.
  • For interoperability, precedence should go to XMP (RAW) or image file (RGB) over DOP. If the DOP file has a later timestamp than the XMP or image file, then it could be used if the user approves.
  • In any clash, the database should have the least precedence, at least with XMP synch enabled.

The XMP synch setting is confusing. For RGB files, there is no XMP file, but for RGB files, the image plays the same role as the XMP file at least with respect to storing keywords and probably other metadata. How does XMP synch affect RGB files? How should it affect RGB files?

I feel close to making a feature request here, but some of you have looked at this longer than I have and might want to counter-propose.

And what about this written by

Oh, and not forgetting that DxO have not maintained the idea of SPOD (single point of definition), by allowing a keyword to be written to…

the database
the DOP file
an XMP file

?
We can add a fourt a rgb file’s properties

Dopfiles shouldend be a part of this storage group.

She knows alot about this matter so if she writes this i take this as true. Atleast until someone else proves otherwise.
:slightly_smiling_face:

1 Like

Yes correct, i ment no data other then exif and edit data stored in the dopfile.

i did feel free to enhance, add your findings

i found out a quirk:

[type] in bridge is not a keyword it’s a drawer (header) with keywords inside. so i never should see “type” visual in the jpg properties as “keyword” but it does.
that’s in my eye’s a mismatch/bug.

Thanks for this @BHAYT - is Bridge “free” again - The last time I tried it, I had to download a trial version of LR for it to work and then delete LR.

Currently use Mylio (definitely NOT a proper DAM but still quite powerful). I am still exploring digKam, and good to know others are using it - however, it does not appear to play nice with DxPL when it comes to coloured labels.

I also have Excire running, and it is analysing my 79715 images. The only downside of Excire (so far) is that it has no interest in video files from what I can tell. I’m not sure if this is a deal breaker just yet. The interface is beautiful - I have not done any DxO interface analysis yet either. I have to wait for the AI to finish evaluating all my images.

Regarding the table (WOW - some serious time must have gone into that - fast on the keyboard, I would assume as well) - it got me thinking that perhaps attempting to fix this universal metadata problem between packages is just too difficult and as you point out better to put the heat on DxO to fix.

An alternate that it appears some are doing is to build their own metadata “Ripper / Recompiler” so that DxO gets the information it needs and works as we would like it to. The problem is then back compatibility to the DAM. From here, it is “Turtles All the Way Down”!

Most of my video’s are short movie additions of photo’s.
A proper DAM for all image content should be supporting also video, gifs, all kinds of stills.
Therefore DxO’s DAM will never in that position. Which is fine by me.
That’s why there should be a bricked interfacestandard between DAM’s
Too much people uses there phone as a kind of DAM by using apps as instagram and god knows what more (yep i am old and not interested…:sweat_smile:) because of the buckload of new images every day on that device a propper DAM should be helpfull.

I am genuin interested how influencers organise there mixed content and will this be privacy proof?

I assume with AS is meant the sync of the xmp files. What stands AS for?

George