Exact rules for the creation and use of sidecar files

You can add keywords with Exiftool too. Or if you’re a Nikon shooter download ViewNx2, the old one. If you want I can send you one.
What I did.

  1. delete the db
    2)opened PL and viewed the keywords already added to the nef file: guardia,spanje
    3)closed PL
    4)opened ViewNx2 and added keyword “viewnx”, saved it.
    5)opened PL and looked for the same image. No “viewnx” added.
    6)indexed on that directory: no viewnx seen in that image.
    7)checked with other program: keyword “viewnx” is present.
  2. the date in PL shows the “shot date”. That one didn’t change, 13-09-23
  3. the date modified changed to 26-12-23

I only can stick to my conclusion that PL doesn’t look at the raw file anymore once it was added to the db. Within the conditions I tested.

George

The forumsoftware is changing my layout. Nr 2 and 3 must be 8 and 9 :astonished:

George

The following is on Mac with DPL 7.2.0 set to not automatically r/w/s sidecars.

Starting with a RAW file without xmp sidecar and without DPL database

  • Adding a (hierarchical) keyword and saving metadata manually createed an xmp sidecar
    …containing the (hierarchical) keyword
    …containing “exif:DateTimeOriginal” set to the time of the original capture
    …containing no other timestamp, no matter if keywords was changed in DPL
  • Changing the xmp file’s modification date did not trigger the “md change” badge
  • Changing the xmp file’s creation date did not trigger the badge
  • Changing the xmp file’s content (e.g. change “test” to “toast”) did trigger the badge

While Lightroom Classic (version 12 used in the test) wrote an “xmp:MetadataDate” timestamp, changing that timestamp did not trigger the “md change” badge in DPL.

Conclusion for the given conditions:

  • PhotoLab for Mac only checked xmp sidecar content to trigger the metadata change badge on a thumbnail’s frame.
  • File or file internal image metadata timestamps had no effect during the tests.

Just curious what happens when you delete the xmp file. Will you see the change back or not. In other words does changing the xmp file manual also change the database.

George

What happens on your PC in this case?

It looks like a manual chance of the xmp file is stored in the database. Deleting the xmp file does show the change.

George

I’m having trouble following all the information posted here. I’m willing to create some flowcharts and post them for review and corrections.

I propose flowcharts for the following situations:

  • An image file is first encountered.
  • An image already in the database is encountered.
  • An image adjustment is made to the image.
  • A keyword is added to the image (with PL).

The images might be RAW or RGB and they might have DOP and/or XMP sidecars. The flowchart should include actions for XMP sync on/off.

Is there anything else that should be included?

I am not including situations such as where the image, DOP, or XMP sidecars are modified by an external tool because the the first two flowcharts should allow you to determine the results of such changes.

It is possible there may be some additional flowcharts needed for the case in which DxO is running and an image or sidecar files is externally altered, but I’m going to skip those because such operations seem like asking for trouble.

they called now “external selections”
test

How bridge is set up now:



ive setup PLv7.2


(This updates old DOPfiles to latest version.)
(i would need to redo all backups for update to this version i am afraid)
keyword setup “OFF”

And Keywords palette and IPTC palette is only visible in Photolibrary modes.

The point is somehow DxO doesn’t have any kind of selection tab for advanged searching: to click in before search.
which folder(s)?
What type of files?
which parameters : keywords/F number/shuttertime, lenstype etc.
date year something…

which i can in Bridge:


you can downsize and search as detailed as you like.

in a seach window which not unatended changes applies:

i think what i would like to have is a set up like: ( a flexible setup so every one can set up his favorite layout and structure)
library of archived rawfiles and source files (thus folder selective indexing)
library of inbetween files (not completed files, work in progress.)
library of endproducts (my jpegs)

if i want a finished product to find? click on LOE and use advanged search to find my image.
do need to find a image i was working on in my “folders of imported files to edit” : click on LOI
Do i need to find a old processed rawfile to re-process?
click on LOA
needly pointed to three main folders which holds subfolders. blinded for other folders so searching and indexing is quick and swift.

at this moment Bridge finds more then DxO when i search and often just start up Bridge to see in which folder my subject is and navigate to that in DXO or start up by making a external project.

And i can’t write it often enough: searching and selecting interfaces must be clear of editing and changing metadata knobs and choices , yes only if i choose by choice to pop up a window for this purpose. So you don’t have the risk of changing things wile searching without warning.

1 Like

The database is the leading system as soon as info is stored in it. DPL perfectly keeps all customising and metadata, even if no sidecars are used. We can think of the sidecars as a backup and a means to exchange info between different versions of DPL orwith other applications. Nevertheless, both types of sidecars carry and transport different information, albeit with slight overlap.

The only place where DPL keeps everything is the database.

Correction: I would have to include the use of external tools to find out which files are ignored under which cases.

Another item: Given a test I posted earlier in this thread, PL treats PL-produced RGB files differently from other RGB files. A flowchart should reflect the different actions.

I realize some of you already have all the information needed for the flowcharts. If you want to take a stab at a flowchart, that’s fine with me. Right now, it seems like all the information is scattered across various paragraphs of various posts and using various terminology.

While having a flowchart that shows the exact logic in all possible cases might be nice to have, day to day use might be covered by trying things with one’s proper files in their habitat and see what happens.

  • manually write metadata to file
  • check if an xmp sidecar has been created or changed

Not for 100%. The manual change of the xmp file was stored in the database.

George

Right and wrong :wink:
The leading system is the human who initiates the change. Then, when the human has decided to update metadata from the changed file, the changed metadata is stored in the database. From there on, the presence of an xmp sidecar is irrelevant as long as the database is not deleted or harmed, be it by the user or technical accident.

As long as one does not delete the database, it is THE repository of anything that a user has done to an image file, be it in customising or in metadata.

1 Like

We’re examining PL, a program.
We both found that when changing the xmp these changes where used. But I did delete the xmp also and found that these changes still existed. So there must have been a change in the db based on the change we added. I don’t know if this was done based on a time stamp or not. But if so, then it might be possible to change keywords in another program.

George

I also did another test. Changing the keywords in the dop file doesn’t change anything. Changing the keywords in the xmp file does change the keywords in the dop file.
All under my conditions.

George

@freixas Lets start from first principles because none of this happens by magic!

This is Bridge adding a keyword for the first time with DxPL “watching”:-

Detected by DxPL in real time:-

This is the event log for Bridge after making a change, from PFM (“my” Python script), I was also running Folder Monitor to verify that my reports were accurate

Bridge:

2023-12-27 09:19:14.869540 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.tmp - FILE - created
2023-12-27 09:19:14.994008 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.tmp - FILE - modified

2023-12-27 09:19:15.008004 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG~RF5ee2e4.TMP - FILE - created
2023-12-27 09:19:15.028996 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG~RF5ee2e4.TMP - FILE - deleted

2023-12-27 09:19:15.055988 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG - FILE - moved or renamed
2023-12-27 09:19:15.055988 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG~RF5ee2e4.TMP - FILE - renamed

2023-12-27 09:19:15.107973 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.tmp - FILE - moved or renamed
2023-12-27 09:19:15.107973 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG - FILE - renamed

2023-12-27 09:19:15.163954 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG - FILE - modified

2023-12-27 09:19:15.179948 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG~RF5ee2e4.TMP - FILE - deleted

2023-12-27 09:19:15.228932 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG - FILE - modified

PL7 Detecting the change:-

2023-12-27 09:19:43.761095 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG.dop - FILE - created
2023-12-27 09:19:43.922045 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_3 Bridge\P1111341.JPG.dop - FILE - modified

ExifPro showing the change after an “F5” (Refresh):=

File Timestamps before the change:-

File Timestamps after the change:-

PicMeta Pie Report:-

@George I have updated digiKam and it can set ExifTool to change RAW files but I am having trouble getting agreement between the monitoring software as to what fields in the RAW have actually been set!

So still a work in progress.

ExifPro change captured by Bridge successfully and completely missed by DxPL:-

The reason is simple DxPL ONLY CHECKS the ‘Date Modified’ timestamp!!

Bridge before the ExifPro change:-

Bridge after the ExifPro change:-

DxPL after the ExifPro Change:-

The Timestamps Before and After:-

The Event(s):-

ExifPro:-

2023-12-27 09:52:31.846632 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_1 ExifPro watching\P1111337.JPG - FILE - modified
2023-12-27 09:52:31.848631 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_1 ExifPro watching\P1111337.JPG - FILE - modified
2023-12-27 09:52:31.903614 - FILE - F:\___BETA DXO PL5 - TESTS\TEST 27 - KEYWORD RENAMING\Test 27 - 04\Test 27 - 04_1 ExifPro watching\P1111337.JPG - FILE - modified

Folder Monitor only saw one event to PFM’s 3!?

ExifPro:-


2023-12-27 09:52:31.852: F:\___Beta DXO PL5 - Tests\Test 27 - Keyword Renaming\Test 27 - 04\Test 27 - 04_1 ExifPro watching\P1111337.JPG modified

But my event log showed:-

ExifPro:-

2023-12-27 09:52:31.846632 <FileModifiedEvent: event_type=modified, src_path='F:\\___BETA DXO PL5 - TESTS\\TEST 27 - KEYWORD RENAMING\\Test 27 - 04\\Test 27 - 04_1 ExifPro watching\\P1111337.JPG', is_directory=False> 

2023-12-27 09:52:31.848631 <FileModifiedEvent: event_type=modified, src_path='F:\\___BETA DXO PL5 - TESTS\\TEST 27 - KEYWORD RENAMING\\Test 27 - 04\\Test 27 - 04_1 ExifPro watching\\P1111337.JPG', is_directory=False> 

2023-12-27 09:52:31.903614 <FileModifiedEvent: event_type=modified, src_path='F:\\___BETA DXO PL5 - TESTS\\TEST 27 - KEYWORD RENAMING\\Test 27 - 04\\Test 27 - 04_1 ExifPro watching\\P1111337.JPG', is_directory=False> 


Ohw no i don’t.
I have a flowchart which could be shorter.
Ive done some ingestion of images i took this year in my importfolder.
A few i already processed halfway others where new.
Lot of doubles (copied from camera earlier) and mixed from phone and camera.
I skipped FRV step because they where mostly snapshots of things no real important stuff. So raw previewing wasn’t needed.
Opened Bridge 2024.
Reconnected faststone viewer in the open with list for bigscreen viewing

And some how it’s a nice easy working application.
1 select all. => template peter stamp on iptc.
2 cull and tag modes.
3 color and stars can be skipped in this case.
4 migrate, select and drop, images in the folders which are misplaced in the intake process., make new folder if needed.
5 open dxopl tag geodata if needed.(manual export updated iptcdata)
6 start editing in PL.
7 use bridge for extra input in iptc and such. ((That’s why i would like to have a masterslave sync option in dxopl. Now i need to remember to manual sync)

I started this behaviour in the early stages of pl’s metadata library management because of the quirks pl 's DAM had and mentioned in this forum.

Probably a few (or a lot) of this bugs/ quirks/hicups are resolved but i know how much work it is to restore a database with allways a to old backup. :grin:
So im glad your the guy who’s testing the status of dxo’s DAM.:hugs:

As usual, this thread and posts are extremely long: it’s impossible for me to read everything, especially with the translation, even though I speak English; mainly because the software’s technical vocabulary is only loosely translated.
To come to the point, I don’t see how you can rely 100% on one tool to manage your photos, and if you want to switch from DxO to one or more other programs like Lightroom, it’s even more complicated.
The only reasonable solution is based on :

  • all necessary data at the level of the image file and its sidecars, dop and xmp
  • all photos identified by a unique identifier and not by a path, folder + file.

I’m trying to put this into practice :

  • the identifier is that given by the camera DSC12345, in which I replace DSC with a custom code (AIV for my A7RIV) and extend the sequence beyond the 9999 limit (mass renaming in the macOS finder on import)
  • manual organization of files in macOS by year and major themes.

I just wish DxO has the same concept and explain it clearly.
I have made some tests :

  • they show that if I modify xmp by adding keywords, DxO doesn’t read them and read database information
  • if I move a photo, DxO reads the dop file :innocent: but not the xmp :rage:.

Happy end of year for all !

@OXiDant As you may be able to see from my above post with software I trust, i.e. Folder Monitor, I have a different log outcome between it showing 1 event and “my” script showing and logging 3 “Raw events” (not to be confused with RAW images).

Plus when testing changing the RAW image I have different reports from different software with respect to which keyword fields are actually being set.

This doesn’t stop analysing situations but it can throw doubt on exactly what you are seeing!

@be51 I understand the issue of the length of posts and potentially the issues of “language”, English versus ?, Win versus Mac and I am sorry if my posts confuse rather than clarify.

Nevertheless, your comment warrants investigation even if I am working on a Win10 system and you on a Mac.

I have shown in the post above that under some circumstances (but only some) DxPL misses changes made to JPGs but it typically doesn’t miss such changes in RAW when the xmp data is in the sidecar files whose timestamps are always (?) being updated.

I don’t work for DxO nor have any special allegiance other than to determine when DXPL works and when it doesn’t!

My system has a strict hierarchical organisation of files which has nothing to do with DxPL whatsoever, I don’t change the filenames but my folder structure identifies which camera and which lens was used (the image metadata will identify that for each image) i.e.

So please identify the what, where, when and how with respect to the above comment and also this comment

that DxPL failed to handle the situation properly.

But please remember that if AS(OFF) i.e. the ‘Preference’ setting is

then DXPL will not automatically read the xmp metadata ever!

With that setting you need to use one of these commands to read the xmp metadata and to write the xmp metadata

image

With that ‘Preference’ selected, i.e. AS(ON), then DxPL will read metadata from the image when a change is detected and/but also write the metadata to the image and the format of the hierarchical keywords in particular (if you have any) may not conform to the format you expect.

That write will be to embedded xmp metadata for JPG, TIF and DNG and only ever to an xmp sidecar file for RAW (never to the RAW image file itself).

If we understand the actions you took exactly we might get to the bottom of the problem and/or recreate the situation and/or be able to raise a support request.

The longer the doubts and confusion exist the longer the “pain” will continue and the software will fail to meet the potential that it could and should meet.

Regards

Bryan

PS:-

That statement is only accurate after the initial read has placed the data in the database BUT

With AS(OFF) if there is a DOP present when DxPL encounters an image which is not already in the database then the xmp metadata will be taken from the DOP and not from the image.

With no DOP present the first discovery will take the metadata from the image (embedded or sidecar) but no changes will be looked for or detected automatically thereafter, the user must use the ‘Read from image’ command.

1 Like

Hi, thanks all for your contributions. I have learnt a lot, I also RTFMed it up!.

DxO is pretty clear about how metadata is dealt with. Please refer to the following web page as it provides a lot of detail regarding the rules around how DxPL deals with metadata via the Database, image, XML and DOP files. and especially the section titled Useful information about how DxO PhotoLab works near the bottom of the page.

Link to Relevant page of the DxO Website…
https://userguides.dxo.com/photolab/en/menus-preferences-and-functions/

And yes - I spent almost a whole day testing and my results were congruent with the advice on the above page.

My key takeouts and next steps

  1. I am committed to the DxO ecosystem - nothing compares in my view (and it is just a view).
  2. I choose to work with the strengths of the ecosystem and work around its weaknesses
  3. It is clear that DxPL does not read XML files for non-Raw Files.
  4. It is also very clear that DOP and XML files are subordinate to the Database
  5. It is ridiculously clear that no single standard protocol for xml files exists and we are all pretty screwed as a result.
  6. I will find a good software package for updating metadata in non-raw image headers - a few suggested in this and other threads and I am currently trying out both digiKam and a VERY expensive option Excire Foto. FYI I am aware of Exiftool but it is a big nope for me (“skill issue” as my youngest daughter would say - not the software’s fault!).
  7. I will simplify my use of keywords

All the best for the Festive Season and HNY. I wish you all and those you care about a wonderful and fulfilling 2024.