Exact rules for the creation and use of sidecar files

[freixas]

It looks like a number of you are programmers and have done some reverse engineering of various aspects of Photolab functionality.

I’ve gone for years without bothering with the DxO database. I could delete it at any time without worry. In the last week, I decided to try to use hierarchical keyword feature in PL7. I enabled “Synchronize metadata with XMP sidecar files” and “Write keyword hierarchies in the XMP dc:subject tag”. Also, when assigning keywords, I set PL to apply “whole hierarchies of selected keywords”.

Before I had PL, I used Adobe Bridge to create keywords. To bring all my keywords into PL, I had it index my main photo storage folder and all sub-folders. Some of these folders are export folders and should never be part of PL’s database, but there appears to be no easy way to exclude sub-folders.

Now I have files with no sidecars, files with only DOP sidecars, files with only XMP sidecars, and files with both DOP and XMP sidecars. I am trying to figure out when and why these sidecars are created.

Here’s my best guess as to what happens. Let’s start with what happens when PL initially encounters a file:

• PL always stores some information about the file the database. I don’t know exactly what is stored.
• If the file has no sidecars, PL copies the metadata from the file into the database and applies the default preset to the image.
• If the file has an XMP sidecar, PL copies the metadata first from the source image and then overwrites it with the metadata from the XMP sidecar.
• If the file has a DOP sidecar, PL applies the image adjustments specified there, completely ignoring the default preset.

Some information may live in both the image or sidecar files and in the database. The manual hints that PL will display a special symbol in the browser when the data conflicts (at least in cases where the user asks to keep things in synch).

What happens when PL encounters the same file later? PL will have some information about the file in the database.

• If there is no DOP sidecar, the default preset is applied. If a DOP sidecar used to exist but no longer does, nothing from it is retained (this is probably not quite right).
• Metadata is read first from the image file. If there is no XMP sidecar, then metadata additions/revisions are read from the database. If an XMP sidecar exists, the database values are ignored and the additions/revisions come from the XMP file.

What causes the sidecars to be created?

• When any image adjustment is made, a DOP sidecar is created. All further image adjustments are written to and read from this DOP file. Image adjustments never come from the database.
• When any metadata is altered or added, an XMP sidecar is created. Again, I think the XMP sidecar stores only the metadata additions/revisions. The data may also be redundantly stored in the database in case the XMP sidecar is deleted.

Let me repeat that all this is a guess. I would need to do some testing to see if my guesses are accurate, but some of you may already have done the work.

I’m pretty sure some of this is incorrect. Some of my old Photoshop-exported JPGs have wound up with DOP sidecars only. These files shouldn’t even be indexed as they are exports, not source files.

Here’s what I think happened:

• To get all my keywords in the Keywords List pane, I had to index all my images. I export files into sub-folders of my source images, so the sub-folders got indexed, too.
• The JPGs contained embedded keywords back from 2019. (PL also places metadata/keywords directly in exported files).
• When I searched for images with certain keywords, these images were included.
• When I altered the old hierarchy, a sidecar was needed since PL never alters a source file.
• I would have expected an XMP file, but PL stored the keyword changes only in a DOP sidecar. These are not usable by any other program. External software will still see my old keywords.

The last bullet tells me that I do not have a full understanding for XMP/DOP sidecar creation—or PL has a bug.

[OXiDant (Peter)]

At first let me say your are a brave puppy to switch on sync xmp and metadata in PL. i still use bridge.

Are you familiar with syncback backup software?
In there you can write your self which action should be done of each action (thus folderchoice and subfolder choice) and if you want a stop and checklist béfore change is applied.

That would be ideal in a PL database.

• New photo’s, to be worked on photo’s : full write read function one on one sync.
• rawfiles placed on an archive (in principle endproducts with some flexibility of re-editingmoments): Sync with stop and check yes/no for xmp ánd dop
• exported jpegs (endproducts): Read only with edit function for metadata. Manual enforced.

This means that we need hierachical foldertagging for indexing.
So we can tag folders to index and in which manner.
Unless we don’t have a full control on xmp writing and Dop writing i am creating backup subfolders in my raw-archive where i copy xmp and dop files into so if i screwup by letting something update wrong info into xmp or dop i can restore easily. Troublesome is the timestamp ánd keep it updated.

In retroprespective PL’s database and metadata management is in development and is doing a catch up.
We have:
Rawfiles : Dopfiles plus xmp’s (3 places of info; internal, dop and xmp.)
DNG’s : Dopfiles xmp not needed. But DNG’s could be seen as negatives so mayby usefull to use xmp.
Tiff’s: In my case inbetween product’s half development. So internal data is fine. Dop only.
OocJpeg’s as panorama’s made in camera: Dop file and xmp’s (i see them as “negative”)
Jpeg’s : End product. No dop and internal metadata only.

PL’s database must honor this hierachical system i describe above otherwise it is no use for me.

[George]

@freixas

I would have expected an XMP file, but PL stored the keyword changes only in a DOP sidecar. These are not usable by any other program. External software will still see my old keywords.

I just tested. It stores the keywords both in the dop files as in the xmp file. Check the buttons in preferences->metadata.
I do add keywords direct to my nef files. When first opened these keywords are transported to dpl but when I add again a new keyword directly to the image it’s not seen.

George

[BHAYT (Bryan [WIN10, PL8, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@freixas

1. To get all my keywords in the Keywords List pane, I had to index all my images. I export files into sub-folders of my source images, so the sub-folders got indexed, too.

@freixas When you index a directory all sub-directories will also be indexed.

When you navigate to a directory (at any level) only images in the directory (if any) will be “discovered” and imported, subdirectories will only be imported when they are “discovered” (except via indexing).

@freixas

2. The JPGs contained embedded keywords back from 2019. (PL also places metadata/keywords directly in exported files).

DxPL sticks to the rules, all RGB files will have embedded metadata which will be updated by DxPL and for RAW files it will look for embedded metadata, then sidecar metadata and will only put keywords in the xmp sidecar, along with IPTC data, Rating etc…

The DOP is a copy of the database data and will, therefore, contain both edit data and metadata.

Whether the metadata will be retrieved from the DOP will depend upon the AS setting, hopefully this might help

@freixas

3. When I searched for images with certain keywords, these images were included.

DxPL can only ever search the database, if the metadata for an image has been imported via indexing, via directory discovery or via ‘Read from image’ then it should be discovered.

But if the metadata has never been read into the database because of the AS(OFF) setting then it will not be there to be discovered.

@freixas

4. When I altered the old hierarchy, a sidecar was needed since PL never alters a source file.

The rule of the “sanctity” of the original image is not observed with respect to RGB images (JPG, TIF, DNG) in line with Adobe standards but the changes made by DxPL to the metadata of a RAW image will always be made to an xmp sidecar file, either an existing sidecar file or a new sidecar file.

@freixas

5. I would have expected an XMP file, but PL stored the keyword changes only in a DOP sidecar. These are not usable by any other program. External software will still see my old keywords.

If AS(OFF) then no metadata will be written back to the image, xmp embedded or sidecar, unless the user uses the ‘Write to image’ command.

What is in the database will always be written to the DOP BUT

1. Even if complex edits have been selected as the default edits for JPG etc. and RAW I have noticed that such edits are not written to a DOP in recent releases, until another edit of any description is made!

2. The DOP is generally not written immediately there is a DOP update cycle. We were informed that this is every 20 seconds or so but I have seen it take longer than that during keyword testing.

3. DOPs where edits have been made during a session, other than the defaults for the first discovery case, should have the DOPs re-written as part of the closedown process. I have seen reports where other users consider that this has happened and think I have seen this at least once but by the time I had realised that this might have happened it was too late to investigate!

All or none of the above may be 100% accurate because it has all been empirically determined. At no stage has DxO intervened (except with the 20 second bit) and explained the actual working of the process!?

Please note “first discovery” means a given image for a given directory does not have an entry in the database, either because it is a brand new directory or it is a directory that was in a previous database which has been replaced for some reason or another).

DxPL considered the database to be the fount of all knowledge! The DOP is a copy of the database contents at a given point in time. Hence, the DOP can be used to recover that state within a database.

Much of the bad press about DxPL and keyword handling is because the format of hierarchical keywords did not match other users outputs from their favourite keywording product and with AS(ON) the changed keywords headed back into the image!

Sadly there is not AS(ON), AS(->), AS(<-) and AS(OFF) options only AS(ON) and AS(OFF) the safest for users of other products would have been AS(<-) but that means they must use ‘Read from image’ and AS(OFF) to achieve the same thing.

The other problems with keyword handling you mentioned in your other post may well be true and I will investigate when I have more time.

Regards

Bryan

[freixas]

@OXiDant

At first let me say your are a brave puppy to switch on sync xmp and metadata in PL. i still use bridge.

I’ve considered going back to Adobe Bridge for tagging (and may yet). But this would make a lot of operations clumsier. For example, say I wanted to view all the images I’ve keyworded as potential for a gallery exhibition. Bridge would show the RAW images, PL would show the processed images.

I haven’t used Bridge since I got PL (years ago). Does it support opening an image in PL? That would make it a bit more attractive even if still not ideal.

[freixas]

@George

I just tested. It stores the keywords both in the dop files as in the xmp file. Check the buttons in preferences->metadata.

Thanks for running the test. However, I did get only DOP files, so the only question is why. I listed my preference settings for metadata in the OP.

@George

I do add keywords direct to my nef files. When first opened these keywords are transported to dpl but when I add again a new keyword directly to the image it’s not seen.

The general rule is that RAW files (actually any image source file) is never altered, so DxO might simply assume everyone, including you, follow that rule and there is no need to re-examine your source file for changes. Can you re-index the folder? Does it then pick up the new keywords?

[BHAYT (Bryan [WIN10, PL8, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@freixas

Thanks for running the test. However, I did get only DOP files, so the only question is why. I listed my preference settings for metadata in the OP.

You should have seen the image metadata (embedded or sidecar) updated so we need to get to the bottom of this “error”.

@freixas

The general rule is that RAW files (actually any image source file) is never altered, so DxO might simply assume everyone, including you, follow that rule and there is no need to re-examine your source file for changes.

@George There is a potential problem if the RAW is directly amended after the initial discovery.

If I remember some tests I conducted with PL5 they indicated once a sidecar file is in existence the data in the sidecar is always taken by DxPL in preference to the image data. While it might be considered appropriate to use the metadata timestamp I am not sure if this field is used by any software but certainly believe it is not used by DxPL(Win).

[freixas]

@BHAYT

When you navigate to a directory (at any level) only images in the directory (if any) will be “discovered” and imported, subdirectories will only be imported when they are “discovered” (except via indexing).

Yep. When you have thousands of source folders to index, though, going through the discovery process is not a good option. I think PL has a problem in that it needs to allow certain subfolders to not be indexed. There is a feature request for this already.

@BHAYT

DxPL sticks to the rules, all RGB files will have embedded metadata which will be updated by DxPL and for RAW files it will look for embedded metadata, then sidecar metadata and will only put keywords in the xmp sidecar, along with IPTC data, Rating etc…

You seem pretty confident that RGB files are modified. I would want to double-check that. In the case of my JPGs with only an associated DOP file, I found that the JPG and the DOP file both had keywords. The JPG’s keywords were my original values from tagging with Adobe Bridge back in 2019. The DOP file contained my altered keywords. If you were right, I would have expected the JPG to contain my altered keywords. I made no image adjustments to this image, so there shouldn’t even have been a DOP file. The only changes I’ve made recently were keyword tagging and revisions.

@BHAYT

Whether the metadata will be retrieved from the DOP will depend upon the AS setting, hopefully this might help

Yes, thanks.

@BHAYT

What is in the database will always be written to the DOP BUT

I think I may be seeing some bugs here when I move keywords around. It’s frustrating. I move an entire hierarchy of keywords and PL says fine. Then I search for the parent keyword and find that some of the hierarchy reappears in its original location, as though the operation was only partly completed.

Thanks for the help (and on Xmas eve!)

[George]

@freixas

Can you re-index the folder? Does it then pick up the new keywords?

No,it doesn’t.
Once the image is stored in the database pl doesn’t look at it anymore. The unique name of the image consists out of pathname/image name.
For Nikon it was common to add the metadata to the raw file. Following the rules it’s allowed. The xmp specs allows that. My new camera’s are not supported by viewnx2 but I can add metadata to the file. The raw data itself is not changed.
But the question now is why you don’t get xmp files using PL.

George

[freixas]

@George

Once the image is stored in the database pl doesn’t look at it anymore. The unique name of the image consists out of pathname/image name.

Hmm… @BHAYT just pointed out the File > Metadata > Read from image menu entry. Would this pick up your revisions?

I just tried this with another test I was doing. I had assigned a keyword to three files. None of the files had the keyword embedded in the source image. I selected all three files and used File > Metadata > Read from image. The keywords disappeared for all three files. I doubled-checked the DOP files and they were gone there as well.

I then moved to a folder where I had one RAW file with no keywords and two TIFFs with embedded keywords. After using File > Metadata > Read from image, all three had keywords!

I then deleted all DOP and XMP sidecars are repeated the operation. Now, the RAW file had no keyword and the TIFFs had the keyword—this was what I thought would happen the first time.

I presume File > Metadata > Read from image means “read from the source image/DOP/XMP and ignore the database”. It could also mean "for RAW files, use the DOP if it exists; for other files read actual image file metadata. Not sure—I could keep testing all of Christmas day!

[freixas]

@BHAYT

DxPL sticks to the rules, all RGB files will have embedded metadata which will be updated by DxPL and for RAW files it will look for embedded metadata, then sidecar metadata and will only put keywords in the xmp sidecar, along with IPTC data, Rating etc…

I did some testing, and you appear to be right.

UPDATE!!! I forgot to describe the initial conditions!

• Two new folders, one named “1” and the other “2”, neither ever seen by PL.
• Each folder contains one RAW file, one TIFF exported by PL, and one TIFF exported by Affinity Photo.

I began with the preference set to synchronize metadata with XMP sidecars.

Action: Open folder “1” in PL.
Result: No new files; presumably, the database was updated.

Action: Tag all files with the keyword “Test”.
Results:

• All files gained a DOP file with the keyword “Test”.
• The DxO-produced TIFF gained an XMP file with the keyword “Test”.
• The RAW file was unchanged. The two TIFFs were modified to include the keyword “Test”.

Action: Renamed “Test” to “Test revised”.
Result:

• All DOP and XMP files plus both TIFFs were modified to remove “Test” and add “Test revised”.
• The RAW file was unchanged.

I now changed the preferences to not synchronize metadata with XMP sidecars.

Action: Open folder “2” in PL.
Result: No new files; presumably, the database was updated.

Action: Tag all files with the keyword “Test”.
Results:

• All files gained a DOP file with the keyword “Test”.
• No XMP files were created.
• The RAW file and both TIFFs were unchanged.

Action: Renamed “Test” to “Test revised 2”.
Result:

• All DOP were modified to remove “Test” and add “Test revised 2”.
• The RAW file and both TIFFs were unchanged.

@George

But the question now is why you don’t get xmp files using PL.

One interesting result of the test above is that, when metadata is set to synchronize, only TIFFs produced by DxO gained XML files. I still have a bug in that, for the JPGs I complained about, the embedded keywords didn’t match the keywords in the DOP file (I had synchronize metadata enabled).

I suspect @BHAYT’s note about delays in writing out metadata revisions and the possibility of partially-completed updates may be the culprit. If I move a keyword from one place to another, and then move another keyword before the first one fully finished updating all relevant files, the first keyword change might be incomplete. This would explain a lot of the problems I’m seeing.

[BHAYT (Bryan [WIN10, PL8, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@freixas

Yep. When you have thousands of source folders to index, though, going through the discovery process is not a good option. I think PL has a problem in that it needs to allow certain subfolders to not be indexed. There is a feature request for this already.

I was not arguing about or against the indexing process except that on Windows that used to be a background task allowing editing to continue in the foreground but since PL7 that is now a foreground task.

If it fails to index any images you believe should be indexed and therefore discoverable then that is a potential bug.

I actually believe that the normal discovery process should be optional (as a preference) and rendering should be another optional activity. No explicit import phase but a directory option that allows directories to be browsed without automatic importation or with automatic importation but not automatic rendering or with both etc.

Your request could then be an extension of that process but I believe neither will ever come to fruition.

@freixas

You seem pretty confident that RGB files are modified. I would want to double-check that.

No test that I have run since PL5 Beta testing indicates anything other than this for JPG, TIFF and DNG, only RAWs have xmp sidecar files with DxPL. If it isn’t happening then either I have missed a condition that is blocking the discovery or more likely the acceptance of the data and/or the writing of the updated data.

DXPL uses the standard operating System hooks that means it gets informed of an event of potential interest, almost certainly restricted to the directory itself by the request submitted by DxPL to the OS, i.e. not including sub-directories.

I believe that this works for DxPL as well as for every other piece of such software including Folder Monitor by Nodesoft and Python Folder Monitor programmed by me (using a template developed for the Python Watchdog library).

However, once DxPL has seen an event it now needs to decide what to do about it and in relies (almost) exclusively on the ‘Date Modified’ timestamp I believe to determine the potential relevance of the event, way to much reliance in my opinion!

@freixas

The JPG’s keywords were my original values from tagging with Adobe Bridge back in 2019. The DOP file contained my altered keywords. If you were right, I would have expected the JPG to contain my altered keywords. I made no image adjustments to this image, so there shouldn’t even have been a DOP file. The only changes I’ve made recently were keyword tagging and revisions.

1. With AS(ON) DxPL should take you original keyword from the image metadata

2. That will be written to the database and to the DOP (I believe) any changes you then make should find there way to the image metadata and then to the DOP after up to 20 seconds or thereabouts. For JPG, TIF and DNG it will go into the embedded image metadata for RAWs it will go to an xmp sidecar file regardless of whether it came from an xmp sidecar file or from the image metadata. It has done this in all tests that I have conducted but I have not run new tests on PL7 and you are "testing " way more occurrences than I have ever tested.

3. So the metadata of the image should have contained the updated metadata as you found in the DOP.

@BHAYT

Whether the metadata will be retrieved from the DOP will depend upon the AS setting, hopefully this might help

To reiterate, AS(OFF) will cause the metadata to be taken from the DOP BUT only on the first discovery, thereafter no metadata will be automatically retrieved by DxPL for that (no longer “new”) image and changing AS(OFF) to AS(ON) some time after running with AS(OFF) has its issues that can be resolved with a ‘Read from Image’ but ‘Read from’ and ‘Write to’ are destructive operations, there is no "let’s check the timestamp’ if data has been changed in the target it will be overwritten!

If AS(ON) is set during first discovery but there is no image metadata then the DOP metadata will be overwritten by whatever is discovered, potentially nothing!

Hence, if you are hoping to retrieve metadata from the DOP never, never, never present the image for first discovery on a DxPL database configured with AS(ON).

@freixas

I made no image adjustments to this image, so there shouldn’t even have been a DOP file

You changed the metadata that counts as an edit!

@freixas

Thanks for the help (and on Xmas eve!)

Christmas Day afternoon/evening here in the U.K

@freixas

I think I may be seeing some bugs here when I move keywords around. It’s frustrating. I move an entire hierarchy of keywords and PL says fine. Then I search for the parent keyword and find that some of the hierarchy reappears in its original location, as though the operation was only partly completed.

I have yet to investigate this issue but did investigate renaming keywords in DxPL and it is only straightforward just after you have added a new keyword.

I wrote a post but never posted it and now need to retest on PL7.2!?

@George

No,it doesn’t.
Once the image is stored in the database pl doesn’t look at it anymore. The unique name of the image consists out of pathname/image name.

@George There is no way that DxPL doesn’t see an item it is re-indexing, I cannot believe that existence of the file in the database will mean it is ignored, that defeats the entire process of indexing. So I ran a test

The test data

From

In ExifPro

Added new keywords

Search of directories be re-indexing

Indexed “Test 27 - Keyword Renaming” and the repeated the search for “Christmas”

So I beg to disagree @George

@freixas while I have been writing this you have made two more posts so I will post this and visit the new posts as soon as I can but metadata writes from DxPL are not delayed, it detects changes almost immediately they are made by other software (as detected by the monitoring software I frequently run) and the xmp write is not delayed when a change is made but the DOPs can be delayed waiting for a DOP update cycle, or so it seems.

PS:- even though it is an ancient piece of software I do like the keyword overlays of ExifPro (but way before the days of Hierarchical keywords).

PPS:- But the very first tests I did during PL5 Beta was with a JPG with keywords set by ExifPro and while other software detected the change DxPL didn’t!

DxPL relies on the ‘Date modified’ timestamp (alone I believe) and ExifPro makes the keyword change to a JPG without causing that field to change!!

[OXiDant (Peter)]

@freixas

Does it support opening an image in PL?

yes as a project.
You can open selected images as a project in PL

You can select exention for searching.
What i do is tagging and creating xmp by bridge.
Have PL as read only.
Use bridge as keyword and template manager iptc.
Geotaging can be done in PL with googlemaps just copy paste.
Then manual export metadata of that/those images done.

I use the database as reading and searching DB.

[freixas]

@OXiDant

yes as a project.
You can open selected images as a project in PL

Thanks. After looking at Bridge a bit, it doesn’t seem to help with the specific problem I have right now.

My keywords are mess. Here’s where I’m at:

• I have the hierarchy Birds > Kinglets > various types of kinglets.
• I have the hierarchy Wildlife > Bird > Kinglets > various types of kinglets

I need to change every image tagged with Birds > Kinglets > something to Wildlife > Birds > Kinglets > something. To do this requires a database so that whatever tool I use finds all the images with the first tag and changes it to the second tag.

PL doesn’t make this easy, but at least it can find a specific keyword using the database. I believe Bridge has no database and so has to search through thousands of files. The documentation even states that any keyword changes apply only to selected images.

If I’m wrong, let me know.

[OXiDant (Peter)]

I am sorry to say i am not very active in dxopl and bridge the past few months/actually this year. Bought a house so much time was bricked.
So i updated bridge and dxopl but didn’t spent much time in it.So many things could be changed ,now possible sins last application updates

My workflow is:
1 FRV to cull and rate tag => those who got a tag gets a FRV created xmp file.
2 Open bridge: Select all in folder and apply a template iptc.
3 add keywords in bridge.
4 open dxopl or select in bridge and open a project in dxopl.
5 NEVER change keywords nor iptc inside dxopl except when i manual push write back from dxopl
I use bridge to do changes and forces dxopl to re-read image properties/metadata.
6 as it be correct the database of dxo can be deleted without loss of metadata because everything is outside in folders.
I have a mirrored foldertree.
Rawfiles <=> endresult jpegs archive.
And new files to be edit folder.
And a dumpfolder of exported files of every photodeveloper. ( from this folder i cut paste suitable folders of finished jpegs in the folder tree photoarchive.) when i did this i also copy paste dmp and dop in a sub folder “backup 2023/12” in the rawfiles folder and migrate this in the rawfile foldertree.
And yes this triggers new entries seen by dxopl.
So it has the latest xmp data.
Normally i do this outside dxopl. Which could cause database corruption by missing files and folders. So maybe it’s douable inside dxopl at this moment.
Folder control is extended in v7 i believe.

So you could enable sync modes IF you never use the edit modes for iptc and keywords and use bridge for that.
Then you could check 1,2 and 3. Not 4. ( because i use blinded hierachical system)
In this case you could change geo data and add edit remarks in iptc fields and hopefully not corrupt keywords.

I use the upper left for searching.

Following @BHAYT 's excellent flowchart
As long sync is alway’s <=> and we can’t set each folder plus subfolders => (read only) and <= write only for export of dxo files in to a dumpfolder.
Dxopl’s metadata manager not controlable if you have a second system which act different or write different. And is backup VERY important of your image database.

For your task you could go folder by folder systemacticly inside bridge altering all keywords and then push read from file in dxopl.

Because i have sync off my editing remarks in iptc fields live only in the dxo Database.(unless i push write to file)

[freixas]

@OXiDant

For your task you could go folder by folder systemacticly inside bridge altering all keywords and then push read from file in dxopl.

Wow! Thanks for the detailed list.

I realize a lot of people are gun-shy about DxO and metadata/keywords, but despite some complications with a somewhat messed up hierarchy, I’m slowly fixing the problems and the ability to use keywords directly within PL is going to simplify things a lot.

As for using Bridge the way you suggest, it would take so much longer than what I’m doing now. I have thousands of folders each with hundreds of files.

My life may be simpler than most. The only metadata I am bothering to modify is keywords. And the only program I’m using that messes with keywords is PL. I have cloud backup with revisions going back for a year, so I can theoretically recover if I really mess things up.

Actually, what I think I might do is finish fixing my keyword hierarchy (with PL redundantly saving data to sidecars), then rename my database and re-index everything to see what I get. If all my keywords show up and point to the right files, then I can be fairly sure that I am not dependent on the database.

There are some mismatches between keywords in some JPG files and their corresponding DOP that I’ll need to look into.

[George]

@BHAYT

@George There is no way that DxPL doesn’t see an item it is re-indexing, I cannot believe that existence of the file in the database will mean it is ignored, that defeats the entire process of indexing. So I ran a test

I was adding a new keyword directly to the raw file.

1. When opening a file for the first time the xmp section is read. The data is transferred to the database.
   2)Now I’m adding a new keyword to the image directly using viewnx2. The keyword isn’t read.
   3)reindexing doesn’t add that new keyword either.

My conclusion is that PL relies solely on the database. Using PL for adding keywords does update the database.

I’m not testing exports.

George

[BHAYT (Bryan [WIN10, PL8, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@freixas

I presume File > Metadata > Read from image means “read from the source image/DOP/XMP and ignore the database”. It could also mean "for RAW files, use the DOP if it exists; for other files read actual image file metadata. Not sure—I could keep testing all of Christmas day!

@freixas two sets of commands to access embedded or sidecar metadata and the DOP data, please remember my warning about AS(ON) (or as @OXiDant shows it <=>) and no image metadata available with a first discovery situation that will result in a DOP being written from the database containing no metadata which will replace ( actually DxPL executes a Delete, followed by a Create and ends with a Modify DOP file operation) to replace the original DOP that contained some metadata, i.e. “bye bye” DOP metadata.

It is too late to use the DOP ‘Import’ because as soon as the directory containing the image is opened the “old” DOPs will be gone! Saving off the DOPs before (first) discovering such a directory would allow an opportunity to retry (I think??).

1. Test data contains 1 RAW, 2 x TIF, 2 x JPG with no keywords, no DOPs and no sidecar files + empty database

2. Discover directory in DxPL and no DOPs or xmp sidecar file created

3. Select all and apply keyword “Test” to all 5 images and the outcome was

i.e. a DOP and xmp sidecar for the RAW and a DOP for the other images.

Picmeta PIE shows the details as shown above

4, Changed “Test” to “Test Revised” and

That is always going to work in this simple case because the keyword is held as a single entry in the ‘Keywords’ Table in the database and that one entry will be renamed from “Test” to “Test Revised”!

Change preferences to AS(OFF)

1. Added “Test” to all 5 images and no change to image metadata but DOPs created with “Test” in the DOP
   

   

2. Changing “Test” to “Test Revised” was achieved in the same way in the database and reflected in the DOPs but no change was made to the image metadata.

But both my TIF files were created by DxPL exports so I need to use Affinity 2 to create a TIF and repeat the test.

PS:- @freixas The renaming scenario was exactly what I was investigating some time ago, I need to dig out whatever I saved and try to make sense of it.

@OXiDant I will read your post in detail later today, although our eldest son will be coming to join us for lunch, joining our youngest son and his family who were with us yesterday and overnight, so two sons and daughters in laws, 4 grandchildren (3 girls and 1 boy) , me and my wife for lunch and then tea!

[BHAYT (Bryan [WIN10, PL8, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@George

When opening a file for the first time the xmp section is read. The data is transferred to the database.
2)Now I’m adding a new keyword to the image directly using viewnx2. The keyword isn’t read.
3)reindexing doesn’t add that new keyword either.

@George Thank you for your update.

Unfortunately I cannot test adding keywords directly to a RAW, I used to do that with a PhotoMechanic Trial Copy and had a way of re-loading that whenever I wanted to do a test with PM but I made a mess of that and cannot do it anymore!

However, the following will cause issues with DxPL recognising a new event, i.e. I don’t believe the DxPL is actually ignoring the change so much as not recognising that it has happened and that could be for one of a number of reasons (different reason but the same result, i.e. no update will be found and added to the DxPL database).

1. The ‘Date Modified’ field has not been changed by the later addition of the metadata so it effectively does not appear to be new and that applies if DxPL is actually showing the same directory (S1 - situation/scenario 1) or if scans the directory as part of a re-discovery of the directory or as part of a (re-)indexing operation (S2).

2. If an xmp sidecar file has been created at any stage this will “block” any changes made to the RAW image, i.e. DxPL will take the sidecar as being newer than the image, possibly regardless of any timestamps associated with any file.

In case (1) above this certainly applies to JPGs and TIFFs etc. and there are two packages that DxPL fails to handle successfully, ExifPro for JPGs - always, and XnViewMP for JPGs but it seems to depend on how the change is made in XnViewMP as to whether the timestamp is changed or not (I need to do more testing to determine what works and what doesn’t).

If the ‘Date modified’ timestamp remains unchanged when another program updates the keywords, Rating etc, then DxPL will ignore any changes in both S1 and S2 situations.

In the S1 case it will have received an event notification for a specific file name but if the ‘Date modified’ timestamp is not greater than that held in the database entry then the event will be ignored!

Other software seems to be able to detect ExifPro JPG changes every time but not DxPL and DxO have ignored my many, many complaints about this since February 2021!!

So we need to know which scenario is the case or whether there is a new scenario that I have missed and/or a bug in the software.

[OXiDant (Peter)]

@freixas

I realize a lot of people are gun-shy about DxO and metadata/keywords

Yes in my case it’s about:
1 dxo’s database was in the past not very stable and hicups mismatches can destroy your carefully build metadata system. I like to know/the fact that i can delete dxo’s DB and revive with external, backupped or not, data.
2 also important, the user interface in keyword and iptc tagging.
Dxo has for me a strange interface, i have to see the last version doh.
No templates possible for iptc in dxo. (as far as i know.)
Selecting and modifying in one window. (risk of unwanted changing things.)

I would love to migrate my tagging into dxopl IF it’s working the same as Bridge.
Why? I never liked to be trapped in a application for things that is spread over decades. Applications and situations changes over the years and being sucked in one ecosystem could be tricky if this eco system is holding it’s own rules of metadatamanaging. Read pushing data in to unique sidecars which can only be read by dxoPL. => dopfiles.

I am glad your testing and investigating bridge vs dxo v7.2
Lots of people here are much more clever then me with hierachical keywording and metadata structuring and it is take a lot of time to find out which things are lost, changed, different in view, … Between the bridge i use and dxo.

That said, bridge 2022 and 2024 is also changed alot. I have to read in the possibility’s again. And set all preferences again to my liking.

Have a nice Christmas day today. For me i try to spent 1 evening every week on my pc in dxo and bridge to renew my knowledge level.