Metadata: IPTC and GPS-data missing on JPEG after export

Again, it has been my experience that in Windows these actions are not enough. Windows will still intervene. As you discovered, you needed to do something which matches the file privileges to the privileges of the parent folder(s). I’m supposing the reason is that the file came from another user on another computer, which requires not just downloading/moving the file but a process akin to importing the file. (Usually a copy operation as the local user who runs PhotoLab will work.) Does this make sense? Or have I missed something?

@Stenis with this flag set DxO will exchange data automatically to and from the image metadata (embedded and xmp sidecar) whenever it (DxPL) makes a change or when (if) DxPL detects that a change has been made!

The if part of my statement above relates to any situation where a change to an image is not accompanied by a change to the ‘Date Modified’ field. Other software seems to be able to detect changes when that field is not changed but DxPL does not, I consider this to be a major issue but DxO have ignored my complaints for years.

My principle is if other software can successfully detect the changes why can’t DxPL. It actually seems to detect such changes if you observe the behaviour of the product but if the ‘Date Modified’ fails the “has it changed” test DxPL ignores the event and no metadata is changed!

I first discovered this issue way back during PL5 Beta testing when I used ExifPro to change the keywords of JPGs I was using for tests and that was the only program that seemed to cause issues with DxPL, until I believe I found the same issue with XnViewMP more recently and in both cases it was confined to JPG files because both of those products always use xmp sidecar files for RAW metadata and DxPL detects that well!

I never had problems with Photo Mechanic and DxPL but Photo Mechanic can change embedded RAW
metadata and if it did that and didn’t change the ‘Date Modified’ flag then such changes would go undetected by DxPL!

Now DxPL changed the policy with respect to using metadata from DOPs at PL5.3.0 I believe (I know they changed the policy and I believe that was the release when it happened).

At the start of PL5 it wrote metadata changes to the DOP but never read that data back into DxPL under any circumstances and there were various complaints made by users about this change from DxPL 4 where the DOP would be used for metadata retrieval but with a way more restricted list of metadata.

So from PL5.3.0 DxO changed the behaviour with respect to DOP handling when the Auto Synchronize flag is unset (AS=OFF) in the “first discovery” situation so we have

  1. New folder (to the DxPL database) + AS = OFF i.e.

and DxPL will take the metadata from the DOP, I repeat from the DOP, and not from the image metadata (embedded or sidecar).

If the DOP contains no metadata then none will be added to the DxPL database regardless of what metadata may be attached to the image.

I believe that it is a valid strategy for compatibility with the past but it should have been assigned its own ‘Preference’ flag @Musashi and the Auto Synchronize flag should not have been “Overloaded” in this way, e.g.

I have not repeated any of my tests with respect to metadata handling on PL7, life is too short and while I appreciate my reward of a free copy of PL7, which I am now using for tests after my trial copy ran out, I will need to buy a copy as well to avoid the issue of paying full price on the next release, i.e. PL5 was the last release I bought!

So @Stenis what you have above is my knowledge with respect to metadata handling up to PL6 and even with the problems I encountered because of the “rogue” ‘Read Only’ attribute being set I still did not manage to recreate your problem of the missing metadata in your exports.

@Egregius The data came from @Stenis because I wanted images with real IPTC data for some tests I was doing and he kindly supplied some images. But I have conducted tests on these images before but not on my BaseLine directory. Typically I use Beyond Compare to compare the BaseLine to the Test directories in order to monitor the changes being made!

So this is a copy of the Test directory from my 5600G system which has a roughly duplicate set of disk drives and I found the original directory tucked away there

2023-11-01_073355_

I successfully managed to conduct tests in the two Test directories and the DOPs indicate that the last time I tested was PL6.0.0.

I used the BaseLine directory because I was being lazy and couldn’t be bothered to create new directories for the tests!

From that “backup” machine which has a better configured xPlorer 2 I have

The BaseLine images:-

The Test images used for tests in the past:-

I don’t remember having to remove the ‘Read only’ attribute to conduct the tests but it is absent on these old images!?

But, although I was responsible for the problem at no point until the final ‘Export’ command errors did I receive any notification that DOP writes were failing and we can see what a mess can result from a user just navigating too and from a directory when DOP updates are not taking place!

I am “glad” I made such an error because I learned more than if things had “just worked” and thank you for your assistance.

PS:-

I compared the two machines and have restored the BaseLine to its original state but looking at the files and directories I have ‘Read Only’ attributes on some files and on some directories in a rather arbitrary manner. The files may not have come from @Stenis with the ‘Read only’ attribute set they may or may not have been put there by me to stop changes happening arbitrarily!?

@Stenis and @Egregius there is a subtle change in the xmp sidecar file metadata!

I had added “a|m|b” to the original @Stenis metadatata during my testing way back and in PL6.0.0 the order of the simple keywords had been alphabetical “a”, “b” and “m” in the sidecar and now it is “a”, “m” and “b”!

Plus that directory has a ‘Read only’ flag and the DOP has not been changed but the xmp sidecar has been changed!!??

The XMP standard for the dc:subject tag is a bag, which implies an unordered set. There is no requirement to write things in any particular order. Ordering is a purely UI thing.

I don’t know about Windows but macOS uses Unix permissions, where it is possible to have write permissions to files that are different to folders.

@Joanna most other software I tested would put the simple keywords from “a|m|b” as “a”, “m” and “b” rather than what DxPL was putting which was “a”,“b” and “m”.

It was an observation that there appeared to be a change rather than a concern and one where DxPL appeared “motivated” to update the sidecar? Thank you for the rapid response.

It was probably a left over from a previous test but the reason for the Read only attributes has long since escaped my brain!! However, the ‘Read only’ attribute assigned to the DOP certainly caused some problems with my testing!

It was late yesterday and I was rushing to resolve the issue so I didn’t snapshot exactly what I had or had not done in the order in which I did or did not make the attribute changes but since I have located the exact original data I can repeat the exercise if I really want to understand the implications of those attributes on DxPL.

However, at no stage in my testing was the fact that the DOPs were not being written, other than the fact that I had noticed that they were still the same, flagged up by DxPL.

Had there been no negative outcome that would be acceptable behaviour but in the special case that the DOP UUID was not being re-used for the database UUID because I had blocked that by “importing” an identical directory prior to that part of the test, I was going to get additional VCs every time I revisited the directory!

I thought I had attempted individual DOP exports prior to attempting to export all DOPs which were not flagged up but … The final group export threw up an error which was more useful that it was not but pointed to permissions and I am not sure if any permission can override the ‘Read only’ attribute?

@BHAYT
Bryan, thanks a lot for that summary. Even I have experienced problems with PM Plus when it comes to image updates. Changes in images in Photolab are not propagated when using version 6. To get that to sync I need to reindex the image folder in PM after the image editing have been done in Photolab. Metadata though has never been a problem either way with the sync check box checked.

I always start with adding metadata in my work flow and as you said PM can be configured to both update IPTC and XMP. So these files ought to have had that but the IPTC -data is gone in the JPEG-files despite that. I start with adding metadata to the RAW also because it is the only way to secure that even virtual copies gets metadata and in their turn the JPEG-files that are the derivates from those RAW and their virtuals.

So for the metadata the sync works perfectly fine.

Concerning your experienced duplicate problems I would like to say that I started with erasing all my 4K -files from the inside just to avoid the problems you have got. I saw that a long time ago and that might be that old .DOP-files are spooking. I suspected they contained old search paths.

If keeping the database in sync with the .DOP-files demands keeping delete and renaming-actions inside PL, than we ought to do so in our work flows until the dataintegrity gets sorted in PL.

Concerning XnView it seem to work fine with IPTC when PhotoMechanic owns the metadata NOT so when updating from Photolab since Photolab only seem to send IPTC to the IPTC namespace in the XMP. One compatibility improvement ought to be to make Photolaab even update the older classic IPTC in the files and when they do it would be good even to change the way the change time flag is handled.

That will not change though how PM sees this with the image updates because PM is lacking live update functions enterprise DAM-systems have with the exception of supporting hot folders.

The DOP is simplicity itself with regards basic housekeeping, not so with edits and trying to compare edits is next to impossible! I sometimes feel that DxO are deliberately mixing up the edits to stop plagiarism but that might just be me being paranoid!

So here is one of your DOPs in PL7.0.2 format now containing the IPTC metadata etc. There is no path data contained in a Windows DOP and even the file name in the DOP is ignored on a Windows system!!

Sidecar = {
Date = "2023-11-01T00:47:20.6538662Z",
Software = "DxO PhotoLab 5.15.1",
Source = {
Items = {
{
Albums = "",
CreationDate = "2023-11-01T00:38:54.9856048Z",
IPTC = {
contactAddress = "Hamngatan 5 A",
contactCity = "Vaxholm",
contactCountry = "Sweden",
contactCreator = "Sten-Åke Sändh",
contactEmail = "sandh@bredband.net",
contactPhone = "+460708714220",
contactPostalCode = "18531",
contactWebsite = [[https://www.fotosidan.se/blogs/stenisfotoblogg/index.htm

http://www.fotografer.n.nu/sten-ake-sandh]],
contentDescription = [[Sweden Stockholm Södermalm 2022, 
Advertisements for mostly music events close to the City Museum of Stockholm]],
contentHeadline = "Sweden Stockholm Södermalm 2022,",
rightsUsageTerms = "BY",
statusCopyrightNotice = "© Sten-Åke Sändh",
}
,
Keywords = {
{
"Advertisement",
}
,
}
,
ModificationDate = "2023-11-01T00:39:57.9823298Z",
Name = "Ladakh 1978_009_KASH.ARW",
OutputItems = {
}
,
ProcessingStatus = 1,
Rating = 0,
Rotation = 1,
Settings = {
AppliedPresetDisplayName = "1 - DxO Standard",
AppliedPresetUniqueName = "1 - DxO default.preset",
Base = {

Sorry that is from a PL5.15.1 test I ran early this morning!

This is the PL7.0.2 DOP

Sidecar = {
Date = "2023-11-01T11:00:50.8068339Z",
Software = "DxO PhotoLab 7.0.2",
Source = {
Items = {
{
Albums = "",
CreationDate = "2023-11-01T00:06:58.0365792Z",
IPTC = {
contactAddress = "Hamngatan 5 A",
contactCity = "Vaxholm",
contactCountry = "Sweden",
contactCreator = "Sten-Åke Sändh",
contactEmail = "sandh@bredband.net",
contactPhone = "+460708714220",
contactPostalCode = "18531",
contactWebsite = [[https://www.fotosidan.se/blogs/stenisfotoblogg/index.htm

http://www.fotografer.n.nu/sten-ake-sandh]],
contentDescription = [[Sweden Stockholm Södermalm 2022, 
Advertisements for mostly music events close to the City Museum of Stockholm]],
contentDescriptionWriter = "Sten-Åke Sändh",
contentHeadline = "Sweden Stockholm Södermalm 2022,",
rightsUsageTerms = "BY",
statusCopyrightNotice = "© Sten-Åke Sändh",
statusJobIdentifier = "Berlin_2014_007.ARW",
}
,
Keywords = {
{
"Advertisement",
}
,
}
,
ModificationDate = "2023-11-01T00:07:37.0613073Z",
Name = "Ladakh 1978_009_KASH.ARW",
Orientation = 6,
OutputItems = {
}
,
ProcessingStatus = 1,
Rating = 0,
Settings = {
AppliedPresetDisplayName = "1 - DxO Standard",
AppliedPresetUniqueName = "1 - DxO default.preset",
Base = {

Please check PM’s manual for this. Writing metadata to XMP does not mean that a .xmp sidecar will be created. Let me explain: IPTC metadata used to be tagged as “IPTC” metadata. Newer IPTC standards propose to tag IPTC metadata as “XMP” metadata.

I touched this in my post above where I found that Lightroom tags IPTC metadata as “IPTC”, while DPL tags the same metadata as “XMP”. I compared metadata in JPEG files that I had exported by Lightroom and DPL respectively.

As long as the two possibilities exist, apps should be able to read both “IPTC” and “XMP” family metadata. I suppose that this will be necessary for a looong time, unless we willingly drop all legacy “IPTC” metadata.

I know, I have rigged it to export even according to the legacy standard. PM can also write metadata to the RAW if one likes that too but PL cannot.

There us quite a few metadata elements lacking in version 7 DOP and one seem to be ModificationDate.

@Stenis The format I used has scroll bars so you need to scroll down to get the rest of the data!

For the sake of completeness, with macOS, here is what happens with PL6.

At the time of entering the hierarchy…

At the time of accepting the text by pressing the Enter key…

This seems to be a DxO choice to display the tokens in alphabetical order, which is acceptable, if not “different” from some other software.

Nonetheless, the XMP file contains…

         <dc:subject>
            <rdf:Bag>
               <rdf:li>a</rdf:li>
               <rdf:li>m</rdf:li>
               <rdf:li>b</rdf:li>
            </rdf:Bag>
         </dc:subject>
         …
         <lr:hierarchicalSubject>
            <rdf:Bag>
               <rdf:li>a</rdf:li>
               <rdf:li>a|m</rdf:li>
               <rdf:li>a|m|b</rdf:li>
            </rdf:Bag>
         </lr:hierarchicalSubject>

… which is perfectly fine.

And the DOP file contains…

			Keywords = {
				{
					"a",
					"m",
					"b",
				},
				{
					"a",
				},
				{
					"a",
					"m",
				},

… which, to me, seems totally without any order, but still shouldn’t present any problem.


Now, I have organised my app to not allow the creation of hierarchies “on the fly”, but to insist on creating hierarchies explicitly in the keyword manager, to help avoid spelling errors when entering in the keywords field…


This then means that, when entering keywords for an image, only valid keywords and hierarchies are allowed…

… which, when accepted, shows as…

… which is written to the XMP file thus…

  <dc:subject>
   <rdf:Bag>
    <rdf:li>a</rdf:li>
    <rdf:li>m</rdf:li>
    <rdf:li>b</rdf:li>
   </rdf:Bag>
  </dc:subject>
  …
  <lr:hierarchicalSubject>
   <rdf:Bag>
    <rdf:li>a</rdf:li>
    <rdf:li>a|m</rdf:li>
    <rdf:li>a|m|b</rdf:li>
   </rdf:Bag>
  </lr:hierarchicalSubject>

However, if you select another image and then return to the edited one, it now shows the keywords, along with any other, non-hierarchical, keywords in alphabetical order…

I did this because the box is a single purpose UI element, only designed to show keywords, irrespective of any hierarchical context. It is the job of the keyword manager to organise and verify hierarchies, thus avoiding mistakes at entry time.


Searching is simplified by presenting available hierarchies, which are then ANDed…

… to look for the complete hierarchy

Or they can be ORed…

… to allow for searching for images that possess part of the hierarchy, as well as images that are marked with only part of it. This is something that PL doesn’t yet cater for.

As an example of something that you cannot yet do in PL, try searching for images that contain Orange in all of the following hierarchical contexts…

Sorry it would be logical though to see to that the database is the master.

I always use the geografical elmements of “Picture Taken” and “Location” plus “Event” but can’t see any of those in these DOP -files.

@Stenis this is the xmp sidecar data for that first image that I received from you last year and have been using in the tests documented above

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 5.6.0">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""
    xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
    xmlns:xmpRights="http://ns.adobe.com/xap/1.0/rights/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:xmp="http://ns.adobe.com/xap/1.0/"
    xmlns:Iptc4xmpCore="http://iptc.org/std/Iptc4xmpCore/1.0/xmlns/"
    xmlns:Iptc4xmpExt="http://iptc.org/std/Iptc4xmpExt/2008-02-29/"
    xmlns:photomechanic="http://ns.camerabits.com/photomechanic/1.0/"
   photoshop:CaptionWriter="Sten-Åke Sändh"
   photoshop:Headline="Sweden Stockholm Södermalm 2022, "
   photoshop:TransmissionReference="Berlin_2014_007.ARW"
   photoshop:Urgency="1"
   photoshop:DateCreated="2021-11-16T00:17:42"
   xmpRights:Marked="True"
   xmp:CreateDate="2021-11-16T00:17:42"
   xmp:Rating="0"
   photomechanic:ColorClass="3"
   photomechanic:Tagged="False"
   photomechanic:Prefs="0:3:0:-00001"
   photomechanic:PMVersion="PM6">
   <xmpRights:UsageTerms>
    <rdf:Alt>
     <rdf:li xml:lang="x-default">BY</rdf:li>
    </rdf:Alt>
   </xmpRights:UsageTerms>
   <dc:subject>
    <rdf:Bag>
     <rdf:li>Advertisement</rdf:li>
    </rdf:Bag>
   </dc:subject>
   <dc:description>
    <rdf:Alt>
     <rdf:li xml:lang="x-default">Sweden Stockholm Södermalm 2022, &#xA;Advertisements for mostly music events close to the City Museum of Stockholm</rdf:li>
    </rdf:Alt>
   </dc:description>
   <dc:creator>
    <rdf:Seq>
     <rdf:li>Sten-Åke Sändh</rdf:li>
    </rdf:Seq>
   </dc:creator>
   <dc:rights>
    <rdf:Alt>
     <rdf:li xml:lang="x-default">© Sten-Åke Sändh</rdf:li>
    </rdf:Alt>
   </dc:rights>
   <Iptc4xmpCore:CreatorContactInfo
    Iptc4xmpCore:CiAdrExtadr="Hamngatan 5 A"
    Iptc4xmpCore:CiAdrCity="Vaxholm"
    Iptc4xmpCore:CiAdrPcode="18531"
    Iptc4xmpCore:CiAdrCtry="Sweden"
    Iptc4xmpCore:CiTelWork="+460708714220"
    Iptc4xmpCore:CiEmailWork="sandh@bredband.net"
    Iptc4xmpCore:CiUrlWork="https://www.fotosidan.se/blogs/stenisfotoblogg/index.htm&#xA;&#xA;http://www.fotografer.n.nu/sten-ake-sandh"/>
   <Iptc4xmpExt:LocationCreated>
    <rdf:Bag>
     <rdf:li
      Iptc4xmpExt:Sublocation="Södermalm"
      Iptc4xmpExt:City="Stockholm"
      Iptc4xmpExt:ProvinceState=""
      Iptc4xmpExt:CountryName="Sweden"
      Iptc4xmpExt:CountryCode="SWE"
      Iptc4xmpExt:WorldRegion="Europe"/>
    </rdf:Bag>
   </Iptc4xmpExt:LocationCreated>
  </rdf:Description>
 </rdf:RDF>
</x:xmpmeta>

This is what PL7.0.2 shows in the ‘PhotoLibrary’ screen for that image (if I have done things properly)

also for completeness…from right to left

  1. IPTC data as entered in DPL 7.0.2 on macOS 12.7.1 (30 entries)
  2. IPTC data as seen in the .dop sidecar (30 entries)
  3. IPTC data as seen in the .xmp sidecar (35 entries)
  4. IPTC data as seen in the JPEG exported by DPL7, used CLI exiftool -G -a -u to extract metadata (38 entries)

Localisation issue: @StevenL: please check the keyword palette title. It’s in German, even though I set macOS to use English for DPL7. We’re in PhotoLibrary and I’ve not set any custom workspace. I think that I’ve already reported this. The issue is present in the RH dock only of the PhotoLibrary views in DPL versions 7, 6 and 5 and possibly before that. Didn’t check earlier versions though.

I have tested a little with integrating the new Capture One 16.3 with PhotoMechnaic Plus both with both the “Load” and “Sync” alternatives and I must say it was a great disappointment. I can´t trust that integration at all it seems really inferior to Photolab. I can´t get it to sync automatically in a reliable manner at all.

The test I have made with Photolab 7 and PhotoMechanic in both directions seem to be rock solid.

I don´t know if I do something wrong with CO but especially the keywords get really screwed up. Despite I just use a flat structure it manages the keywords really weird. I use to test to add and delete but nothing happens automatically and when I make a manual Load or Sync I dont get what is on the image in PM but instead I get ALL keywords I ever have written on that image :-(. So I must say using CO with PM does not seem to be an all that good idea. I just thought I would give it a try but so far it doesn´t seem to be an option, just a great disappointment.

I will continue to use Photolab and PM and continue to use CO for the more demanding tasks and for tethering. It would have been great if it had worked but I don´t think it does.

While I acknowledge that it’s comfortable to write keywords and more in the app that currently handles an image, I’d rather stick to a single point of definition.

Synchronizing metadata automatically seems to be the one idea that can create the biggest mess most easily. In earlier tests, I found that DPL can sync colour labels correctly with CO, but not with Lightroom, and that with keywords, it was the other way 'round. And whatever the current findings are, they might change with the next update. Lesson learned: SPOD!

The integration between PM Plus and PL has been stable since version 6 and the about 25 elements I use in PM is covered by the IPTC set in PM with the exception of one element and that I can live with.

Capture One is a completely different story. They haven’t managed to sort their syncing problems despite they have been known for quite a few years now. It is no problem to get the metadata in too when the images are imported the first time but then there are big problems as soon as something get alterad outside CO.

PL supports a dual flow where these 25 elements can be kept in sync with no visible problems but I avoid that since there are things happening with other elements since date gets “forked” a bit from both directions. I can see evidences of that looking in XnView and it is from PL to PM the storage os the file embedded IPTC classic in JPEG outside PL seems to be handled in a non correct way. PM maintains both the classic IPTC and the IPTC namespace in XMP as I have set up PM but PL seems not to since it gas it’s focus on XMP. So of that reason I avoid updating vua PL today.

When I experienced the loss of IPTC exporting JPEG from PL I at least managed to recover that data by conducting a manual sync of these JPEG-files. I don’t understand really where that data comes from. Does a JPEG import that from the sidecar of the RAW or is it just reading it from the embedded XMP IPTC of the JPEG that updates the embedded classic IPTC?
… or ???

I’d really take that issue to support.dxo.com. If PhotoLab can occasionally “forget” to add metadata to exported files, DxO should know about it and try to fix it for the benefit of all.

Meanwhile, I’d disable MD autosync and use manual import and export. Take control of your metadata flow until the issue is gone.