Metadata: IPTC and GPS-data missing on JPEG after export

Stenis · October 30, 2023, 11:55pm

This might be a serious metadata bug.

I have refreshed a couple of JPEG libraries after updated them to Wide Gamut and Deep Prime XD. All RAW had these metadata but no metadata at all got transferred with version 7 despite all metadata checkboxes were checked.

Can somebody verify or rather falsify this error? Maybe @Joanna?
I have started to use version 7 in production but now I might have to revert to version 6 again. I have never seen any problem like this since version 4. I have exported over 1000 files today and all that I have to do again. It is real sour cream and onion.

So be aware of that it might be a problem here and that version 7 might clear all your metadata from your JPEG-files when exporting.

Update: I managed to push metadata manually to all these images through “File - Metadata - Read from Image”, so I don´t need to reexport everything at least but still this has to be investigated.

platypus · October 31, 2023, 5:19am

I do not understand what you wrote.

you say that IPTC and GPS metadata is missing
you also say that you can get metadata back by reading it from files

That‘s not logical: how can you read absent metadata? Can you give us an exakt step by step description of what you do and what you (don‘t) see and how? Which files miss metadata, which are the ones that you can read metadata from?

Pathal · October 31, 2023, 10:49am

No issue here exporting jpeg or tif with PL 7.0.2 on Mac.
All metadata are in exported files.

Joanna · October 31, 2023, 11:04am

Post a RAW file, its XMP and its exported JPG.

Did you make sure that metadata checkboxes were ticked on the export dialog?

Also, what are your preferences for reading and writing metadata?

Stenis · October 31, 2023, 1:14pm

Yes it is. There is XMP-data when I checked with XnView.

Stenis · October 31, 2023, 1:18pm

Joanna, is there any logics that can import XMP from the XMP-file attached to the master RAW?

I havn’t thought about how these readings of xmp really is carried out.

Stenis · October 31, 2023, 1:20pm

Preferences is reading from xmp-files.

All checkboxes selected at export.

George · October 31, 2023, 1:52pm

I tried with a NEF with embedded keywords. They’re visible in the JPG.

George

platypus · October 31, 2023, 2:21pm

PhotoLab automatically synchronises metadata from .xmp files and the PhotoLab database. Newer parts replace older parts. If e.g. the database has newer metadata, all metadata from the database replaces what’s in the file, be it .xmp for RAW or image file for non-RAW files. Not every metadata field has a timestamp, only the complete set has and that is why the sync is either “read all” or “write all”.

To be in control of what metadata gets read or written, it’s necessary to disable autosync in DPL’s preferences/settings and read or write manually with the respective menu entries.

Nevertheless, metadata should be written to output files, unless the boxes are unchecked in the output dialog. Running a short test with DPL 7.0.2 on macOS 14.1 shows that DPL exports IPTC and GPS to JPEG files indeed. I also found that DPL writes IPTC in XMP, while Lightroom 13.0.1 writes IPTC in IPTC, which is the old (legacy) way to write IPTC tags. IPTC Standards currently promote writing IPTC tags in XMP.

BHAYT · October 31, 2023, 6:49pm

@Stenis I don’t think I fully understand your post but let me see if this fits the bill!

You wanted to re-export some (1000) old images with ‘Wide Gamut’ and DP XD noise reduction.
Using PL7.0.2
The resulting JPGs were missing the IPTC and Keyword metadata.

So I took some .ARW images that you kindly loaned me for my testing some time ago and

Discovered them in PL7.0.2 and they contain IPTC data and a keyword of “Advertisement”
I did nothing and exported the images and all the original IPTC data and the keyword were intact!
I set ‘Wide Gamut’ and DP XD and re-exported the images and essentially the same thing, certainly with respect to the metadata

So I am confused, nothing new there then!

However, I did wind up with this mess which has me seriously confused

given that I started with this

I also exported this other directory successfully but without the additional VCs.

Returning to the original directory seems to indicate that they are “breeding”, adding two VCs per original each time

Away from the directory and back and we have even more VCs restarting DxPL does not change the problem a return to the directory adds another 2 VCs per original image !

I have seen something like this before @platypus

platypus · October 31, 2023, 7:02pm

It’s my turn to not understand what you relate to, @BHAYT

BHAYT · October 31, 2023, 7:29pm

@platypus basically DxPL is not recognising the DOPs as being DOPs it already knows about and has already stored in its database, so it keeps creating additional VCs, i…e. I cannot repeat @Stenis’s problem and managed to find another “problem” and one I think I have seen in the past and I addressed it to you in case you had seen anything similar.

So everytime I revisit the directory it does the comparison between DOP UUID and database UUID and the test fails and additional VCs are added. That directory was my “BaseLine” directory which remained largely untouched by my previous (PL5.5.0) testing.

So I started with 4 images with DOPs originally supplied by Stenis for the monochrome images and I created the additional VCs where I removed the monochrome edit back when I did the PL5.5.0 test!

I had not taken a snapshot of the original (“BaseLine”) directory I used for the test today (set Gamut and DP XD and export) but I did have another old “identical” test directory so I took a snapshot of that for the post and then re-ran the export tests using that test directory.

However, while DOP from the first test continue to be identified as

and produces new VCs every time it is visited (and contains no metadata because it was before DXPL acquired some or all of its metadata handling facilities) the test directory for the second test contains DOPs that have been updated to

and were probably PL5.5.0 DOPs as a result of my original tests.

Why PL7.0.2 is not updating the old (PhotoLab 2) DOP I do not know but it will go on adding VCs everytime that directory is revisited!

platypus · October 31, 2023, 7:40pm

Okay, you are referring to VCs. I an still connected to the original topic, but cannot produce the issue I believe the OP has. But I’m on May, so MMMV/YMMV.

PS, I don’t get any VCs while testing.

BHAYT · October 31, 2023, 7:48pm

@platypus I got them because I used BaseLine images, which are there to provide a BaseLine for comparison and should remain and did remain untouched in my original PL5.5.0 testing but lazy me “cheated” and found an issue.

The images I used were images I got from @Stenis for some tests I wanted to do and came loaded with image metadata and some old DOPs and it is the old (PhotoLab 2) DOPs that seemed to cause the problem, or so I believe!

Using @Stenis’s images I do not get any loss of Metadata whatsoever, but do get VCs in the directory with the old DOPs because it appears that they are not being overwritten and so are being “re-discovered” again and again and …

platypus · October 31, 2023, 8:51pm

Well, that is strange. I’ve never seen more than one .dop per RAW. Have you checked version numbers in the .dop files? Don’t they change?

BHAYT · November 1, 2023, 12:14am

@platypus Sorry there is only one DOP with two entries but if DxPL does not update the DOP with the UUID from the database when it has created the DB entry then every time it “finds” that DOP it will fail the UUID check and a new database entry (a VC) will be created for this “newly” discovered DOP and that will be repeated ad infinitum!

So every discovery adds another VC to the database but the DOP continues to remain unchanged for some reason or another!?

I hate finding these problems simply because I manage to “just step off the path” for one reason or another!

Given that the DOP appears to be unchanged I am going to clear the database and copy the image and DOP and repeat the “test”!

The repeated test worked or rather the attempt to repeat the issue I encountered with an empty database failed to encounter the problem I reported in the post above!

I believe that is because the database contains the same UUID as the DOP which it has taken from the DOP.

The only thing that will prevent such a re-use is when the UUID is already in use in the database, if the new UUID is not then written back to the DOP then the VC count will increase every time the image is “re-discovered”.

So having discovered the image with the DOP

I then discovered a copy of the directory in another location, same images and same original DOP, “just” a different directory!

Then I re-discovered the second location, i.e. navigated away and back

Then re-discovered again

These additional VCs are being created because the UUID does not match the database entry, i.e. DxPL allocated a new UUID to the entry in the database because the DOP UUID clashed with an existing database entry (deliberately engineered by me in this case).

In this latest test that situation is arguably to be expected because I have not made any edits but in my previous test (in the post above) I changed the Gamut and set DP XD so why was the DOP not re-written with the updated UUID!?

So I discovered the same directory in yet another location and changed the Gamut and set DP XD and checked the DOP and it was unchanged I then tried to force an export of the DOP. Where previous attempts to export the DOP appeared to work but actually did nothing a final test gave

which is a bit more representative of the reality.

Why DxPL considers that I do not have the necessary permissions I have no idea and re-running the tests with PL7.0.2 run with Administrator rights made no difference!?

Egregius · November 1, 2023, 12:31am

Based on my own experience in Windows, I think the explanation is this: if the original .dop file came from a user’s folder on another computer, then trying to own it on your own PC with administrator privileges isn’t automatically going to work. The file’s access permissions have to be rewritten, and Windows doesn’t do that without confirmation. So I wouldn’t expect PhotoLab to do it without complaining.

BHAYT · November 1, 2023, 12:51am

@Egregius I understand and having just checked and “normal” users did not have permission to alter the files but

I ran PL7.0.2 with administrator permissions
I adjusted the file permissions to all users and no success and then adjusted the directory to the same thing and no success

I changed the DOPs and the directory because there were some ‘Read Only’ files marked but that had no effect, so I changed the entire Beta Test directory and removed 'Read only and finally got the Export to work!

so I now know why PL7 and PL6 and PL5 didn’t update the DOPs (I think)

Stenis · November 1, 2023, 12:53am

Bryan, this is a good example of how problematic it can be with structural growth of a software. In Photolab we have several competing datasources. Metadata is stored both in the database and the XMP-files and the DOP-files and the database is overlapping too and the program will act differently if I check a checkbox or two.

I don´t really know what to do now but I might delete the database and start a reindex from my topfolder and start all over again because I don´t trust the data integrity i PhotoLibrary. When I checked the database before I saw no implementation of cascading update or cascading delete. These are methods used especially not to get orphaned records or data mismatches.

I have seen what you have seen once or twice before when i have accidently managed to delete DOP-files. I might also say that i cleared my JPEG exportfolder from files previous to a reexport with a slightly different file name. The older files had a suffix of 4K and the new ones are called _P3_4K_because I want to be able to see what ICC-profile my JPEG-files have embedded in an easy way. I guess that is the cause of this mess I have created in these folders I have updated.

Thank you once more for revealing the effect of my “clueless” actions. I think I have to be more careful when doing these updates in the future despite I don´t really have a new strategi in place yet for a working new workflow. I really feel I need to change my old file names when I convert my JPEG-files from sRGB to Display P3

Stenis · November 1, 2023, 1:04am

I wondered earlier over this in Photolab. Is it the case that Photolab reads the .XMP-file data and then duplicate/maps that data to the database and that is the database IPTC that populates the metadata fields/elements in PictureLibrary?

I have worked a couple of years now with Photolab synced with PhotoMechanic without the slightest problems and it even works in both direction even if I newer in production update metadata in Photolab, så I have really trusted Photolab and that workflow but now I feel I have to make some testing before my trust can be restored.