PL4.3.6.32 Ghosting Ratings for Same image in different directories (Win10)

BHAYT · January 14, 2022, 1:43pm

@joanna that was all I was testing but the first program I used was my “old faithful” ExifPro which caused problems from the start and PL5 still cannot cope properly with that program leaving all the keyworded photos that I have (all family photos") stuck!

My posts got so little response that I finally gave up about the time that some real progress was being made with PL5 so I am catching up rather late in the day!

But if DxO doesn’t become more interactive with the Beta testers during PL6 Beta testing then we may simply have a repeat of the PL5 release(s) and that would be a lesson not learned and an opportunity wasted!

Joanna · January 14, 2022, 4:49pm

I just took a look at the source code for ExifPro. It looks like it was created using C++ in Visual Studio for Windows - I wouldn’t even begin to fathom out why code does what. Do you have an image that has been keyworded with it that I could take a look at?

At this point, I think it might be a good idea to summarise this thread in a new, appropriately titled topic in the beta groups in anticipation of things kicking off.

We have got to a point here where things are far too confusing and I would doubt that this thread is clear enough for the guys at DxO to easily summarise. Admit it, even we’re confused

Drop me a DM and maybe I can coordinate what you think is important with my findings both here and in the previous beta.

BHAYT · January 14, 2022, 4:58pm

@joanna will do a bit later. Re-commissioned the central heating after flushing out the X800 fast cleaner refilled and put the x400 slower cleaner into the system and spotted a slow leak on an isolation valve tried tightening it up but the leak became worse so need to drain down part of the system to isolate the isolation valve and repair connection or replace!!!

Joanna · January 14, 2022, 5:23pm

Well, cleaning out old sealed water systems is a well known stimulus for leaks. Let’s hope you don’t find too many more.

John7 · January 14, 2022, 5:32pm

My younger son found that, had to get flooring up befor the downstairs flat got wet.

Joanna · January 14, 2022, 5:35pm

Well, it’s a mixed blessing but our heating is a mixture of a wood burning stove built in to a chimney breast, topped up with electric inertia heaters. Not always as convenient but wonderful to sit in front of the wood fire on a cold evening.

BHAYT · January 14, 2022, 5:38pm

John (@John7) & Joanna (@joanna} it might be the cleaning or disturbing pipework nearby or that I never fitted it properly in the first place. It is just over kitchen in the loft, but when tightening a compression joint makes things worse then its time to dismantle and check what is wrong!

Off to get a new isolation valve and 22mm and 15mm olives to replenish my stocks, then careful checking to make sure I have got enough water out of the system!?

@joanna unfortunately I have asthma and one of the things I hate to smell is burning wood (just to be awkward).

Joanna · January 14, 2022, 5:51pm

Ah, but that’s the clever bit. Our fire is sealed from the room with a glass door and, because we use the “top down” lighting method, we don’t get smoke except on very rare occasions.

jch2103 · January 14, 2022, 7:19pm

Joanna:

What I see is confirmation that how my software writes the lr:hierarchicalSubject tag is valid.
<lightroom:hierarchicalSubject>
  <rdf:Bag>
    <rdf:li>animal</rdf:li>
    <rdf:li>animal|mammal</rdf:li>
    <rdf:li>animal|mammal|bear</rdf:li>
  </rdf:Bag>
</lightroom:hierarchicalSubject>
… with one line per member of the hierarchy explicitly spelled out. To my mind, this gives the most comprehensive transmission of how the hierarchy is constructed.

At the risk of prolonging an already too-long discussion, I wanted to point out the implications of the above hierarchical keyword structure for the user(s). You mention below (above?) using something like the standard Foundation list as an example of managing/coping long lists of keywords.

The issue your app currently presents is that it multiplies the number of keywords nodes to manage. As I’ve mentioned elsewhere I use IMatch as my DAM (still Windows only). IMatch presents several different views of files, including Media & Folders, Categories, Timeline, etc. The Categories view includes hierarchical keywords (and a special IMatch group of Categories, which is a hierarchical list of items that don’t need to be stored in the image itself). My keywords list is somewhat detailed in places (e.g., different kinds of animals/plants, etc.) but nothing like a detailed scientific name hierarchy.

But even with my somewhat limited list, I would find it annoying to have ‘animal’, ‘animal|mammal’ and ‘animal|mammal|bear’ all added to the keywords list/hierarchy as it adds more items without (in my case at least) adding any useful functionality. See screenshot.

Although it doesn’t show in the crop, IMatch provides tools to search/filter keywords to very quickly find desired images.

I’m the webmaster for our local photo club, and already have to cope with keywords in images I have to handle for club purposes. Having PL5 add more items to these ‘extraneous’ keywords would not be helpful…

But I recognize everyone’s needs are different. I’d just like options in PL5 that would minimize impacts on my workflow.

BHAYT · January 14, 2022, 7:41pm

@joanna nice idea.

One 22mm olive later but on old 3/4" imperial copper pipes (the 3/4" is the inside measurement of imperial pipes while the 22mm is the outside measurement of metric pipes) so 21.49mm versus 22mm and some very careful tightening and that joint at least seems watertight so now back to re-commissioning the system!

Touch wood all is currently working. I’m going to miss climbing the steep aluminium ladder to the loft many times a day!

Joanna · January 15, 2022, 9:36am

Actually, it doesn’t really. My keyword dictionary management window consists of two lists: one that contains every keyword, the other to organise them into any required hierarchies. The hierarchical list is purely a record of the relationships between keywords - none of the keywords are duplicated.

It is very important to make the distinction between:

the dictionary, with its organisational structure, which can be as simple or as complicated as a user wants. There is no obligation to even have hierarchical keywords.
the presentation of that hierarchical list for the purposes of choosing keywords. My app provides the (optional) possibility of choosing from a tree view or simply typing in part of a keyword and being presented with a small list of contexts that contain a given keyword, no matter where it may be in any hierarchy. See this screenshot of choosing a keyword…
the inclusion of hierarchical context in the XMP. There are two purposes for keyword metadata: one is to facilitate searching, the other is to transmit any hierarchical structure to other apps that do not contain the dictionary of the app that wrote the metadata.

dc:subject is the only tag that is required for searching and must contain all keywords that are mentioned in any hierarchies.

lr:hierarchicalSubject contains a list of the relationships between any keywords mentioned in dc:subject. If there are no hierarchical keywords, then lr:hierarchicalSubject doesn’t even need to exist. If there are keywords that are hierarchical, the full hierarchy down to any mentioned leaf keyword must be mentioned in order for its relationships to be transmitted.

And this is where confusion starts to rear its ugly head. Some apps put too many keywords in lr:hierarchicalSubject, believing that even standalone keywords constitute a hierarchy of one level. This is nonsense and totally unnecessary for the purpose of transmission of hierarchical structure.

There are two possible ways to record a keyword’s hierarchical structure:

The abbreviated notation

    <lr:hierarchicalSubject>
       <rdf:Bag>
          <rdf:li>Animal|Mammal|Bear</rdf:li>
       </rdf:Bag>
    </lr:hierarchicalSubject>

This is perfectly acceptable since it describes the hierarchy if only a single leaf node is being recorded, but it omits the recording of the structure of any intermediate keywords.

The full notation…

    <lr:hierarchicalSubject>
       <rdf:Bag>
          <rdf:li>Animal</rdf:li>
          <rdf:li>Animal|Mammal</rdf:li>
          <rdf:li>Animal|Mammal|Bear</rdf:li>
       </rdf:Bag>
    </lr:hierarchicalSubject>

This is what my app writes as it allows for greater compatibility with other apps. As we have seen, C1 uses the same notation.

The biggest source of apparent complexity is not how the metadata is recorded, but how it is presented.

If an app like PL5 attempts to present both structure management and keyword selection in the same small palette, things quickly get very confusing. Which is why my app makes use of a separate management popup window so that, after a dictionary has been constructed, it no longer has to be visible unless a user wants to refer to it.

Which is what my app does, but the presentation of the tree view is optional, but it includes the ability to simply press the spacebar on a selected keyword to see a browser of all images that contain that keyword. Otherwise, a user simply types in the keywords they want to search for and presses the “Search” button. Unfortunately, if you use Windows, my app is only written for Mac.

PL5 doesn’t currently add “extraneous” hierarchy structures but it does add extraneous and unnecessary non-hierarchical keywords to the hierarchy structures. But this appears to be due to how they have organised the search mechanism.

jch2103 · January 15, 2022, 8:36pm

A few comments:

I certainly didn’t intend to attack your app, but rather criticize how PL5 is currently handling keywords. As I believe you agree, PL5’s current processing of raw image/xmp hierarchical keywords into a hierarchical keyword and multiple top-level keywords (i.e., animals|mammals|bear|black bear AND animals AND mammals AND bear) in output files is a problem.

And that’s what PL5 is currently doing in its output images. Doing so loses all hierarchical structure for those isolated keywords (and clutters up my keywords list…!).

I think we have the same criticism of PL5’s current behavior.

Having additional export options in PL5 to keep existing raw image/xmp keywords intact w/o change in output files would be helpful. I.e., I’m content to continue managing my metadata with my own DAM separate from what PL5 decides to implement. I just don’t want PL5 to interfere with what I’m doing.

Regarding your app vis-a-vis IMatch, aside from being Mac v PC, both seem to use generally similar approaches for metadata management, including but not limited to adherence to the MWG guidelines, avoidance of proprietary approaches, multiple views of hierarchical keywords and use of a thesaurus (your ‘dictionary’, I assume). I should mention that IMatch does have additional tools for keyword management beyond those I showed in the screenshot, but that’s extraneous to my issues with PL5.

Joanna · January 16, 2022, 9:31am

I didn’t think you did.

The correct, full, definition of the lr:hierarchicalSubject tag for that particular hierarchy should be…

    animals
    animals|mammals
    animals|mammals|bear
    animals|mammals|bear|black bear

… where the top level of the hierarchy should be mentioned, but there is no need for mammals, bear, or black bear to be mentioned.

I don’t know how you are maintaining a keywords list but I would have expected all keywords to be mentioned once; and only “repeated” as the child of a parent in a hierarchy.

I keep two lists - one for all keywords and a second for the relationships between them.

As does my app:

addition, removal and editing of keywords to/from the dictionary, with accompanying correction of metadata for images that refer to them
preview of images containing selected keyword in the dictionary
drag-drop addition of keywords from the list to the combined editing/search field
drag-drop of keyworded files to the edit/search field for the purpose of copying their metadata to other images
drag-drop of delimited text from a text editor to the edit/search field, which is then tokenised
cut/copy/paste of keywords between images
copy/paste of keywords from delimited text
…

BHAYT · January 16, 2022, 11:36am

@jch2103 I was about to comment on your statement above (some days ago) but fixing and re-lagging all the pipes of the central heating and re-testing all the scenarios got in the way!

I was going to state that the the topic ‘Unwanted Virtual copies’ ran from December 2020 to December 2021 but it actually only “racked up” 92 posts and we have exceeded that “somewhat” in a much shorter space of time, oops! I understand your concerns about the number of posts in this topic (my “central heating” references simply add to the “clutter”).

I understand your concerns and it would be useful to have certain features made optional providing PL5 can then interwork successfully with other products, indeed those options may well be related to such interworking. DxO see Lightroom as their role model and seek to ensure compatibility or so it seems.

@jch2103 and @Joanna

I have redone the whole of the spreadsheet again and added rows for ACDSee, Zoner and ExifPro (which works fine with PL5 for RAW images). All images are now RAW because reviewing ‘xmp’ sidecar files is quicker than getting to the data in JPGs, however I then re-discovered PIE and that simplified things further!

The discussion of the merits of IMatch versus Joanna’s program is sadly academic for those of us using Windows 10 (or 11) systems and vice versa for Mac users, IMatch is Windows only and Joanna’s program is Mac only. However, describing both has merit with respect to others on the forum who are looking for such programs and, possibly/hopefully to DxO with respect to their own (further) aspirations in that direction.

However, I do have two major concerns in this whole issue;

Many of the low/middleweight photo editing packages are still essentially IPTC oriented. For RAW this means that they only look at and populate the ‘dc’ Subject data with their (IPTC) Keywords. They are completely “blind” to ‘hr’; Hierarchical Subject data!
This is related to item 1 and my concern about the “high jacking” of the ‘dc’ Subject field as the repository for only the flat elements of hierarchical keywords and flat keywords themselves. This is effectively “deprecating” an existing capability, namely that the ‘dc’ Subject can be populated with keywords that have an hierarchical architecture but should not be used for that purpose any more!

In the spreadsheet there are entries in a difficult to see, dirty yellow colour which IPTC oriented software such as ACDSee, Zoner, Bridge (when I was populating the IPTC keywords in previous tests) etc. will never be able to see at all.

Others on this forum have stated, in past posts, that they use ACDSee or Zoner, it is possible that they have never even considered using hierarchical keywords with those products (me for one) to describe their images since no major advantage accrued from doing things that way, but it was a feature of the developing standards that has been “conveniently” highjacked!

I apologise for my previous errors in transcribing the data to the spreadsheet and @platypus’s comments about his approach being more accurate was correct but would have led to an even more complicated spreadsheet than the one I have now got.

The intention of the test was to investigate how different packages populated the ‘dc’ and ‘hr’ fields in response to a small number of different scenarios and then investigate what happened to that data when passed through the PL5 processing pipeline and exported!

The variants on this theme would be to take PL5 outputs, both keyword updates (synced or explicitly written) plus exported JPGs etc and use that data as input to your favourite editor/DAM etc. The tests are easy and quick but the transcribing is …

In each row of the spreadsheet the output from the package is shown in black and the results from the exported PL5 JPG shown in blue (or dirty yellow for those where the original software will simply be oblivious to its existence), i.e. the baseline outputs are copied to a test directory and PL5 navigated to that directory and used to export JPGs. Many cells contain the same data but the order may be changed.

The attached spreadsheet has jumped from version 04 to version 06 with version 05 retaining the old layout as well as the new as sheet 2; version 06 dispenses with the old layout completely. All the normal caveats apply, I might have left an option off in a program, I might have made an error in transcription etc. Copy of meta data setting _06.zip (13.8 KB)

Please notify me of any errors and if possible add a snapshot of your settings for any given application if you think that they are applicable.

Joanna · January 16, 2022, 2:12pm

Before we go any further, I think it might be a good idea to see what Phil Harvey writes about IPTC…

IPTC section

In first place, IPTC metadata section was made for archiving purpose. It specifies many “about” (photographer, photo content, etc.) tags, which are ment to be populated by user.
At first, IPTC also allowed using ASCII/ANSI characters only, but now, Unicode/UTF-8 can be officially used as well. Of course, IPTC section has limitations:

1st limitation: Officially, tags defined in IPTC section are length limited. Some tags can only contain 3 characters (i.e. Iptc:Category), while other can contain several hundreds characters (most tags are limited to 32 characters, though).
That’s officially. However, in most cases, more than “allowed” characters can be (and many times are) written into IPTC section, and most software will show them all. But the fact remains: officially, limitation exist.

2nd limitation: Being a bit “old” standard, IPTC section doesn’t specify tags we wish to have and need today. For example, there’s no place, where you could save “rating” of your photo. The same is true for (photographed) people names, etc.

3rd limitation: IPTC metadata specification for IPTC section isn’t maintained anymore -instead, IPTC organisation decided to move IPTC metadata specification into XMP section.
This fact added some confusion among photographers… Anyway, with many software today, when entering “IPTC” metadata, data actually isn’t written into IPTC section only (or not at all) -by most “up to date” software, it is (additionally) saved into XMP section.

To sumarize: Above limitations are only valid for metadata inside IPTC section. That is, in case you use ExifTool command like:
exiftool -Iptc:City=Paris -Iptc:By-line="My name" ...
…values will be written into “old” IPTC section -because metadata section is specified. Ok, By-line tag only exist in Iptc, but you get the idea.

Conclusion: Old IPTC is dead… time to move on.

My conclusion - that being the case, PL5 has no obligation to achieve compatibility with IPTC-driven apps.

Joanna · January 16, 2022, 2:34pm

See my post quoting Phil Harvey.

Just to make sure I understand…

Delimited hierarchical keywords have no place in the dc:subject tag. Its sole purpose is to record every keyword attributed to an image but “hierarchy delimiters” are not normally allowed or even advised.

So, no, animals|mammals|bears|black bears is only allowed if it is intended to be one “keyword” and not a hierarchy. Some software does, out of good grace to legacy users, when reading such, allow this and translates it into a hierarchy. But I suspect this was intended for those who used to record hierarchy under IPTC tags.

Well behaved modern software can allow this but should then go on to “update” it by separating it out into the dc:subject tag and writing it correctly to the lr:hierarchicalSubject tag. IMO, no attempt should be made to update the IPTC tags.

I will comment on the table later, when we have had a chance to discuss this whole IPTC mess a bit more.

jch2103 · January 16, 2022, 7:02pm

Yes. But note the distinction between Legacy IPTC (‘Old IPTC’ as discussed by Phil) and ‘modern’ XMP-based IPTC. The fact that there are two flavors of IPTC causes considerable confusion when talking about ‘IPTC’. So for clarity (for DxO’s benefit) I would revise your statement to refer to Legacy IPTC/‘Old’ IPTC apps.

platypus · January 16, 2022, 9:39pm

IPTC has been around for some time, but they adapt their standards every few years. Some interesting info about IPTC metadata use can be found here: Photo Metadata User Guide

IPTC focuses on press and telecommunications industries and pro use. DxO cannot ignore this field, if they think of DPL as a tool for professionals, even though longer standing pros already have a way to handle IPTC metadata by whatever means they use.

Non-pro users can probably get along nicely with keywords and a subset of what IPTC offers.

Nevertheless, DxO needs to get its MD handling right and I’m looking forward to the moment, when this is achieved - for Rating/Rank and all others.

jch2103 · January 16, 2022, 11:19pm

As noted in posts above, there are two ‘flavors’ of IPTC metadata, Legacy (IIM/‘Old’) and XMP-based IPTC Extension. IPTC provides more details in a section of the document you cited above: Photo Metadata User Guide
In particular:

After development over two decades IPTC Photo Metadata can be embedded in the following ways:

IPTC Core fields can be embedded in the IIM format and/or in the XMP format. A key challenge for metadata embedded in parallel in IIM and XMP is that the values are synchronised - this should be taken care of by the image management software. [Emphasis added]

IPTC Extension fields can be embedded only in XMP format.

IPTC would have done everyone a big favor by formally declaring IIM to be deprecated and superseded by the XMP-based IPTC Extension. They chose not to and dumped responsibility for keeping this (duplicate) information synchronized on makers of image management software. All of us, including DxO, would benefit from treating IIM as a read-only source of data and perhaps optionally synchronizing and writing IIM metadata only where requested by the user and needed for backwards compatibility. The XMP-based IPTC Extension metadata tags include all the information in IIM and much more, so no information would be lost by this treatment and things would be much simpler for all of us.

Rating is of course one of the standard XMP-based IPTC Extension fields.

BHAYT · January 16, 2022, 11:59pm

@jch2103 I believe that ACDsee and Zoner in particular are not using the old IPTC standard but are using the newer IPTC-XMP standard and they are both up-to-date versions and ACDSee in particular is being marketed as a general purpose data management system as well as a photo editor.

I am no expert on metadata handling, hence my post reaching out to @Joanna and you and anyone else who had something to say to help me understand this particular aspect of managing photos. I did not intend to “stir up the mud at the bottom of the pond to see what comes up…”

However, I am no novice when it comes to testing (just in tabulating the test results - it is extremely tedious - more so than reading it!) so I became concerned when 2 + 2 = 3 in the case of hierarchical keys in the ‘dc’ fields.

I first encountered this almost a year ago and it left me very confused but if you look at the table you will see that all the ‘hr’ savvy programs effectively “disappear” the “interloper” hierarchical key when one is encountered in the ‘dc’ Subject field, i.e. they all follow the pattern of replacing the “hr” keyword by its flat components in the ‘dc’ Subject field and the “hr” keyword finds its way into the ‘hr’ Hierarchical Subject fields in one form or another.

Unfortunately this leaves the ‘dc’ only software bereft of the very keyword that they created!! How can that be right was what I thought at the time and still think now! I’ve been a software engineer since 1964 and it does not add up, regardless of what the standards may say.

It is a clash of the Titans and one definitely comes off the loser, I feel - not particularly passionately but… Zoner I have used for a very long time on and off, off for some time when it went subscriber only but I got a good “family” version deal some years ago. ACDSee I have used since version 2.5 and the last licence was back in 2018 so I took an interesting offer the other day and now have an up-to-date version, it is the software I use to upload my photos from the memory card and does that job exactly the way that I want it done.

These are reasonably respected pieces of software (not as good as PL5 for “developing” photos) and they have their uses but not, it appears, when they “tangle” with the heavier weight metadata handlers like PL5 etc.

PS I “lied” about being a software engineer since 1964, that was when I started my degree course in Computing.

The following shows the original ACDSee information before adding a new keyword in PL5 and the same record after the key has been added.

PPS:- I do understand what is going on and why but the process of “normalizing” the data results in the original photo not visibly carrying the metadata that it started out with. If that program goes on to be processed by an ‘hr’ savvy program all will be well but if it goes on to be processed by another ‘dc’ only program then…!?

If PL5 implemented an option that allowed the “hr” data to be propagated to the ‘dc’ fields that would preserve the data from/for the ‘dc’ oriented programs but what would then happen when/if that data was processed by another ‘hr’ savvy program? I will run some more test, when time permits, processing the ‘dc’ program output from ‘hr’ savvy programs and then also adjust the output to see what would happen if the hybrid solution (‘dc’ = “hr” + and ‘hr’ = ‘hr’ +) is fed back into an ‘hr’ savvy program?