One of the apologets?
Making XMP-files or embedded XMP is NOT a substitute for an Import-Export function. You might experience that the day you want to migrate otherwise the risk is high you will never understand what we are discussiong here (Joanna)
Note: Given the high amount of irrelevant content, I plan to delete this thread in a few hours. I will try to find alternative ways of providing SQL scripts to PL users who might find them useful.
Freixas, do you really believe that this “Feature Request” hasn´t been written already and discussed here in many discussions before??
I also wonder if you really have been working practically with synching other applications like PhotoMechanic and iMatch with Photolab. It works fantastically well. It is both fast, almost instant and rock solid. This is really no issue att all motivating turning off synchronization. This synch is really on of the features DXO has got right from the beginning.
Compared to both Lightroom and Capture One, Photolab has had a far better interoperability with PhotoMechanic and iMatch than the other two since we got PhotoLibrary. It also goes for the need of being able to open a selected group of pictures from Photolab or iMatch. CameraBits took four years to solve that with Capture One that was fixed for Photolab four years ago. Still Lightroom is manual synch with iMatch and I still think the same goes for Photomechanic.
If you look carefully my post over your was written as a response to Joanna - not you. There is a little picture above my text that indicates that.
Your tread is not taking up something neglected or unknown here by this community. It is just neglected by DXO for a loooong time and “emergency solderings” like your suggestions are good because it gets this to float to the surface again but to be frank, they should not at all be necessary if DXO just had been a little little more interested in the suggestions they get here from the users, so don´t be disappointed if no one from DXO will participate - that is totally normal. This is important for the users but so far not for DXO.
Yes, I understood and was agreeing in a somewhat unusual way.
And even if you’d be able to do it, effort would probably still be bigger than waiting for an indexing run to finish.
Unlike Adobe’s offerings and other software, PhotoLab doesn’t make use of a “catalogue”. Instead, it uses DOP files to transfer editing steps and XMP files to transfer metadata.
Yes, PhotoLab does have a database, but it is a real mongrel with both editing and metadata in the one place - but also duplicated, but not necessarily synchronised in separate DOP and XMP files.
You might be able to “extract” metadata from the PL database but you can’t easily import that metadata into other apps because they might not use the same tag names. e.g. DOP files and the PL database uses keywords only, but XMP compliant apps use a combination of dc:subject and lr:hierarchicalSubject tags. Which is what gets recorded in XMP files but not in the PL database.
@freixas You should be aware by now of the way that the Forum works.
Unfortunately, as the author of the topic you will receive all the “traffic” that it generates, good, bad and indifferent, a fact of Forum “life”.
Yesterday (actually before your “hiatus”) I collected your scripts and saved them so that I could easily retrieve them when I wanted them, not necessarily to “protect” them
@freixas SQL Scripts.zip (5.3 KB)
I had also found a new “toy” in the shape of “SQLiteStudio” and ran the scripts through that program one after another and I could bore you with all the outputs but I have them if required and have included the outputs from the Script 4 test and the run history summary, the latter because it is a useful component of the program
My concern with script 4 is what the output is actually telling me and why!?
The database being used is just one against which I have run numerous recent tests, i.e. I have used the “_Downloads” directory and picked up images I have downloaded from Forum posts and tested in the past, and Script 4 has detected an issue or two.
But first one comment you made in your discussion about Script 4 went something like this (taken from my file of your script)
* It’s tough to match a UUID to a specific image/virtual copy;
* if the tool reports problems with an image name/keyword pair and the image has one or more virtual copies, then check them all.
*
The statement is certainly true but what we discovered when checking data with @platypus, from his Mac, was that DxPL was almost certainly sorting the “Album” entries in the Mac DOP to re-establish the order and using the ‘CreationDate’ entry of each ‘Albums’ in the DOP.
On the PC they are all arranged in the DOP in the expected order i.e. [M]aster, VC[1}m VC[2] etc, but that is not the case with Mac DOPs but the order can be determined by sorting the ‘Albums’ into ‘CreationDate’ order.
However, there is no such index in the database and no indicator in any structure about whether an item is a {M]aster or a VC, nor what the order actually is.
In fact, I believe it is on the ‘Id’ field, which is allocated when an image is discovered or a VC is created by the user and, as far as I can tell, the ‘Id’ is an ever increasing field with no re-use of “old” numbers, i.e. the order in which the entries were added to the database, first entry for the original image and then one for each VC as and when added.
but as you stated there is actually no way of knowing whether an image is the [M]aster or a VC simply by looking at the database entry.
@Joanna The XMP sidecar is not necessarily a useful “export” device because users are deliberately stopping DxO from updating the XMP sidecar for fear of it “re-formatting” their carefully crafted hierarchical keyword layout.
That “re-formatting” of keywords into the chosen DxPL format has always been there with PhotoLab but was restricted to the Export files which users seemed happy to accept until PL5, when it was extended to “adjusting” the keyword in the image (JPGs etc. or XMP sidecar for RAWs).
However, as a result I turned my attention to the one sidecar PhotoLab users do tend to keep, namely the DOP.
As for the process of making use of the outputs from the scripts or from my DOP analysis program (the new version is still under development @RAGING_FURY) the answer lies in a bit of coding somewhere to either take the output list from the scripts and run the list through a program that submits it as a command string to DxPL or, as in the case of my DOP programs, read the DOP directly and submit the located files to DxPL in a command line.
There may be issue with the number that can be passed at any one time, i.e. in the past I have managed up to 350 images at a time but the program can pass any number as a group of 300 at a time again and again and …
@Joanna As I have stated above, the DOP contains both metadata (including the ‘Pick/Reject’ flag and edits, much to the chagrin of many users, you in particular because it well and truly breaks the SPOD rule.
The “risk” is that the (latest) XMP sidecar file may not have been “absorbed” into PhotoLab, automatically or manually, but my program could easily be enhanced to interrogate the XMP sidecar file just to be on the safe side.
Any PureBasic program creates an exe file on Windows and any user could take the symbol file and compile it using the free version of PureBasic providing it is less than 800 lines long.
I am prepared to supply both if I decide to “publish” anything!
I want to avoid using ExifTool if possible, simply because of the added level of complexity right now, and your product would be off limits because it (potentially) pushes the metadata into the RAW image etc.
However, the Mac is off-limits anyway, for me at least, because I don’t own a Mac computer
Weather is looking OK. so no coding till later today or tomorrow…
I know how to match a UUID to the main image or one of its virtual copies. What I meant is that it is difficult to do so in the SQL script that I provided.
Yes, the Id fields are auto-incrementing fields. They start at 1 and increment with each new record added.
@freixas That was the point I was trying to make about there being an absence of data that identified the “Copy Number” of a particular ‘Items’ entry, i.e. there is nothing to ask SQLite to find.
A custom program on the other hand could do that, because that is exactly what DxPL does, and the DOP has that data already to hand in the Windows DOP and available after a Sort in the Mac DOP.
Once again, thank you for the scripts and for confirming the auto-incrementing Id.
PS:- The view count is at 331 in spite of or because of the dialogues contained herein sadly we can’t tell which.
@freixas , I’d not delete the thread, others carry way more noise and still contain interesting information. It simply takes more digging to get to it, but such is the character of unmoderated forums.
As for the scripts, they’d be easier to access if they were saved as text files, Copying text out of a forum post is not complicated, but it’s not comfortable.
For those interested:
Keyword SQLs MAC.zip (5.9 KB)
@platypus The title of the topic has changed as has the first post from @freixas
Some database querying tools that you might find useful which now contains the scripts for both the Mac and Windows but thank you for including a Mac one here.
Why is it hard to understand Freixas SQL and why some people that know SQL might find it necessary to use it since DXO as it is now have locked them in if they want to migrate for example. Both Lightroom and Capture One can export
I can see several cases:
First: As you might know there are organizations and companies using so called controlled vocabularies. They are impossible to use though with Photolab since there is no possibility to import one or export one. Just of that reason Photolab is not a serious Picture Library. I have myself once started using hierarchic keywords in PhotoMechanic just by importing one from a Lightroom-resource. The format was in TAB-separated text. This is the normal way to do this.
Second: Since I found PhotoMechanic like all RAW-converters or Photo-DAM (except iMatch (will explain that exception) totally hopeless in maintaining hierarchic keywords manually, the only rational way I found was to export them all from PM PLus in TAB-formatted text - editing them in Excel and then reimport them. So I guess that since it really is terrible to maintain hierarchic keywords rationally most organisations and companies using them don´t alter them if they don´t have to.
The need for exporting a vocabulary is that it might contain keyword structures that haven´t been used yet. So extracting them from the database records might not be sufficient right?
AI, yes! That is fantastically efficient when it comes to handling even hierarchic keywords BUT it can also create a total mess if you just let it run without any control. Then we comes to how to handle that because you just can´t let a tool like the Autotagger lose without a thought. That is why the developer Mario Westphal has felt it necessary to build a lot of tools and processes to harness it.
Here the Autotagger is configured:
Number of keywords are specified, formatted and handled both through predefined user interfaces and through three different AI-prompts that controls the AI-services output (Descriptions, Keywords and Landmarks.
This is how my Keyword prompt looks like:
(I have limited the maximum number of keywords above to eight.)
Prompt
[[-c-]]
Keywords never in plural form
Use simple English, common words, factual language.
Max one word in each keyword.
No geografic data or time info as keywords.
No words with only capital letters.
Write the specie of the animal, animal family name and scientific name in latin into Keywords.{File.MD.keywords}
If you cannot detect any animals, return.
Here you can totally control the behavior of this keywording-system and how the AI generated keywords are handled compared to your active vocabulary.
As you understand these tools are pretty unique to other softwares you might have seen before because unlike software like Photolab it is built with a real “holistic” engineering where very little is falling besides or between.
This is the “Thesaurus” or Keyword-list if you prefer that word and as you can see I don´t use any hierarchic keywords at all since I have handle that already via the keyword prompt.
As you can see there is an interface to support migration: the options to Import and Export Thesaurus. With the import from database you can import both all the plain keywords used on the pictures in the database or hierarchic ones if people use those instead.
To migrate the XMP-metadata to iMatch is a non-problem as long as it is OK for you just to let iMatch use the keywords-data that is stored in the JPEG-files embedded XMP or in the RAW-files XMP-sidecars. In that case you just import the pictures into iMatch and chose the function “Import from database”. That will build a new Thesaurus in no time based on the keywords imported to the database.
If you are lucky you can use the same way to do it even with other softwares but there are issues when it comes to hierarchic keywords in some of them. You have to chose what works for your special case in the Preferences og Photolab.
Maybe at least some people understand there are far better alternatives to use if you want to build yourselves a really good picture archive with very little effort and it isn´t even expensive! So the days when there was a need for writing SQL-queries to extract keywords from a Photolab-database or building your own keyword application are gone. It is just not worth it - unless you think it´s fun of some reason.
I´m fully convinced we will see AI-supported tools like the Autotagger in iMatch, even in RAW-converters with Photo-DAM solutions from other vendors than Photools in the future and if they are implemented they also have to take care of the keyword related problems that comes with it.
Thanks for this detailed explanation, iMatch looks quite interesting. I never used keywords myself and I wonder if keywords are even necessary at all with AI. I think instead many applications will include a direct text search, it is already included in Apple Photos, you can just enter a text description in the search box and it will list all the pictures that match that description. On the iPhone that does not work that well yet, but I guess that is the future with better models that run locally on the phone.
I can see the use for keywording for professional applications (which I would hope Photolab falls into), where you want to be sure not to miss any images, but for consumer grade apps I believe they can just skip the keywording step altogether.
You are right about the limited need for keywords outside local environments like a personal archive or in smaller companies up to global enterprises or for that matter organisations of all sorts. BUT, they are absolutely essential when it comes to a rational and effecive way to handle a picture library. You just can’t do without them. It is fantastic what a difference they make in a tool like iMatch for fast searches without the need for typing at all - just selections - where the keywords are used as so called “cathegories”.
Doing something like that in Photolab would not be hard at all since almost all of it already is there in the interface.
So, I don’t really believe in something spectacular will happen with the PictureLibrary in Photolab BUT that doesn’t really matter because no other RAW-converter is a better and more seamless match with thirdparty DAM than Photolab and THAT is an indeniable strength, advantage and sellingpoint for Photolab. If you really want to scale up scale up and get a really efficient XMP-metadata workflow all the metadata maintenance tools are there in iMatch.
Nothing gives a better image quality today than Photolab and it is a seamless and rock-solid match for consuming and displaying iMatch XMP-data that have supported all but one of the 25 XMP-metadata fields/elements that I use right out default. Just update and maintain the metadata in iMatch or even in PhotoMechanic if you have your XMP-metadata life there and Photolab will instant display the changes by its excellent synchronization.
This really leaves the metadata maintenace environments both in Photolab PictureLibrary, Adobe Lighroom and Capture One is the historical dust from an ancient time.
There are a lot of people out there these days looking for alternatives to Lightroom and a ever more gready Adobe so I would love to see Photools and DXO join in a good and mutually benificial market partnership by offering attractive bundles with DXO products and iMatch. That could be a really European killer alternative to a technically tired Adobe and CameraBits that developes PhotoMechanic.
After my latest posts in this DXO tread, it seems to have totally died. That was really not my intention.
I don´t know if people might have felt stupid about the level of that somewhat bizarre discussion that was going on here when I pointed out where the development-front around the keyword issues really lies today with iMatch and Autotagger as an example.
What’s even more bizarre is the fact that these discussions have been going on for probably 4 years or more here at DXO Forums, not the least around the problems DXO Photolab previously had with the handling of hierarchical keywords (it has not always been fully compatible with, for example, PhotoMechnaic’s handling of keyword data exchange between applications). So, it has far from just being a matter of being able to export and import vocabularies.
Mario Westphal the developer wrote a comment in the Photool community: “From the thread you posted, I learned what I already knew: Many people don’t know what they are missing. This applies to both RAW processing and Digital Asset Management (DAM). Users often express surprise upon discovering that other RAW processors produce superior results compared to Adobe Lightroom!
Additionally, many learn about the subpar state of metadata in their files and how improving it—through keywords, controlled vocabularies, descriptions, AI-generated content, and standardized, interchangeable metadata via IMatch and ExifTool—can be beneficial.”
I think that comment: “many just don´t know what they are missing” is spot on. Many of us including me have for long been stuck in a “bubble” where Photolab, Lightroom or for that matter Capture One has been our “whole” monolithic metadata management world and relatively few of us have taken the steps to scale up by using PhotoMechanic och iMatch and know what it really has to offer.
I also just find it astonishing what Mario Westphal just wrote: “I began working on AutoTagger in July 2024, delving into AI technologies and testing with OpenAI, Mistral, and Ollama. During this period of discovery, I encountered the dynamic nature of AI—vendors like OpenAI frequently updated their APIs to adapt and evolve, introducing new models that significantly enhanced previous offerings.”
So, he hasn´t even spent a full year developing Autotagger (about 10 months?) to the state and full power and control it now empower the users of iMatch with, together with at least one on the AI-providers that Autotagger supports today. I think that is both pretty unique and remarkable not the least when it comes to how brilliant all keyword related tasks are handled and controlled in iMatch 2025. … and again, this not really a question about the need for enormous R&D resources and big AI-centered development teams to achive this but more about one single developers brilliant skills och totally focused Holistic Engineering. Until now it hasn´t either helped Adobe with all their billions or anybody else, has it? They might still be dreaming of doing what Mario Westphal already have done.
With Holistic Engineering I mean how both iMatch Face Detection and the Autotaggers handling of the AI-driven image analysis through the four different promps together manage to write so often very well written Descriptions and even hierarchical keywords to the pictures and on top of that even managege to automatically manage our Thesaurus to our liking totally automatically regardless if we prefer using “controlled vocabularies” or prefer our own hierarchical keywords or just unstructured ones. Keywords with “Zero Administration” who saw that coming a year ago. Did you?? I guess most people could not believe that even would be possible. Today this is already solved and have become a “non-issue” for the iMatch 2025 users.
It will be very interesting to see how both Adobe and the rest will react and respond to this because from now on old excuses will no longer be excused.
@freixas has started the thread with sharing an idea and a few scripts for us to use or not use.
We could test the ideas/workarounds/solutions and say thank you or propose improvements/extensions or simply go our ways.
But it is quite common that human reactions revolve around whether something is good, useful, nonsense or whatever qualities people want to tag things with, often based on preconceptions or hearsay. This speeds up discussions … unless someone else will reiterate the process of adding a tag to the new response etc. and the reason that created the thread gets lost in the process.
Elephants can lift tree trunks, but when it’s about toothpicks, elephants are probably not the preferred solution.
Having said that, let’s return to the topic of the thread. We can always start threads about trees and toothpicks separately.
Let those, who want to discuss these, open a respective thread.
I don´t see what you see Patypus. Freixas has edited his first post three times and in the first version I saw something like this script which I have copied from Bryans post #16 :
FROM potentials
JOIN Items ON Items.Id = potentials.ItemId
JOIN Keywords ON Keywords.Id = potentials.KeywordId
JOIN Sources ON Sources.Id = Items.SourceId
WHERE NOT EXISTS ( SELECT 1 FROM ItemsKeywords AS ik WHERE ik.ItemId = potentials.ItemId AND ik.KeywordId = potentials.KeywordId )
ORDER BY filename, keywordpath
… and Freixas originally also displayed the result of the script which was a list full of hierachic keywords from the database of Picture Library and that is just part of a long history here at DXO Forums with all sorts of problems revolving around DXO:s poor implementation of hierarchical keywords and how to handle that.
These discussions about all the problems people have found and suggestions of how to work around these problems have gone on and on and on for I think at least four years. From the beginning it also touched non working mark-up with stars and/or color labels in our workflows. Joanna even ending up building her own application taking care of both stars, color labels and keywords that looks very much as a file viewer that we even have here in Photolab now.
My conclusion is that after all these years of endless keyword-discussions DXO hasn´t really listened at all and here comes new community members every year that keeps this eternal flame burning. Freixas contribution here is just another attempt to handle things that are not there and like Joanna he might end up building jet another parallell solution that doesn´t really change anything.
I just described where the innovation front lays today and that Mario Westphal during dedicated work under ten months has managed to build a complete solution that practically solves all these old keyword-related problems. So these old keyword issues have become a non-issue now since iMatch integrates very well and seamlessly with Photolab and solves all the problems Photolab don´t. Properly configured it gives Zero Administration even to the use of hierarchical keywords with and automatic maintenance of a Thesaurus too.
For the ones really interested in scaling up and establishing a really efficient Keywords- and Descriptions-workflow with practically Zero Administration even with Photolab - now you know where to find the tools to fix that.
For the ones not interested - I leave you here in this “Death Valley” of the keywords to find your way out by yourselves. Good Luck!
…which seems normal, different people see things differently.
