Your database is corrupt, so we’re going to create a new one…
After this, PL loads, all my photos are there, those I’d added star ratings to have those ratings, those I’d colour-coded are colour-coded, so…
So what?
What’s the database for?
PL8 (per screenshot) on Win11. Database being stored on external USB (same place as the photos). USB was connected and without any issues - I had Windows Explorer open and browsing exported image files on the same USB before starting PL - which is when the above error appeared.
@oatleyphotography Why your database was corrupted I do not know but your edits are help in a DOP file which resides in the same folder as the images, so DxPL will look for edit data in the database or the DOP.
The database contains additional data on both the Mac and Windows version namely any ‘Projects’ you may have created and the various bits of metadata that has been collected in the database which can then be searched and both will be lost if the database is removed.
On the Mac the ‘Advanced history’ is also maintained in the database and available from session to session to session, on Windows that only lasts for the current session anyway.
Some users choose to throw the database away after each edit session but (I believe) that the database came first followed by the DOP so you will always have a database but can choose not to save the edits in the DOP (not a strategy that I personally would recommend).
Hope that helps.
Actually they are not in the new database until you (re-)index the directory or re-visit the directory. So every time you visit a directory that is not in the database, e.g. after a failure, the database will be reloaded with the data from the DOP!
Never used the database in 6 years with PL, only NEFs + the DOPs in the same folder. And I have moved those folders, copied them and renamed their path several times. And it still works.
To me, database is USELESS and only wastes disk space.
@Lucabeer Then join the group of users that deletes the database before each edit session but don’t expect DxO to remove something which I believe is tightly embedded in their product because I don’t think that will ever happen.
Much of what I know about the workings of DxPL come from analysing the database and watching how it changes from edit to edit. I a was not then nor am I now an SQLite expert but was a database consultant for DMSII (a Buroughs/Unisys database product) for 36 years and the DxPL database is a simple one to understand.
If I only had the DOPs to go on my understanding of what is going on would be much more limited than it is. However, with the DOPs you have a portable store of your edits (but not the edit history) which can be taken from system to system or used for recovery after you accidentally delete the database or DxPL ties itself in knots or …
From my experimentations, the database is mainly useful for indexing and caching, but it certainly isn’t necessary if you use DOP files.
My gut feeling is that the code of the database has been kept “just in case” but mainly because no-one dares touch it in case it breaks something, because most current developers weren’t round and don’t understand it enough to know what is still necessary or useful and what can be ditched.
The database collects the metada, the keywords and the projects. If you want to use the Search feature, you need the database. Otherwise, DPL would have to scan all the DOP files each time you start a search request.
Mine is the reverse, it is the foundation around which the product was developed and will remain so until the product is no more. That means that your statement is also correct, i.e. that the task of removing the code would be monumental and as far as I am concerned a complete waste of a scarce development resource.
But why bother, if the users doesn’t want to use the database then delete it after or before every run and for those of us who actually want to see it enhanced with a better search, the storage of advanced history (which could also go into the DOP but would make it somewhat bigger), the storage of ‘Projects’ which is actually in the database but should also be in the DOP.
Actually, for the Mac version, the system looks after indexing metadata, keywords, etc. All you need is to use the macOs Spotlight database that is constantly being updated every time you edit a file.
But, since Microsoft still haven’t implemented such a system, DxO is stuck with having to maintain the database just for them.
@Joanna You say that as if it is a particularly onerous burden to maintain what is an essentially trivial database and how many other Photo editors and even Photo viewers do you know that don’t use a database of one sort or another?
You used to proudly boast that your metadata product didn’t use a database but it does it is just that it is a component of the OS.
Sadly, if you had embraced SQLite you could have created a product that we could have used on Windows as well.
It might have increased your development time but you would have had a much larger userbase by now and I could have taken great delight in trying to see if I could break it!
I did read that thread, but although it described seemingly duplicated rows, without seeing the data itself, it isn’t clear whether there’s a legitimate reason or not for those duplicated rows … and just knowing there are rows representing images didn’t really help me understand why there was a database at all … hence this new question.
Are you saying that on Mac, PL8 relies on “spotlight” - a Mac OS feature - to track edits to files? If so, are you saying the database in windows is supposed to be used for tracking edits to files, and that PL8 has to maintain that edit history? If so, then what are the sidecar files for? Everyone is saying the edits are stored in the sidecar DOP files.
Perhaps I should have asked: where is the PL8 documentation about its database file? I mean PL8 gives you options on where to save the file. Surely there is documentation explain pros and cons for your options?
If we consider the DB to be just a cache, we can calmly trash the DB whenever PhotoLab is NOT running.
If we use DPL just as a developer to any asset management app, we can trash the DB anytime when PhotoLab is not running. And we don’t need to store the .dop sidecars in that case neither.
If we consider the DB to be part of DPL’s asset management - DPL’s search is based on the DB - we want and need the DB.
Whether we need/want/hate the DB depends on how we work. The sad part in this is, that DxO has not yet presented any database maintenance features. Use case #3 is therefore not reliable enough.
I used LR4 for a long time. Sometimes it would say the “catalog” (read: their db) encountered an error and needs to be fixed. You’d wait a few minutes then LR would restart and everything was fine.
I was surprised that the PL8 error was “we’re going to create a new one”.
I think proper documentation on what the database is for, how it is used, what sidecar/dop files do … would go a long way to satisfying my curiosity. I have yet to search the web properly - came here instead to ask the question … and am getting many varied descriptions … but docs from dxo would be great!
I might try what others have done - do view the database file using some database application and see if that explains anything.
DPL has no database maintenance tool. Fortunately, fixing the database when some problem occurs or cleaning it up is easy with an external tool. Often, these will be no need to re-create a new database.
Please see this thread :
SQLite Expert is a Windows tool but you can easily find an equivalent if you are running Mac OS.
It’s not just the maintenance that is the problem. Apple took a good couple of years to get the Spotlight indexing from a place where the whole machine slowed down to a point where, now, you don’t even know it is indexing.
Indeed. But, as I’ve just said, the Spotlight engine takes advantage of an extremely efficient background indexing behaviour that doesn’t require explicit invocation, or setting up background threads with all the slowdown and synchronisation errors that can produce.
To write an app that would work on both platforms would require being familiar with the APIs and mechanisms of both platforms, as well as having to create two separate UIs based on two different UI libraries. Plus, it would mean writing code in C# for Windows and Swift for macOS. It took me more than two years of fourteen hour days, seven days a week, to get the one version done.
Why do you think DxO takes so long to implement requested features?
So far, my Mac project contains just over 35,000 lines of code. Start again for Windows with APIs I have long forgotten? I think not
I am not privy to exactly what DxO do but I’m pretty sure that, because a Spotlight-like mechanism doesn’t exist on Windows, they decided to write a cross-platform SQLite database that would work with either, rather than separating out metadata persistence to Spotlight for Mac and SQLite for Windows.
The database in both versions tracks edits but, to my way of thinking doesn’t have to track metadata used for searching. That can be separated out.
Because I have written my highly efficient indexing app, I never have need for the DxO database. I have never found a need for an editing history and my app creates “smart folders” based on search results, as a project management mechanism.
The database implementation is, essentially, a black box that you are not meant to delve into. As @platypus has often mentioned, there should be a managed access to it but, unfortunately, nobody at DxO has yet bothered.
Ah! That approach would explain so much about why stuff takes so long to get implemented.
Mind you, one of my consulting clients once wrote code to calculate sales tax, five times - and each one did it differently
Stenis
(Sten-Åke Sändh (Sony, Win 11, PL 6, CO 16, PM Plus 6, XnView))
20
The maintenance feature you are longing for is already there really and it is called reindex.
I just select my topfolder for all my folders and files and run it - problems fixed.
The database is good in a few respects and it serves me as giving me an overview always present that shows my metadata is OK.
Otherwise the databases most severe problem is that it lacks usable efficient tools for the everyday metadata management and it also seem to have problems maintaining basic dataintegrity.
The good thing though is that nothing else I have seen integrates as good as PL with Photomechanic or iMatch.
