Yes. As I wrote above, the support had written to me with another problem that the DOP files are not necessary. So I switched it off.
Ah! Yes, .dop files are not necessary if the database keeps working. If it doesnât, the .dop files are something like a distributed/granular backup. They arenât strictly necessary if one keeps up-to-date backups of the database.
The bonus of sidecar files is in the migration. And you can still reclaim the sidecars by doing the following:
- if only a few files have been touched, select the folder and export sidecars manually with the respective command from DPLâs menu.
- for bulk export of sidecars, use search to find files and export sidecars manually Search is based on the database and presents results that can span several folders. Caveat: There is a limit in DPL on Mac: Only 1000 files will display, so one needs to refine search to circumvent this limit. It might be absent in DPL on Win though.
Note: Tested bulk export on Mac. it works (and macOS can easily remove the sidecars again)
!! 
!! ⌠Not absolutely essential, perhaps ⌠but, highly recommended, regardless.
1 Like
@John-M They are absolutely essential, and the DxO support person who suggested that they werenât should be made to read the forum as punishment and for enlightenment and to discover what DxO should be doing to make the product fit for the year 2025 with respect to data handling.
Without DOPs then there is no recovery if the database ties itself in knots, not a common problem but (k)not unknown.
Database backups are also essential but losing one edit session is frustrating for some users and a potential business disaster for others, i.e. when returning to the last Backup made.
Database backups are fairly quick and could be made numerous times during an edit session, each with a different timestamp, sadly not automatically provided by DXO. i.e. the DxO designers obviously decided that backups would be a daily thing so they add a date automatically but that should include a timestamp i.e. Backup_2025_03_08 _1114 will do (possibly just add the milliseconds for completeness) but not just Backup_2025_03_08.
I am being âpickyâ I know but in my own coding one of the first things I wrote was a time coding Procedure including milliseconds and appropriate output for logging and for file naming.
The DOP offers a continual audit, albeit of only the last thing you did to an image, which is a maximum of about 20 seconds behind the actual edit made by the user.
The limitations are that it doesnât record and cannot recover Projects (Mac & Win), Advanced History (Mac only, never retained on Win beyond the current session, i.e. it is in memory), and has issues with DCP it appears (not my domain at all). But at least it provides the starting point for further editing rather than leaving the user back at the original image!
@platypus I seem to remember that there is also a 1,000 response limit to searches in DxPL(Win) as well but havenât retested.
Your suggestion is a neat trick to overcome the directory boundary limitations for those users who âonlyâ take less than 1,000 images per shoot (which would include me I typically take between 350 - 500 per visit to a garden).
Search is not limited to âshootsâ; depending on what is in an archive, searching for e.g. DNG files or CR2 files taken in 2012 can yield hit lists of almost 1000 files which then can get the dop files or updates thereof.
I donât use many DNG or TIFF files and was able to add dop files to these with two searches. Searching for CR2 files is vain though, too many of these, but combined with some additional search terms, the hit list can be pruned to just below 1000. Anoying though.
Re-did the test with the database edit and went all-in and now, all the image files have their .dop sidecars. The 29.5 k files were added in 5.5 minutes.
Hereâs the feedback from the database for the SQL (3 lines in capitals) statement that set the flag where needed:
Execution finished without errors.
Result: query executed successfully. Took 38ms, 29569 rows affected
At line 1:
UPDATE ZDOPSOURCE
SET ZSIDECARNEEDSTOBESAVED = 1
WHERE ZSIDECARNEEDSTOBESAVED = 0;
You could see how the sidecars roll inâŚif attachments could be .mov files.
Now, letâs trash all those sidecarsâŚ
Iâm not sure I understand what this is really supposed to do.
In my Windows database the field sources/SidecarNeedsToBeSaved is simply empty (not 0, no 1 itâs NULL)
In my MacOS database - some entries have an 1 some 0 and some NULL
The goal of re-testing was to, with one action, export .dop sidecars to all files registered in the database in order to prepare a Win-to-Mac migration. (As of now, the only way to transfer edits and metadata is through the .dop sidecars)
Based on the database attached by @BHAYT a few posts above and based on what I found with DPL on Mac,
- the field âSidecarNeedsToBeSavedâ is set to â1â, if DPL has registered changes (edits and metadata) that should be written to the .dop sidecar whenever DPL is dealing with updating those files. If the fields are set to â0â, DPL has written the sidecars. My guess is, that a NULL value means that DPL has had no interaction with these files yet.
- indexing folders forces DPL to export/update sidecars if âSidecarNeedsToBeSavedâ is set to â1â. Indexing (and writing sidecars) can cover nested folders, therefore I can push DPL to update all sidecars (or create them) within the scope of the folder selected for indexing. In my test, the 29â434 sidecars were written within 330 seconds, a time that is short compared to the search/find or any other method.
Now, if my procedure worked on Windows, you could make sure to give your database on Mac the best head start because all image files have a dop file that contains all the information they can store. Itâs not too complicated imo, but your mileage might vary. But if you have spent some effort in developing and keywording your photos. Whether you want those changes on your new computer or not is entirely up to you and weâve already outlined to possibilities that leverage features out of DPL, be it with low profile search and store or a somewhat more involved database manipulation based procedure. Whether you chose one way over the other or decide to just copy over the image files (and sidecars as present) depends on how many images you have to deal with. The bigger the numbers, the more youâd want some automation.
@RobiWann I had stated my scepticism about that field principally because I have never seen it with any value other than âNullâ on my Win 10 system, albeit I havenât spent much time studying it either.
My tests, both those that didnât shut down the database and those where I did, left the field as I had set it, namely = 1, and didnât create any DOPs.
Hence, my belief that DxPL(Win) simply doesnât use that field!?
@platypus I stick with my conclusion based upon following your procedure and nothing whatsoever happened.
I am glad it worked on the Mac but I feel it wonât work on Windows, or rather I am sure it wonât.
PS:- I am surprised that the database is used to progress the saving of DOPs, although useful a series of writes to the database just to flush out DOPs to be written isnât really the most efficient way of doing things.
Plus it would be ultra inefficient on Windows because the database has no index to locate the entries!
The field on the Mac is an Integer
CREATE TABLE IF NOT EXISTS "ZDOPSOURCE" (
"Z_PK" INTEGER,
"Z_ENT" INTEGER,
"Z_OPT" INTEGER,
"ZMETADATANEEDSTOBESAVED" INTEGER,
"ZSIDECARNEEDSTOBEREAD" INTEGER,
"ZSIDECARNEEDSTOBESAVED" INTEGER, <----------------
"ZAMBIGUITY" INTEGER,
"ZDAMBODY" INTEGER,
"ZDAMLENS" INTEGER,
"ZMETADATA" INTEGER,
"ZPARENT" INTEGER,
"Z5_PARENT" INTEGER,
"ZNAME" VARCHAR,
"ZUUID" VARCHAR,
"ZCAFID" VARCHAR,
"ZCAFLISTVERSION" VARCHAR,
"ZMETADATADIDFAILTOLOADVERSION" VARCHAR,
"ZAVAILABLECAFIDS" BLOB,
PRIMARY KEY("Z_PK")
But on Windows it is a Bit field
Plus there doesnât appear to be an index on the Mac either.
No worries here, it simply means that DPL on Windows uses a different way to manage sidecar updates. Iâve meanwhile sent a ticket to support in order for them to create (or think about it, at least) the necessary capabilities for platform migration and using DPL on more than one computer⌠Iâd be positively surprised if weâd see results within less than three years.
I think that weâve been doing what can be done with what we have, know or found out. And weâve delivered a few ideas to help the migration. Itâs now up to the OP to decide and do whatever suits best ⌠and to write a few lines about it, so that others might learn something.
1 Like
@platypus That quick!?
1 Like