Search not working after images moved

roadcone · June 22, 2023, 7:55pm

I’ve copied my images and sidecars to a USB connected SSD (the SSD has a mount point but not a drive designation.). I’ve pointed PL6 to a database folder on the SSD and I have indexed the root folder for the images. After some hours, the databases sizes had increased significantly and the Keywords list in PhotoLibrary had been created and the quantities shown appeared to be plausible. But Search does not work. Previously, I started to type in the search field and found terms started to appear (folders, images etc) but now … nothing. Even if I navigate to a folder with known content then initiate a search that is there, still nothing. Filtering that folder (say, by colour) does work.

Does anyone have any suggestions please?

Clive

BHAYT · June 22, 2023, 9:24pm

@roadcone something is not exactly as it should be!?

Ignore keywords etc and try something simple like the make of the camera and see if anything materialises, i.e. in my case typing “pana” gave

‘Filtering’ should not affect the result in the table shown but will “filter” the displayed images if you click on the table entry, e.g.

where I am ‘Filtering’ on the ‘Label’ = “Red”

roadcone · June 23, 2023, 9:31am

@BHAYT I had already tried what you suggested after posting the question, but to no avail. I pointed PL6 back to the original database on the internal SSD and all was well but when I pointed it back to the USB SSD it again failed to search.

I deleted the database and started indexing again, this time by groups of folders. I have over a dozen folder groups - four contain the bulk of the collection with the rest containing up to a few hundred images. I checked after each index operation. It failed after I had indexed my old Lumix TZ60 folder structure. This structure is unlike the others. Capture One does not have a correction module for the rw2 files and does not even see/display them. Ages ago I converted a few to dng using the Adobe convertor, which CO can process. With the advent to Deep Prime, some time ago, I chose to leave the computer running over-night and convert all rw2 to dng then ingest them into CO. Process worked fine. So this folder structure contains pairs of files, same name but with rw2 and dng extensions - also there exists xmp files but I don’t know whether these attach to the rw2 or dng. I’ve opened a few and the odd one seems to have the following line:

photoshop:SidecarForExtensionRW2</photoshop:SidecarForExtension>

but many seem devoid of that line.

Things should not be this difficult … I seem to recall that before I started having image processing software problems (much, much more so with CO) I actually found time to go out with my camera!

Clive

BHAYT · June 23, 2023, 11:20am

@roadcone I am tired at the moment for one reason or another and my brain is “fogged” but first things first, how did you point PL6 at the various locations, i.e. the database on the internal SSD and the database on the USB SSD?

Here I have adjusted the ‘Preferences’ to use a database on a SATA USB3 drive

and PL6 has copied the current database to the SSD but is still using the database on the C: drive (a SATA SSD as it happens).

It will switch to using the database on the USB drive only after a restart, as shown here!?

The ‘wal’ and the ‘shm’ files are the “giveaway”.

So I need to wrap my tired brain around what needs to happen for PL6 to correctly adopt a database in another location because

when you change the database location it copies the existing running database to that location and
when you ‘Restore’ a database it copies the “restored” database file to the current location specified in the ‘Preferences’ and states that a ‘Restart’ is needed after which it will start to use the “restored” database in the location specified in the ‘Preferences’.

Watch for a PS or another post!

PS:- Statement 1. is only true if there is no DxPL database in the location already (which I knew but had forgotten)!!? If there is a database in “residence” in the chosen location it will remain untouched ready for use after a DxPL restart (which is not flagged as necessary by PL6!)

PPS:- When making such switches it is best to “park” DxPL on a “neutral” directory, i.e. one with no images or DxPL will start “discovering” the “last” directory used into the database to which you have just switched!?

BHAYT · June 23, 2023, 11:55am

@roadcone if memory serves me right CO even creates xmp sidecar files for JPGs or I never found the option to stop it doing that! DNG files should not have sidecar files, any metadata should be embedded in the DNG (I believe) only RAW images, RW2 and ORF in my case, should get xmp sidecar files unless the product being used can (and has been configured) to “embed” metadata in the RAW, e.g. Photo Mechanic can be so configured.

roadcone · June 23, 2023, 3:43pm

@BHAYT I did edit the preferences to change the database location. I found that when the internal drive database was active, and I pointed it to an empty folder on the usb drive, it did copy the active, internal drive database over (which I then deleted as I wanted a clean start), However, when the usb drive database was active, and I pointed it to the internal drive folder which already contained the ‘donor’ database, it did not copy the usb active database over but appears to have ‘adopted’ the original ‘donor’ after a PL6 restart. This was verified by checking and again by carrying out the reverse process when I changed the then internal pointer back to the usb database.

I have duplicated the TZ60 folder structure with all files and created two copies - one in a sub-folder titled dng and one titled rw2. I deleted the dng (and the CO session files to reduce space) from the rw2. I ran up PL6 and asked it to index the rw2 folder, which it did successfully. I have closed PL6 and reopened it and it continues to work.

I just have to go to CO and tell it that I have moved the files - and that will cause the database to become corrupted as it seems mostly intolerant of that process on my computer.

Clive

BHAYT · June 23, 2023, 11:31pm

@roadcone I apologise for not having got to the bottom of your original problem and I have limited time for further testing this weekend, I am off to West Wycombe near Bromley to help my oldest son with more work on their house and won’t be back until Sunday evening!

In the meantime I have discovered another weird problem!?

My “theory” was that you could do the necessary discovery, indexing etc. with the database on the main drives but the data on the USB drive and then “simply” change the location of the database to the USB drive, having made sure there was no database left on the USB drive!?

My first attempt appeared to leave the project I had created behind, i.e. it was not present when I started PL6.7 with the database on the USB drive? I repeated the attempt and the database on the USB drive (T: in my case) was smaller than the original and further investigation showed that the ‘Project’ was indeed missing @DxO_Support-Team!?

I am sorry Clive, because this should have been easy and straightforward and it is turning into a …

Physically copying the database from SATA SSD to USB SSD keeps everything intact but is less “elegant”, but seems to work.

Adding more data and projects while it resides on the USB SSD and copying the database back to the SATA SSD also retains all the data. In my case this has been done without the use of Mount points.

So

Empty DB on C: (SATA SSD)
Discovered directory on USB SSD
Added all images to a project
Set DB location to T: (USB SATA)
Closed PL6
Copied DB to T:, saving any DB present
Opened PL6
Indexed a directory of Nikon images.
Did a search and discovered nik images
Created a new project
Checked original project still valid, all present and correct.
Changed DB location back to C:
Closed PL6
Copied DB from T: to C:
Opened PL6 (DB now back on C:)
checked projects both all present and correct

danielfrimley · June 24, 2023, 6:21am

CO writes everything to XMP sidecar files, image files are not written to so that’s why Photomechanic chokes on edits to JPG/ DNG ratings and colour class made in CO - PM expects them to be embedded per standard so ignores them

CameraBits:

“For JPEG, TIFF, PSD, and DNG filetypes, Photo Mechanic reads and writes to embedded XMP metadata (IPTC/XMP Preferences) .

It will not read from XMP sidecar files associated with those filetypes, as that does not follow the XMP specification (https://developer.adobe.com/xmp/docs/XMPSpecifications/).”

Capture One:

From https://support.captureone.com/hc/en-us/articles/360002544898-Metadata-in-XMP-sidecar-files

NOTE: Above all, Capture One will never write metadata to source files; all metadata changes are done via proxy file.

BHAYT · June 24, 2023, 10:23am

@danielfrimley thank you for the update I remember reading somewhere that CO “tucked” everything into the sidecar file and that included sidecar for al image types. My only brush with CO was with a trial copy when testing keywords with PL5.

My first attempt to use the test data from my last test failed with ‘!’ on all images when my USB3 Bay appears to have failed, however, connecting to a rea USB3 port fixed that issue and I quickly made a test of the “missing” projects before packing the car for my 55 mile trip!

Using the update from the end of the last test I added three more projects to PL6 with the database on the C: drive and then changed the DB location to the T: drive (external USB 3 SATA) and found the latest 3 ‘Projects’ were missing but the original two projects were intact!?

@DxO_Support-Team it appears that some “pending” updates are not making to the new location!?

roadcone · June 24, 2023, 5:48pm

@BHAYT I’ve put all images and support files on to the usb ssd together with the PL6 database. I’ve added the same mount point to both the desktop and laptop and amended the Preferences on both to point at the usb ssd - and so far, it works OK though I have only accessed the files and poked around but not yet carried out any adjustments (though given that they get written to the dop, that should be fine).

Got myself in a right mess trying a similar trick for Capture One. So much so that I have had to delete the database and start the import afresh - I’m fairly sure it is operator stupidity. I started the long build of thumbnails on the laptop and have stopped it and move the disk to the desktop to see if it plays nicely and at the moment, it is - though I have managed to get database corruption on both the catalog and session but this is now routine for me. I think I am going to keep my head down now for a while.

Clive

PS - Hope the work on the son’s house went to plan.

BHAYT · June 25, 2023, 11:01pm

@roadcone Lets hope it continues that way.

A quick comment about

Those two files are a “giveway” of one of two things

The database is alive and well and in use, typically by PL6 but also by DB Browser for SQLite in my case (when testing).
or PL6 has (or has been) terminated unexpectedly and these files are left to be found the next time it starts.

The absence of these files indicates that the database is not inuse or was closed correctly by PL6 and under these conditions it is safe to copy PhotoLab.db.

The situation of the “missing” ‘Projects’ from the PL6 moved database indicates that some updates are not finding their way into the PL6 copied database, which I consider to be very “messy” coding.

However, I am not an SQLlite expert so I am unclear about what might be necessary to ensure the database in its complete state is copied except maybe a close, followed by a re-open, followed by the copy might do a better job, i.e. when I closed the database and the re-opened and moved its location a database with all 5 projects present and correct was the result!

I travelled up on Saturday (about 55 miles) and we installed two doors, one weighing 41Kg and the other 46Kg, taking the doors from the garage to the first floor bedrooms! I stayed overnight and we installed two more 41Kg doors and then I had to help dismantle a scaffold tower to move it further along an outside wall and re-assemble it before I could leave for home!

I left my son up the tower painting the brickwork! The target was four doors so we met that target, but they sure were heavy and my son did most of the lifting!

Regards

Bryan

roadcone · June 26, 2023, 9:08am

@BHAYT I’ve realised that the usb ssd is working between the two but I have not (yet) set up all the Projects so can’t yet be sure.

I had noticed the two additional files in the database folder. When I was indexing by folder and copying the files at each stage I did so without closing PL6 and noted three active files. After closing PL6 I noted just the one so concluded they were temporary.

I was walking yesterday in a very hot Derbyshire - and I’m sure that beats door-hanging!

Clive

BHAYT · June 26, 2023, 11:05am

@roadcone The WAL is a Write Ahead Log file and I believe that my adding projects and then changing the location of the database to one that does not have a database already, results in PL6 copying the database or rather most but not all of it is the only situation that could result (and does in the tests that I have run and repeated a number of times) in the “loss” of data, i.e. there may be pending updates that get missed which is “messy” to put it mildly!

Possibly but the outlook from the house is over fields and it was very hot but a strong wind got up while we were moving the tower! So I left my son to continue painting “Windy Corner” and headed for home, 55 miles South!

roadcone · June 26, 2023, 7:51pm

@BHAYT This is driving me nuts. I had got everything working really well on the desktop (database et al on the usb ssd) and when I plug it into the laptop it starts generating virtual copies for virtually all my images. And where virtual copies already exist, it creates a further copy for both the master and each of the virtual copies. (insert series of very rude, very short words).

So I conclude that PL6 cannot run from two computers/installations on to one usb disk, called the same thing on both computers, with one set of images/dops/database. I’m mystified but it has shot my plans completely now as that suggests that a network copy of the images etc. could not be run from two computers (though would work with one computer of course).

I set up Albums on the desktop use and I am wondering if that is confusing things. Bugger.

[Edit}

I’m thinking it is to do with the Albums. I have not done any adjustments in this phase of moving things over so on the usb disk running on the laptop, I have deleted all Albums then copied over all the dop files from the laptop internal disk and will check that on the desktop in the morning. This sounds drastic but bear in mind that until all is proven well, the laptop internal disk remains the master and the usb a working copy.

BHAYT · June 26, 2023, 9:46pm

@roadcone The most obvious cause for a Virtual copy per image is a mismatch between the image in the database and the DOP. Every DOP has a UUID and every database entry also has a UUID.

Providing they are the same then nothing untoward happens and the date/timestamp will govern which is considered to be the latest edit (or so I believe). However, if they do not match then the database entry will be treated as the [M]aster and the DOP as the Virtual Copy [1].

I believe I have encountered situations where something similar happened for a reason I could not determine but the most likely reason is a UUID mismatch! One thing to check is whether there are any apparent changes between the [M] and [1].

The problem then is that you cannot easily replace [M] with [1], i.e. swap them over en masse and deleting [M] will automatically delete the VCs!!! We have requested DxO do something about this but …

You can identify all the [.] VCs and delete them using a ‘Sort’ but not a ‘Filter’ After that all the DOPs will be aligned with the database and the UUIDs will tally but you risk losing edits this time around, and potentially worse, every time.

So I/we need to understand

Have you encountered the “classic” Unwanted Virtual Copies situation or
Encountered some other problem

I have done it in the past but I need to understand what actually worked for me and what “seemed” to work but created new copies in the database. So please do the following, since although I know what would need to be done and have considered writing a Python script that isn’t going to happen any time soon and we need to understand exactly what has created the situation in your case so that I can try to recreate it on a small scale and determine why you have encountered this problem!

Please do the following, when/if you have the rime:-

Write down the steps that led to this problem, including copying images from A to B, B to A etc. etc.
Please indicate exactly how you would like to process between the two systems once we have determined the exact nature of the problem, my biggest problem is remembering exactly what tests I have conducted and what problems I encountered!

All my posts are in a database which can be searched but minus the snapshots, i.e. text only with a URL to the actual post.

So I need to find the best fit to the problem (but it will boil down to a UUid mismatch but is that genuine or is DxPL actually comparing the wrong images)?!

I then need to find the best fit to the required scenario and make sure it is going to work the way you expect or want or find acceptable!

BHAYT · June 26, 2023, 10:08pm

@roadcone Simple Unwanted VC test

Intitial data imported into PL6.7.0

With DxPL closed I amend the second DOP UUID to

and when I re-open PL6.7.0 I get this

The problem is how have you got the wrong UUIDs in your DOPs or why has the comparison failed, i.e. what is PL6.7.0 comparing that results in the test failing, the former is the most likely scenario!?

If I use this

then all the VC [1], VC[2] etc will cluster together and can be deleted but deleting the [M]aster will mean all VCs for that image will also be deleted.

There is no facility to swap [1] with [M] or to promote [1] to [M] except one image at a time and we have asked DxO to provide such facilities but it does get extremely tiring when you bang your head against a brick wall again and again and … just as I have done there!!

EDIT:-

Set up a directory of 2 images and one of those had a VC, i.e. used from the tests above on the H: drive (a conventional HDD)
Transferred the database from a SATA SSD (C:) to a USB SSD (T:).
Copied the test directory from HDD (H:) to USB SSD (T:) using DxPL to create the directory and copy the images.
Closed USB SSD and disconnected from Main machine
Connected USB SSD to Test machine
Transferred database to USB SSD (D: on this system) and closed and restarted DxPL
Navigated to the test directory on the USB SSD (D:) and data appeared as expected.
Added 1 VC to first image and another VC to second image and closed DB and USB SSD on Test machine
Transferred and reopened on Main machine as USB SSD (T:) and all appears to be present and correct with no additional VCs!?
The original HDD data (on H:) is O.K.

The USB3 SATA Data (on T:) is O.K.

roadcone · June 27, 2023, 6:32am

@BHAYT I know clearly the process I went through.

On laptop: Copied all images with sidecar dop (and CO Settings and miscellaneous Afinity Photo .aphoto files) on to usb ssd. Via Windows virtual disk management, pointed laptop PL6 at:

c:\usb\sandisk\dxo\database

where \usb\sandisk is an empty folder in the laptop root and \dxo\database\ is the folder structure on the sandisk ssd. Restarted laptop PL6 to check all was well, and it appeared to be so.

[Aside: my image folder structure puts all images within My Pictures folder then camera name then camera model then date (a good idea when I had two cameras and very few images is now not so good) - so a typical structure would be:

\My Pictures\Nikon\D90\2018-01-01 London Trip\ …images…]

In PL6 Projects I had hoped to ‘recreate’ the ability to show ‘all images’. The plan was to have a ‘half-way-house’ to my structure by having the following Project Groups or Projects:

Collection
Collection - Nikon
Collection - Nikon - D90
Collection - Nikon - D610
… etc
Collection - Olympus - E-M1MkII
Collection - Olympus - E-M1MkII IR
… etc

Then, ignoring shot dates, using XNViewMP, drag all D90 images into the D90 Project and so on for each of the not-so-many cameras. I did this on the laptop for my old collection of Bridge Camera images and this worked absolutely fine.

Close PL6 and move to desktop. Pointed desktop PL6 at same NTFS empty folder as for laptop. Opened database and all appeared OK except, oddly, Projects appeared absent. I recreated the Bridge Project structure and dragged the images in using XN and that was fine, so took a copy of the database ‘just in case’. I repeated with the 7K+ collection of Lumix TZ60 and it caused PL6 to close half way through. I opened Pll6 and it did its ‘rebuilding database’ thing. I allowed it to do that but then renamed the database and took a copy of the backup from before the incident and renamed that. I opened PL6 and it was fine, showing the state at the time before it crashed. So I dragged the TZ images in, this time with no incident. So I rinsed-and-repeated the drag, exit PL6, copy database for at each further stage and completed with no further incident.

At the end, I clicked on Project Group Collection and went for tea. On return it had populated the screen with 48K images and the disk activity was nil. I exited PL6 desktop with no incident.

I opened the ssd on the laptop and it all went horribly wrong with almost all folders I checked containing M and 1 and where a 1 already existing, it then contained M, 1, 2 & 3. It was all instantaneous so I do not think the laptop created them but rather believe that when I clicked on Collection on the desktop and went for tea, the high disk activity was PL slogging through the collection and duplicating all dop files.

As \i have said, I have now replaced all dop files on the ssd with those from the master laptop internal drive but prior to do that, I have deleted all the Project Groups and Projects from the ssd database. Next I will open on the desktop and see if that and resolved the matter.

Enough for now - I’ll read your other email when I have taken fuel onboard.

Clive

roadcone · June 27, 2023, 7:11am

@BHAYT I don’t think I fully understand the uuid thing. I have looked at three jpg dops on the ssd and the final entries on all three show a uuid - in fact, all three show two uuid - and on the three images, none of the uuid is duplicated … so I have six uuid in total. The three jpg files were all shot by someone else at the same time. Here is a copy/paste of the final lines from one of them.

Overrides = {
}
,
Version = “17.3”,
}
,
ShotDate = “2018-11-08T13:27:16.0000000Z”,
ShouldProcess = 2,
Uuid = “5BC120AA-0724-4618-AF7C-32AAF5D64745”,
}
,
}
,
Uuid = “45C8B495-0097-4066-8DF8-B8AD021CE752”,
}
,
Version = “17.0”,
}

BHAYT · June 27, 2023, 7:50am

@roadcone There are a number of Uuids in the DOP but the one that needs to match the database entry is the one after “ShouldProcess = 2” in your example. If an image has no VCs (that you or DxPL created) then it will be located towards the end of the DOP as in your example.

So every copy, [M]aster or [1] VC, in the DOP will each have an entry in the DOP with its Uuid and this must match the Uuid in the database that corresponds to that image (copy).

If/when the Uuids fail to match then DxPL cannot decide which to use so it retains the database entry and creates an additional VC, either going from a [M]aster only (which has no special designation), to a [M]aster plus a [1] VC or adds additional VCs if a [,] VC already exists!?

That combination can only be adjusted manually on an image by image basis by copying the edits from the VC to the master and then discarding the VC!

So the answer must be to avoid creating a mismatch in the first place at all costs! I need to go through your procedure to see where you created the mismatch but you almost certainly did (I believe)?

It is possible (but generally not desirable) for ‘Projects’ to be present but the images to be absent but to have no ‘Projects’ at all seems to indicate that you opened the wrong database somehow (notwithstanding the issue I discovered in the earlier posts)!?

I am concerned by the huge ‘Project’ sizes you are creating but that should not be the problem!

I will post this now and see what might have gone wrong but we may need you to realise the issue as much as me trying to get my tired old brain to spot it!?

Regards

Bryan

BHAYT · June 27, 2023, 8:29am

Which database and where was it located.

The database location on the desktop must point to

If that database has been genuinely and successfully populated by the laptop so that we should be using the same database with entries that correspond to the DOPs residing on the disk which you should then be able to “re-discover” on the desktop and your ‘Projects’ should have been all present and correct.

Looking through the rest of your description it appears that you rediscovered the images and added them to a database on the desktop (?).

Re-discovering them when you switched back to the laptop and the original database arguably should have worked because it is a variant on my round tripping of images between two machines with two databases, i.e. when I took images with DOPs to a second machine (from A to B) and discovered them there no new DOP Uuids were created (unless a Uuid was already in use on B) the Uuids of the DOPs were used on Machine B.

So those images with their DOPs could safely be brought back to to A and the Uuids would still match the database entries on Database A!?

The strategy I was using in the A to B to A to B to … is fine but useless for any scenario were ‘Projects’ need to move (be retained) with the images!

So I can see where things might have gone wrong and the clue is in the missing ‘Projects’ but I would not have expected every DOP Uuid to have been changed if re-discovered on the desktop (D).

Rather than “play” with the whole library please take a handful on images from a numbers of directories and place them in simple folder structure of Test/1, Test/2 etc… on the laptop.

This is a mostly “Blank” database

PL 6.5.0 - Blank DB for Testing.db (71 KB)

So find a home for it so that you can use it like this

but always create a backup of your current database before you use it!!

I will continue after breakfast!