Open individual file rather than directory - impossible?

BHAYT · December 21, 2021, 8:00pm

@Stenis thank you for your post supporting my “old way” of doing things.

However this is slightly off topic and started because I commented on the size of the directory that @md999 was working with. I tried not to be judgemental in my comments and what suits a user is what suits that user and it would be boring if we all trod exactly the same path, but perhaps I should have made no comment at all and just tried to find a way round the apparent situation of 10,000 files being opened by PL5 with no option available.

With respect to my comments about XnView MP, I am now up to over 7,000 RAWs in one directory and XnView MP seems to make mincemeat of scanning them and putting up the thumbnails! It is also possible to drag and drop the tagged files from XnView MP to PL5. The reason that I looked at the products I suggested was to provide a reliable mechanism for loading PL5 with a restricted number of selected photos as cheaply as possible.

I have only ever used Photo Mechanic (not the Plus version) and while it is a pleasing product it comes with a hefty price tag. Nevertheless, it is good that the integration between PM(Plus) and PL5 is as good as you describe.

FRV stops when it is able to fill the screen with thumbnails (I have a slightly noisy disk and no side on the computer so I can detect disk access that way) and continues whenever the thumbnails are scrolled. However, there are a few parameters on offer in this area!?

Even more off topic here is a sample of the wallpaper being generated every 3 minutes from the 1920x 1430 reduced size library of all my photos where I am free to play “guess the garden”. Two of my favourite gardens are included here, Great Dixter and Wollerton Manor and it is a challenge to remember some gardens particularly if you have a picture of just one flower!

So here we have our garden, Greate Dixter, Leonardslee Gardens (been shut for years that’s why I didn’t get it), Regents’s Street, Highdown Gardens, Cranbourne Manor, our back garden, Wollerton Manor and Nymans Gardens (missed that one as well) - must do better!!

!

JoJu · December 21, 2021, 8:00pm

@Stenis wins, sorry @BHAYT : 583 words against 452.

Whenever somebody needs a lot expanlations to make the point “my way of organizing images works best for me”, I just nod and think “everybody needs to find a way which works best”.

Both of you were simply posting their way, please keep it, I won’t take it away. And both used so many assumptions and guesses - please keep and enjoy them.

1.035 words written without a single question mark clearly shows: you found your way and have neither questions nor doubts. Great. Really? I do need to solve some of my questions, but not with your answers, if you don’t mind.

EDIT: Now @BHAYT catches up with 382 more words. And the first question mark, congratulations! Makes a total of 1.417 words. You both should start writing books…

Stenis · December 21, 2021, 11:15pm

Lovely Bryan: I understand you are brittish. You have far better climate than we have for gardening but we can always enjoy your work in Brittain on TV

There are a number of tools that are better than integrated RAW-converters like Lightroom to manage images with speed. It would not be a problem to handle tens of tousands of images in one folder only because when they are indexed only the index is used for searching. It will only affect the indexing time but generally in large DAM-systems they do avoid more than 1000 files per folder. When 1000 is reached the system automatically will create another folder.

If you don´t believe me when I´m talking numbers like this just open the "Recycle Bin in Windows after neglecting it some time or if you open this folder with 7000 images in the File Explorer

Stenis · December 21, 2021, 11:39pm

JoJu, I reacted mainly on md999 who started this tread with a complaint about how slow Photolab 5 was when trying to open and render previews for 10 000 files in one folder. I guess if you rather would like to stick with that folder structure (like some users seems to do) than breaking it up in subdirectories, you have to get something else than Photolab if you are not satisfied with the performance! You can´t get both without changing something in your workflow.

Once I was really was a system developer (for 25 years or so) and have written many manuals :-), to answer a few of my users questions. I have got most of my question marks in life that way too? People using software always use to come up with a lot of questions and that goes for me too but I think I have posted most of them the last year to Camera Bits. They have got most of my feature request and they use to include quite a few question marks too. Integration between softwares are often complicated and not so straight foward all the time.

Sometimes it takes quite a few words to explain complicated integration processes and a rising problem in my country is that people that passed the school system in the eighties and later have problems reading deaper texts, understanding them and following a tread. I was once a teacher too for almost 10 years and worked in our elementary school with teen agers. Maybee for example the brittish or frensch school tradition has succeded better than our school system. I happened to work then so “Mea culpa” it´s all my fault

That said I really think the integration between PM Plus and Photolab is really working efficiently and well. It´s not for all but if the price is something to stop people I wonder really if they value their time. I understand though if some users think PM Plus has a too steap learning curve but I think the 220 U$ I spent on it is a piss in the Nile and the best spent money I have spent the last years when I consider how useful it has been. My new Sony A7 IV costed 30 000 swedish crowns and that´s even more than the 2500 U$ it might cost in the US. That is at least 10 times more than PM Plus costed. I guess the 220 I spent on PM Plus is of far more use than the upgrade från A7 III to A7 IV.

md999 · December 22, 2021, 6:29am

OK, so I have tried to Photomechanic plus to open images in DXO as that is the database (PM plus) I am currently using. It works or at least it worked for 3 images but then on the forth image it opened ok but then kept opening that same image whatever other image I selected. So working together nicely is not what I would say at the moment. It opens that same image irrespective of whether I close DXO down or not. Incidentally since I have only recently started using DXO I am not familiar with how to actually close an image as there is not an obvious (to me) button to do this, on the other hand for the first 3 images each was automatically closed when I opened the next one via PMplus, this seemed a bit worrying if I had spend ages working on the image and suddenly lost all the work without warning or I wanted more than one image open at a time.

BHAYT · December 22, 2021, 9:04am

@md999 I am sorry to hear about your issues with PM Plus.

My directory is now made up of the combined JPGs I copied merged with the RAWs and is now 11,954 photos in total. I changed its name to avoid any “cheating” by the software and opened XnViewMP and navigated to the new folder (name) and off XnViewMP went importing all the thumbnails and any data that went with them, it seemed to run at a reasonable pace until about 45% (@ 2 minutes) when it started to slow, the next 10% took 1 minute it slowed further by 77% and almost stopped at 87% but got there in the end!

It took 12 minutes to add all the photos to its Catalog, currently residing at its default location on my C: drive (a 240MB SATA SSD). The photos are on a 6TB Toshiba 7200 drive and occupy 175GB. Processor utilisation for XNViewMP was 12-13% (I7-4790K @ 4.4, 4 cores + 4 threads) I/O about 17-18MB/s.

Reviewing the catalog in XnViewMP ‘Settings’ showed a direct import capability so I purged the Catalog and started again and the import took 9minutes and 30seconds but with no indication of any % at any point.

Refreshing the thumbnail view takes 10 seconds and how often that will happen depends on your navigation. At first sight it appears to provide the submission of multiple images that you require. The tagging function can span folders (just the one in your case) and tagged images can be isolated and then drag and dropped or passed in the ‘Open As’ function to PL5.

I have always looked upon XnViewMP as a utility rather than a heavyweight application but it has stood up to my throwing large numbers of photos at it as well or better than anything else! FSIV uses large amounts of I/O resources and time to import the thumbnails and is not really practical for directories of the size you need. FRV doesn’t attempt to “swallow” the whole directory in order to allow you to navigate and may or may not fit with your work style, free trial and then £22 for a licence. XnView Classic is as at least as bad as FSIV at importing the thumbnails.

XnViewMP is free but donations are apparently welcomed @£4.67 (inc VAT) per seat and I was offered 2 seats by default.

I hope that this helps.

EDIT:-
I tried the import in PL5 and it took around 10 minutes but these are photos with no editing attached so PL5 will be applying the default preset, Once real editing has been applied this import rate will simply decrease, i.e. the time taken will increase.

I have attached a snapshot of PL5 after an ‘Open with’ export from XnViewMP. I believed that Drag and drop from XnViewMP to PL5 worked in previous tests but when I repeated them I kept being taken to the directory and PL5 started the thumbnail refresh. The only difference is that in previous tests I had never opened the entire directory in PL5 before performing drag and drop!!

So if you ever want to drag and drop never open the directory directly in PL5, once it is in the database then drag and drop appears to result in the whole directory being opened every time!!

EDIT 2:
You might find this interesting (or not)

zkarj · December 22, 2021, 9:20pm

Genuine question: how do you find things in a directory of 10,000 files?

JoJu · December 23, 2021, 1:00am

With a good DAM, a good database it’s much easier than to open hundreds of folders, subfolders. And scrolling through thousands of previews handled in a database is also quicker than rendering RAWs.

Btw., the genuine question

already included that @md999 has no troubles finding his pictures. The trouble only starts once he likes to open a single picture in PL. And most of the contributions were about how different people work around the poor speed of PL’s way to check each file in a huge directory.

zkarj · December 23, 2021, 5:41am

Oops! I missed this bit…

But given a database is in play, it shouldn’t matter at all where the files are, so there isn’t anything to be lost in putting them in more than one folder. If such folder structure were automated (as many tools can do) then it should make no difference.

I once tried a thought exercise in which I tried to determine what sort of folder structure I might use for files of a given type and realised there are two extremes:

A single folder with files called, for example, 2021-receipts-household-utilities-electric-December.pdf
A nested structure 2021/receipts/household/utilities/electric/December.pdf

I personally think the right answer for me is somewhere in between, but the whole exercise is moot if you can simply search for what you want. Then, the whole concept of what the file is called ceases to matter for the most part (unless you need to see/use it in isolation outside of its ‘home storage’ area).

My photos are in YYYY/MM folders and then the file names are just what came from the camera. If I know roughly when a photograph was taken I may choose to browse to find it, otherwise I will search on keywords. If I had them all in one single folder then browsing would not be an option.

JoJu · December 23, 2021, 10:01am

I really love your post @zkarj ! If a capable search lets me find my files, I do not need dedicated structures, refined keywords with hierarchies and all the other sha-bang. Being a Mac user I got so used to Spotlight searches, that I only use a super simple YYYY-MM structure. Unfortunately, apps developed for Windows, too, can’t rely on an “onboard search”, so everybody has to re-invent the wheel.

But when I read in the manual “you can search in folders but not in projects” then I get my first doubts if the devs live on the same planet like me…

platypus · December 23, 2021, 11:55am

Mac users are spoilt by Spotlight, but it still cannot recognise people or places in the photos I took - and I’m quite happy with that. Keywords help to find content rather than files, therefore, I consider keywording to be essential for the sake of those who come after us.

Going through a box of photos that came to me because someone died, I was happy to find comments written to the the photo’s reverse by a person who obviously thought farther than her own nosetip.

But again, we’re venturing away from the OP’s question about opening individual files rather than folders…

JoJu · December 23, 2021, 12:13pm

So true. As far as I remember, Iridient developer opens a single file and doesn’t need to produce previews for other files in the folder.

One doesn’t need to have a folder of 10.000 images, PL is already slow with 500 - by “slow” I just mean slower than scrolling through 500 ready made previews. I think, DxO needs to decide wether remaining an excellent RAW developer and focus on that only (like Iridient or Affinity Photo also do) or do something to become a better image manager.

But seeing the differences within the app package, the manuals and the different OS versions, I lost hope this will happen in the next future.

Sorry for again drifting off. But on one side there’s a lot of satisfaction in the user group to be not locked into a database system, on the other side users also struggle with the consequences of always need to wait until PL has generated the previews? Or fiddle around with different standards of keywording inside PL and PM and whatever additional source of bugs is involved.

platypus · December 23, 2021, 12:17pm

A simple setting could help. Lightroom has it, but DxO seems to be too obstinate to make our lives easier. Software is meant to serve us, it’s not the other way 'round.

JoJu · December 23, 2021, 12:20pm

That’s the difficult part to get into the brains of devs… I understand they want to reach their goals: primarily to make the app work as flawless, fast and bug-free as possible. There just needs to be also few persons in the team to watch about usability and handling. Not to forget consistency and terminology.

platypus · December 23, 2021, 1:31pm

…which serves the user…

I think that DxO needs to review its products for common language, operating concepts, shared resources etc. - and then implement it step by step, all while adding quick wins proposed by the users.

md999 · December 23, 2021, 1:46pm

I have images in month/year folders then once labelled move them to topic folders. The trouble is that over time a 3-4 of the topic folders become very large 10,000+ images. I can sort them by various aspects such as date or location etc in some quite fast programs or use the full database program to view them very quickly by whatever or search for individual images by any aspect of the keywords, title, description etc. An example is a folder called trees where I have about 15,000 images and not really sure how to divide this into more than one, if I do it taxonomically then it will be a total pain to put files into and to make web pages from (without a more complex system than I have for web pages).

As I mentioned I have now set Photomechanic to open DXO and get individual files. However two issues, one when I select each additional file the first one is lost from DXO and two I can’t see how to close files in DXO without closing the program itself. I have searched through the help and there are no references to ‘close’ other than to close menus or the program itself. I have used a number of photo editing programs and they all have a way to close files and have ways to have multiple files open at once so it seems very strange that DXO makes it so difficult to find a way to do this - assuming there is actually a way to do it.
Perhaps I should add that I am not trying to say bad things about the program as it does some things much better than other software which is why I am trying to work out a way to use it properly.

BHAYT · December 23, 2021, 2:31pm

@md999 I am not an expert on PM but first are you using PM or PM plus?

I have tried passing more than one photo from PM to PL5 and, like a lot of other programs that I have tried only one photo from the group is passed!! I do not know how more than one can be passed from PM (i.e. what preference settings may be required/is there a special selection method that will result in more than one being passed etc.)?

If that worked for other editing software e.g. Lightroom then the issue is possibly with the PL5 Command Line Interface but as I found with XnViewMP and FRV it can be done by some programs, exactly what others I am not sure. I just opened 4 photos one after the other in PM and all four are available in the PhotoLibrary section of PL5 in the ‘External Selections’ . They can be navigated to and from and as many edits made as required as far as I know.

What I don’t know is whether you wanted all 4 over in one pass from PM to PL5 and what you are expecting back in the PM environment?

Perhaps @Stenis can elaborate but I only have access to a test copy of PM not PM Plus and I am no expert at to how to use the program.

With respect to “closing” a file, PL5 makes any changes (keywords etc. not edits) to the file for JPGs etc. as it goes along and updates the DOP as and when, for RAWs the DOP and the ‘xmp’ sidecar file will also be updated, you close a file by navigating away from the file/folder in PL5 or submitting another file from an external program to be edited, no explicit close is required?

One “problem” with PL5 is that it always starts with the folder/image/images last selected before the shutdown. That proved a bit of a nightmare when testing with the mega directory I created.

Please briefly (or as many lines as it takes) describe the workflow that you seek so that I can try to replicate it and see what I can see!!

EDIT:

Using multiple photos selected in PM (‘Edit selected photos with…’) does submit all the selected images to PL5 but they come in as {no of images} x 1 image entries in the ‘External selection’. Sadly this means that side by side comparisons between the images in PL5 is not possible.

Further EDIT:-
This just carried out on PL5.1.2 with PL5 shutdown submitting 9 photos results in only a single entry in the ‘External selection’ of PL5 PhotoLibrary. If PL5 is already running there will be 9 entries!

Opening XnViewMP using the same command in PM resulted in 9 separate XnViewMP windows being opened!

zkarj · December 23, 2021, 8:22pm

I don’t use Spotlight. For general files I have Alfred which is quicker, mechanically speaking, and lets me do more. But I was referring to DAM searching. If the photos are in a database, that database should offer searching.

On the assumption that PL cannot do what is required in this case, I was looking for alternative ways to attack the problem.

Agreed there is no obvious way to topically divide this without driving yourself crazy. This is why I use date-based folders. As I mentioned, if I want to browse by date I can, but a lot of the time I just use search in my DAM of choice (Lightroom Classic). Often I will do both… search for a photo and then when I find it, I will go to the relevant date folder to see others taken at the same time.

BHAYT · December 23, 2021, 11:34pm

@md999 I have so far documented a variety of potential solutions in the forum but with no response from other forum members or you, which is a little disappointing?!

I believe that the issue with large directories is only relevant insofar as when it is not possible to pass a file (preferably more than one) to PL5 it then becomes necessary to open the entire directory in PL5 with all the problems that can cause , i.e. the time taken to open all the thumbnails in the directory which is further exacerbated because PL5 renders and re-renders the thumbnails with PL5 edits (and does this every time that the directory is opened and re-opened and traversed and …).

With respect to improving directory opening I believe that this could be improved if PL5 abandoned its current thumbnail caching scheme and replaced it with a full time SQLite database for the thumbnails, i.e. “persist” the thumbnails.

This would be complicated with PL5 because as photos are edited their thumbnail would need to be updated in the SQLite thumbnail database and worse, if a “rogue” DOP or DOPs were added to a directory the thumbnails would need to re-rendered in the light of these changes and even worse new DOPs being added might cause the introduction of Virtual Copy thumbnails which would need to be added to the thumbnail database and any other complications that you can think of. But it would improve general speed in PL5 and reduce the amount of spinning icons (if all the ramifications can be satisfactorily resolved)!

However, the database would be large and the whole thing becomes extremely complicated when thumbnail sizes as changed and what happens when the main PL5 database is destroyed (in my case mostly deliberately)? and…

I have a DigiKam system only up to date as of early 2021 which is now updating itself automatically so I will get a better idea of the issues of thumbnails in a SQLite database from that.

Failing such a radical new feature the only other alternative is to never browse the big directories in PL5 but do that browsing elsewhere and submit requests for editing directly to PL5. Now the problems really start because while some programs play nicely others do not or (vice versa) PL5 plays well with some programs but not with others, whichever way you want to present the problem, i.e. try to get your other supplier to change or get DxO to change or find a product that does all that you want with respect to management and searching but including (I would suggest) passing one or many photos from one or many directories to PL5 in a single submission!?

Some programs when passing a group of files will be greeted by PL5 starting to scan the entire directory, this also seems to happen with Photo Mechanic when drag and drop is used (at least in my tests).

Some programs when passing a group will actually have only one opened by PL5, better than nothing but …

Others packages will cause all the photos to be opened but effectively in multiple windows, in PL5 this manifests itself as one entry for every photo in the group with each photo represented as a single item in the ‘External selections’, they are all there - well sort of! With PM and PL5 if PL5 is shut down only one photo will make it to PL5 but with PL5 already running all the photos will be available but as separate selections making it impossible to compare and contrast etc. in PL5.

XnViewMP and Photo Supreme seem to “play” nicely and pass the list of files in a manner that results in PL5 opening all the selected files as members of the ‘same external selection’ where they can easily be reviewed and edited etc…

The above must come with a disclaimer that not all variations of the scenarios have been full explored by me, and that the results were actually what I thought I saw etc. etc.

**However, what would be really nice is if those forum users with DAM systems and other photo managers could use those products to take a selection of photos and pass them to PL5 in a single operation with PL5 already running and also with PL5 shutdown. We can then compile a list of those that “play” in any way, those that “play” nicely and those that do not “play” at all (e.g, nothing happens or PL5 starts reading the entire directory).

In addition it would be useful to know which programs work well with other editing products and how well or otherwise they also work with PL5. If there are many that do well with other editors but run into problems with PL5 then there is a real case to press for PL5 to be improved**

The tests do not need to be done on particularly large directories of files just as long as it is easy to determine whether the files passed have been opened versus the entire directory. The tests are pertinent to both Mac and Windows systems albeit it might work out that no MAC system has a problem or …

The next part of the equation is how useful are the programs available for executing metadata searches in order to come up with clusters of photos to be passed to PL5 for editing etc. BUT if there is no way to get the photos across to PL5 in the way that the user wants then are they really that useful or should another program be used or should we be campaigning for changes to PL5.

jch2103 · December 24, 2021, 12:05am

I can confirm that it’s easy to drag and drop a selection of images into PL5, thereby creating a new External Selection. I did this with IMatch, similar to what you’ve done with alternate software.

In IMatch it’s done by using the Favorites feature, which uses stored Windows shortcuts; once that’s done, any selection of images (including ones from different folders) can be dropped onto the Favorite, triggering creation of an External Selection in PL5 (or PL4 or earlier versions).

For some reason, DxO hasn’t documented how External Selections work (or even confirmed in its Help that this feature exists). PL5 is clearly using a standard Windows function (IMatch wasn’t customized for PL5, but rather uses standard Windows conventions, as, I assume, the software you used). This sidesteps problems with too many images, refreshing folder contents, etc. It makes it easy to create PL5 External Selections containing relatively limited numbers of images from multiple folders containing hundreds of images, using arbitrarily complex selection criteria. (IMatch easily handles databases with hundreds of thousands images, although my own database is a bit smaller than 100,000 images.)

I can also drag and drop a selection from a Windows Explorer folder onto the PL5 desktop icon. However, this simply opens the entire folder in PL5 with the selected images highlighted and it doesn’t create an External Selection that only contains the desired images.

If DxO would document the External Selections feature, it would make it much easier to accomplish the wishes of the OP, which are definitely not ‘impossible’.