What happen in PL when you open one folder? I check and describe!
Its a bit long, so i suggest to read/jump to the SUMMARY section(s) - highlighted with bold. Important note: when i talk about ‘all photo’ i talk about all photo in the ‘filmstrip’ visible portion
May good to know if open a folder with 3000 photo. I do some research / observations on File read/write (File I/O) - based on traces - and describe in main points. Its may a bit long, but now i guess we have some details on the process - all based on my observations, some point may not sure.
Base situation for describe my test(s):
- Folder contains 5 photo
- All with the same camera body + lens. All RAW, Olympus 20Mpix (ORF)
- All applied the same editing (preset), but no mask at all - so .dop files exist. Geometry correction enabled. Note: not Auto-Apply in preferences, i apply manually.
- All photos has camera body+lens combination what DxO know, and related DxO module already downloaded and applied
- PL Cache cleared. So no thumbnail, no preview generated at all.
- Important note: when i talk about ‘all photo’ i talk about all photo what visible (in the GUI) in the ‘filmstrip’.
After folder opened in PL PhotoLibrary mode:
- 5 (five)thumbnails generated (because all 5 photos in the filmstrip)
- 1 (one) preview generated (the first (selected) photo)
See some details about what happens in PL during folder open (main points only, from 7000+ debug line):
- Check (query) all photo (.ORF) creation creation date, lastwrite, changed, etc datetime
- Check xmp files exist for all photos
- Check dop files exist for all photos, check creation date, lastwrite, changed etc datetime
- Read PL database (generally i not describe database later, i concentrate more to file read/write)
- Read PL config and User config
- Check thumbnails cache db (i think it is)
- Read the photos (.ORF) ‘header’ part - very small read in the start of the file, all read may less than <100kb. I guess its about read (extract) the photo metadata, like Camera Body, Lens, exposure properties, etc.
- Check thumbnail cache db
- Read photos (.ORF) again in small reads may less than <100kb. (In batch of 2 photos?)
- Check Preview cache db
- Read photos (.ORF) again in small reads may less than <100kb. Guess metadata extract.
- Load .icm profile(s)
- Check camera body config file (DxO file)
- Check photo files (.ORF) various times for something (metadata)
- Check photos (.ORF) for creation creation date, lastwrite, changed, etc. Actually its does multiple times in the whole process for all photos (.ORF)
- Read all photos (.ORF) file (or at least batch in 3 photos) first small headers, and later the RAW content (seems only the RAW image data).
- Some of the reads happen same time in multiple files (so, process seems paralel sometimes)
- Check / read lens calibration file
- Read some .caf module (i guess based on Lens calibration)
- Read DxoCorrectionEngine (i guess because of geometry correction and editing)
- Read CameraBody RGB custom
- Read the first photo (.ORF) is small reads (may metadata)
- Read the first photo (.ORF) embedded JPG (i guess, read size looks very-very similar than the extracted jpg file size)
- Start some threading
- Read PL config and User config
- Read a lot of DxO registry value
- Read Thumbnails db file
- Start to create Thumbnail files
- Start to read other photos (only raw image part, in 2GB chunks, other places its read in one session (like 15GB)).
- Continue (or Re-start?) create Thumbnail files
- Various times .icm reads
- Read LensCalibration file (DxO file)
- Write preview file (for the first (selected) photo). Note: preview NOT generated for other photos)
- Re-Write first (selected) photo Thumbnail file (Note: its already generated)
- Re-Write last photo Thumbnail file (why the last?, may i miss)
- Re-Write first (selected) photo Thumbnail file
- And that’s it.
I not write various icm reads, file query’s and some point it was unclear how many photos read. Pardon, its already takes many hour.
SUMMARY:
If no no tumbnails / preview generated but .dop exist and open the folder (this test)
- Its query up all .ORF files (what’s in the ‘filmstrip’, and not all in the folder!)
- Its read thru all RAW (.ORF) file for various metadata
- Its check and read DxO files based on that metadata read like the lens correction module, camera body color, .icm, etc.
- Its read up all RAW (.ORF) file for the RAW image data
- Its generate thumbnail (cache) files for all photos (with corrections)
- Its read for the first (selected) photo embedded jpg image
- Its generate preview (cache) file for the first (selected) photo (with corrections)
- Does database read/update, also on cache db’s
Ok, see what’s happen if cache thumbnails and preview already generated (summary):
- Check (query) all photo (.ORF) creation creation date, lastwrite, changed, etc datetime (what’s in the ‘filmstrip’, and not all in the folder!)
- Read the first (selected) .ORF photo header read (for metadata i guess)
- Read the first (selected) .ORF photo embedded jpg read
- Read the first (selected) .ORF photo RAW image part (at least 2 times?)
- No read on other RAW files at all.
- 6. Thumbnails read up (for all photos)
7. Looks the first (selected) photo thumbnail read up twice (? may after one re-write)
8. First (selected) photo thumbnail re-write like 3 times
9. First (selected) photo Preview file NOT write again
Some conclusion / idea / discussion:
May its seems to open one folder takes lot of query’s and lot of reads (file reads). I think its just looks enough. May some point looks less logical, like re-generate the selected photo thumbnails 3 times, but i guess its may like once without any correction, once with geometry correction, and once with all correction. Note: may you not apply for PhotoLibrary view first time the geometry correction, etc.
I think PL looks pretty fast on all of that (in my opinion). And seems logical as its goes thru.
One thing what i not get it, where the preview (cache) file (image) used. Its a relatively low resolution file 1313x972 pixel (vary sometimes, like 1336x972, etc.). May its just to fill the void in some cases. Note: PL able to read AND display the embedded jpg (even for a very small time its may display the embedded jpg for preview for first time you select one photo in PhotoLibrary)
However back to the original start of the thread, i guess for some NAS may not handle requests in this amount for 3000 photos in the folder. NAS also has soem filesystem, some protocol for network sharing (like SMB), all goes thru network, etc, and NAS may get some ‘hit’. Even NAS seems fast (like file copy/transfer, what mostly sequential read/write and its fast anyhow), may not (cant be) at the level of responses like some local SSD. Its not local drive. I think, if someone experience issues with NAS with like 3000 photos, may need some careful check also on NAS side.
Also, the comparison with like Windows Explorer file browsing vs DxO is cannot match, as its like apple vs orange question.
May you wonder about this reads durations. Its not easy to say what the exact, as Windows does file caching quite well - if you like open once a folder, read/query some file, it’s cached for a while.
So, i reboot the PC, to made more clean state. My PC has some older (8+years) SATA SDD.
For 1 (one) .ORF, approx 18MB, PL reads:
- To read RAW image data (approx 17MB): 0.19 sec. Once its reads up (windows11 cached it), the next read-up takes may vary like 0.004 - 0.035 sec
- The small reads (for metadata extraction or similar) takes very-very small times: 212 small read takes 0.009 sec. Note: the 212 small read may just a few read session, but breaks up internally/during the trace.
Later i also does from clean state open folder with 300 photo (.ORF) - folder never opened in PL, no editing just the ORF itself. Open and read up all the 300 files takes 11.5 sec. Actually i think the time is linear. So, 3000 photos in local may 1min 55sec in clean state. After that, clicking between folders re-open the folder takes 0-2 second. After PL restart folder open for this takes like 4-5 sec.
So, actually i think file I/O (read/write/query) operations is pretty fast - even with old sata SSD (and not new fancy nvme drive).