Organisation of raw, working and completed files

I have used Photoshop for many years before moving to DXO 9. I used 3 folders, 1 for my raw files that were untouched, 1 for the files I I copied into for editing before moving the completed files. This is not a system that works for DXO 9. What do others do for their organisation of files?

Welcome to the DxO USER forum, Tigger …

That’s exactly how I use PL …

  • I process RAWs in batches (within my W-I-P folder) of about 20-25 source-files at a time - then move the results into a structured folder system (by date, etc)

  • This approach has added advantage that PL is very fast/responsive - not being “bogged down” from wading thru 100s (1,000s !!) of source-files.

This doesn’t work well with PL only in the sense that the database can’t keep track of images if they’ve been moved without it’s awareness.

There are two ways to manage that;

  1. Regularly delete/rebuild the database over your storage structure

  2. Don’t bother with the database at all (which is OK if you don’t value ability for keyword searching, or use of Projects - and a few other incidentals) ..

  • This is my approach - I run PL via a “wrapper”, which firstly deletes the database (and cache) before invoking PL itself.

Either way, be sure to have PL configured to create & read Sidecar/.dop files

2 Likes

I organise my RAW files into yyyy/mm folders purely as a means of segregating them somehow. It is often useful if I know, for instance, that I was in Singapore in April 2019.

That’s it. They get backed up and archived from there to various places as a form of protection but, operationally, they stay right there.

The concept of work in progress I usually do by ‘picking’ the images I want to work on and subsetting the view. The subset will hold even if you switch folders.

There are no “completed files”. The RAW file is always the RAW file. The edits done with PhotoLab are just entries in a database/DOP file. I export a downsized JPEG which I put on Flickr and in my iCloud Photos, then I delete the JPEG.

So when I’m done editing, as far as PhotoLab is concerned, there’s just the RAW file and the record of edits. Sometimes there are virtual copies, but that’s still no different from a filesystem perspective. It’s just more data.

I organise my photos in folders starting with Year then subfolders for my different cameras and then by folder yyyy-mm-dd.

For special occasions/holidays I use the top folder of “yyyy - Occasion” and then same folder structure of camera and dates.

I also have a folder for exports called Developed for my JPG exports under the year folder plus some other folders as required.

I rely solely on sidecar/.DOP files and occasionally delete the PL database.

For culling I use FastRawViewer and star ratings which are read by PL and I then I filter by star rating on each folder for editing.

Backups are by year folder to offline storage (large capacity spinning disk drives) which then go into a fireproof safe. High value photos get backed up to multiple disk’s.

Finally, I have a Template folder with my folder structure with empty folders. I copy this folder to a new year / occasion folder ready for use.

I have refined this structure over many years from my initial use of Lightroom 1 - 6 and then PL 1 - 9.

2 Likes

@Tigger
I find PL works well with the 3 folder groups you describe very similar to @John-M.

I bring the new RAW images directly into the “work-in-progress” folder first, with appropriate subfolders matching my session organizational needs. These subfolders are temporary for the PL work. PL editing is done in these temporary subfolders with “finished” images exported to the desired permanent folders and intermediate exports to another WIP subfolder based on pixel editor of choice.

Once editing is complete (processed flag) the RAW images with .xmp and .dop are moved from within PL to a permanent storage folder structure. Once empty, the subfolder is deleted. Same steps then with the “intermediates” subfolder when working in the other editors. If images need to be re-worked they are copied back to the WIP. Use the PL “Projects” feature to edit across subfolders but this is temporary. Use virtual copies for for test edits or alternate edits of an image. Moving the RAW image from WIP subfolders to permeant storage and not relying on Projects allows me to ignore the PL database issues. Very minimalist keywording is done in PL. My image database is only looks at “permanent” storage.

Designating separate “working folders” from “permanent folders” helps me keep my photo editing efforts “organized”. Keeping a clean WIP folder makes me feel happy. :grinning_face:

1 Like

PhotoLab never changes RAW files with the exception that it writes metadata to DNG files, e.g. from supported cameras that write DNG files instead of using custom formats like others do.

Not changing RAW files means that edits need to be logged in PL’s database and exported to DxO custom sidecars (the “.dop” files containing the “cook book” that turns the potatoes to fries, so to speak)

Separating the settings files from the source files will inevitably break the relation and the sources re-appear in their original looks. Only upon export will the edits be burned into the exported file, and this file can be redirected into a target folder during export.

If at all, only two folders are needed.

  • Folder 1 contains the source files AND the sidecars.
  • Folder 2 … can be a website, e-mail etc., no need for a real folder, unless you use it to load its content to several receivers. Only then will a separate folder make sense from an effort and processing time point of view.

Because the “completed” files can always be re-created (or re-made with modifications), there is no real need for a separate/explicit/intermediary 2nd folder. Exporting to e.g. Apple’s “Fotos” makes “Fotos” the container for what you want: an exhibition venue.

I create a new folder for each event (mostly sports) and some of these may spread over several days. All RAW, DOP and XMP files are kept in this folder.

I create an empty folder that will hold the exported files.

I put this folder_name into the job_identifier in metadata.

I set the subject in the metadata.

I export my files using this rename function /job_identifier/subject_filename

So I don’t need to create/edit an existing Export recipe for each event.

All files are stored in a RAID1 setup

So far I haven’t given my photo folder organization much attention … however reading the above I may have to give it more care:

To add new photos to my archive I connect my camera, add a new folder <yymmdd - event name> with the “Canon EOS Utility”, give the files a name scheme and transfer them to the new folder on HDD. Depending on the event it may be up to 1.200 photos per “shooting day”.

Next I use PL to work on my photos and can see PL adds .dop files to the respective folder. Once done with the photos I export the relevant ones to new sub-folders to the mentioned (“jpg1” for average quality, “jpg2” for reduced Resolution for WebSite upload)
<yymmdd - event name>
—> jpg1
—> jpg2

Finally I distribute the resulting jpg-photos resp. upload them to a photo website. My PC has an SSD 1TB for OS/software but 2 HDDs (6TB) for about 200.00 photos. One HDD for “use”, second HDD for daily incremental local backup with “Syncback” (plus a Linux-Server for remote backup).

Could it make sense reorganizing the folders?
Also about the PL-database (in fact I was even surprised reading in this forum it exists - before that I was under the assumption it’s only the sidecar .dop files): Should the PL-database also be backuped?

I copy the photos RAW + JPEG form my camera in the folder Photos//, where folder is a folder by theme (travel, nature, family) ou by event whose name is

The photos are named in camera with a trigram for the camera (A7R, AIV,…) ; when there are more than 10000 phots shot by the camera, I increment the 1st digit in the picture name.

I process the photos in project mode, insert some keywords, at least generic ones like name of the location.

After processing, I remove the camera jpeg if the picture has been processed, RAW + jpeg if the picture is bad.

I have made my own DAM where I record all pictures, rename (DSC12345 Description.jpg). Renaming is the same for all files associated to the raw.

Then I make slideshows that I export on my site (see escaich.com).

Organising things is not done for the joy of organising, but the necessity to find things again, later, by yourself or someone else.

Organising Photos by technical aspects that can be found in metadata (camera model, lens used, file type etc.) is, imo, the least helpful. Instead, organising things by occasion/content/customer etc. adds value that is otherwise lost over time.

In order to reduce the risk of having to re-do things, proceed like so:

  1. write down what your most suitable structure might be
    e.g. Date → Occasion → … suitable for private use, choose sensible depth (3-4)
    e.g. Customer → Job/Date → … suitable for business use, choose sensible depth
  2. make a backup
  3. set up the structure and add files (and existing sidecars)
    doing this, you might add levels as needed, e.g. to limit the number of images, which helps to work with PhotoLab (PL doesn’t scale gracefully)
  4. make a backup
  5. have PL index the structure from it’s root up

Backups?

  • regular general backups (use Time Machine or other automatic tool)
  • regular PL database backups (use PL’s built-in feature, it’s manual though)
    Hint: add a prefix of “PLn “ to link the backup to version “n” of PL

My emphasis. I’m not poking holes in your argument, which I broadly agree with, but it jumped out at me as an interesting point.

I use YYYY/MM purely as some means of preventing a single, monolithic folder. I think the key thing that date has is that every photographer will have this dimension to their shooting.

Given I don’t have customers or jobs or even, for the most part, events, I cannot fathom anything else I could break things up by that has any sense. Until recently I only ever had one camera (I now have two). I tend to shoot with the same camera and small number of lenses a lot. All these things are recorded in the files anyway.

So… if you do have a customer or a job then that’s a logical thing to organise by.

Although… I can set a 4 character prefix on my cameras and could, in theory, set a customer or job number right in the file name!

Thanks so much for sharing your org structure (even with a screenshot!) and insights of your personal workflow Bernard @be51 - much appreciated! It helps a lot getting the own stuff organized when reading how more experienced enthusiasts do this.
Also thanks for adding the link to your photo-website … nice site too, by the way!

So you structure your folders by “decade/year”, then “genre“ and finally “yyyy-mm-dd Event-name”
For your files you

  • keep the photo file names as predefined by the cam (unless > 10.000 files)
  • load raw + jpg files from the camera (and keep both in the same folder) where you delete the cam-jpgs (probably overwrite?) when processed by PL
    => I configered my cam that it only takes raw-files, hence I do not load/store cam jpgs
    => also I generate the jpgs into a separate subfolder as otherwise you see the photos double in PL
    So you do not limit the numbers of files per folder …

May I ask:

  • what is the “project mode” you mention?
  • what is DAM?

Again: Thanks so much for your insights!

Thanks a lot for your advice @platypus !

Reason for asking was that I got the impression out of the discussion there may be technical considerations for structuring folders/files in a specific way - e.g. limiting the number of files per folder to “x” as indeed PL needs some time to look thru a folder when the file number is - while you as the user can only wait until done.

My current structure is very flat: “pictures” → “yymmdd event-name” (→ jpg-subfolders) which was “good enough” up to now … in the sense that I found all photos back and had no troubles in backing up (I do a daily incremental backup with syncback locally and on a file server - for OS/software I do random SSD images).

So you recommend a regular backup to the PL-DB, too …
Good knowing that and I will add this to my backup job!

May I ask:
In your list: How would I “have PL index the structure from its root up”?
Where can the PL–DB be found? And if the worst should come to the worst: What happens if my PL should crash and I have to do a new install - how would PL re-generate its DB?

Thanks!!!

Firstly, I import my photos into a “Photos In” folder then rename them (with FastRawViewer) to YYYYMMDD_hhmmss plus a numerical suffix (_01,_02,etc) if more than one photo has been taken in a second.

After culling (using XNViewMP as a viewer):

RAWS into Photolab outputting TIF files for further edits in Photoshop. e.g. YYYYMMDD_hhmmss_PL.tif

TIFs into Photoshop, perhaps some further editing then output full size jpg e.g. YYYYMMDD_hhmmss_PL.jpg Resize & sharpen (for screen viewing) then output smaller jpg e.g YYYYMMDD_hhmmss_scr.

Then move everything that’s been edited (with XNViewMP). Raw, sidecar & full size jpg into a yearly folder e.g. “2025” and resized jpg into a subfolder e.g. “2025 Scr”. Delete the TIFs.

Sometimes I might have more than one folder per year. e.g. a 2025 holiday in France might have folder “2025 France” and of course “2025 France Scr” for viewing/sharing.

Has worked for me for nearly 25 years. I avoid software that relies on databases (e.g. Lightroom) and have no time for adding keywords to tens of thousands of images. Everyone has different needs and priorities though.

Adding hhmmss to the filename makes the filename to complicated for me.
My shots are renamed to GW_YYYYMMDD_nr and stored in a directory YYYYMMDD and an eventual reminder of what the content is. Holiday can be stored in a directory YYYYMMDD with daily subdirectories.
Exports are written in subdirectories JPG,TIFF or DNG. When used.
Until now I’m satisfied with it.
Practical I’ve a main directory for the D80,D300 and D700. They where edited with Capture Nx2
Later on one for the D750 which where edited with PL and another for the Z6.

George

Thanks @David_McA David!

So you use an entirely different workflow:

  • Import photos into an intermediate place and rename following a date/time based scheme
  • select photos using XNViewMP and modify with PL
  • move all to a permanent folder (with subfolder for jpg)

I was also surprised that PL uses a database which I assumed only LR/PS does … I also don’t use keywords avoiding the extra work …

Interesting reading you use XNViewMP - probably it’s quicker than PL?
As of today I use PL for the purpose of selecting with the “star ranking”, only postprocessing/modifying the files given at least one star …

Left sidebar of PL in “Library” view: context-click on your photo archive root folder …
You might need to look for that folder, e.g. if it’s not where PL thinks it should be.

You can find the default location by searching the posts. PL (Windows) can be set to locate the database wherever you like it. On Mac, the location is fixed.

When PL starts for the very first time, it creates the database. And every time PL encounters an image that it supports, metadata of the image is added to the database.

To restore the database, there are two main ways:

  1. restore the DB backup (latest or earlier according to what you want/need)
  2. index the photo archive: Delete the DB → Start PL → Index as described above

Way 1 restores everything, except the work you’ve done between the time of the backup, way 2 restores things from the .dop and .xmp sidecars. Absent sidecars lead to loss of your metadata and edits. Depending on platform (Mac/Win) projects and/or history are lost too.

For details, consult PL’s user guide. Find it on support.dxo.com

Great description!
Thank you a lot @platypus !

2 Likes

My structure is

  • Photos
  • decade
  • year

then

  • theme / day like in the case of art/expositions in my capture
  • or
  • theme like in nature or Paris
  • or
  • mixed

I am pragmatic ; if I have a travel, it is a theme subdivided by day or subject

You will find this breakdown structure on my site escaich.com/tassili

When I rename, all files with same identifier are renamed identically.

No, DxO puts the suffix to pictures processed.

Camera jpegs are useful for a rapid exploration.

I delete those after processing.

For me a picture :

  • is an autonomous object : I can move it from a folder to another one ; of course all associated files are moved together
  • must be found in finder easily ; it is easier to search AIV29255 than a complete time stamp
  • its content must be identified easily (Notre Dame is better than a time stamp)
  • in myself DAM, pictures are identified by the short identifier, so that I can change the description without loosing them

DAM = digital asset management

Difficult for me to give the exact name in English, so a capture will be better

More than 10000 pictures ; my son displays my site on its watch, not very useful but funny !

The interest of “work with project” versus “work on folder” is that you open raws, work , remove of the project without throwing in the trash, closing and reopening projects.

I don’t process all pictures, I explore the production of the day, select some, put some keywords, stars ; it is not a 100% process at one time.

In the beginning of DxO, when working on folders, I removed some pictures of the list and DxO put those in the trash and erased the pictures ; now, DxO put the suppressed pictures to the trash and you can recover.