Good advice! 
Better to ask one of the people here who pretends to know everythingā¦
Better yet, to ask one of the very few people here who likely do know just about everything.
@Joanna Thanks Joanna, since darktable also writes a history (but in xmp) my thought process was that we might be able to use the history in other applications as well. It certainly doesnāt work for DxO, I have tested that. Tools are working in completely separate worlds. It seems, interaction is not wanted by developers. (Except the small range for META data)
@HGF would be nice to have. But for sure it will not work with respect to the editing history. Ratings, keywords and other metadata could work. But how should it work for editing? Each program uses its own modules/algorithms for various purposes. How would you, e.g., map DxO ClearView settings to a corresponding module of darktable (like diffuse&sharpen) or Lightroom which might have completely different settings?
Unfortunately, as you mention, this just doesnāt happen
And this is the key.
The XMP file format is just a general, non-database, small scale, data storage file format.
It was introduced by Adobe in 2003, primarily as a means of storing metadata rather than editing history.
However, the data format and the mechanism for encoding and decoding these data have become a standard for all sorts of āstuffā that one might want to associate with a file.
Not only is it the the format of a sidecar file, its syntax has also become one of the āflavoursā of metadata that can be embedded in image files as well.
If I use ExifTool to investigate one of my RAW files, I find an XMP ādomainā in the metadataā¦
[XMP] XMP Toolkit : Image::ExifTool 12.39
[XMP] Subject : Moon, Reflections, Evening, Lune, Reflets, Soir
[XMP] Create Date : 2023:07:17 22:27:42.79
[XMP] Creator Tool : NIKON D810 Ver.1.11
But I will also find other domains. Some generalā¦
[File] File Name : JNA_0046.NEF
[File] Directory : .
[File] File Size : 71 MiB
[File] File Modification Date/Time : 2024:06:03 10:14:22+02:00
[File] File Access Date/Time : 2024:06:22 21:16:47+02:00
[File] File Inode Change Date/Time : 2024:06:03 10:14:22+02:00
[File] File Permissions : -rw-------
[File] File Type : NEF
[File] File Type Extension : nef
[File] MIME Type : image/x-nikon-nef
[File] Exif Byte Order : Little-endian (Intel, II)
ā¦ the extensive EXIF domainā¦
[EXIF] Make : NIKON CORPORATION
[EXIF] Camera Model Name : NIKON D810
[EXIF] Strip Offsets : 74423436
[EXIF] Orientation : Rotate 270 CW
[EXIF] Samples Per Pixel : 3
[EXIF] Rows Per Strip : 120
[EXIF] Strip Byte Counts : 57600
[EXIF] Copyright : Grandes Images 2023
ā¦
[EXIF] Jpg From Raw : (Binary data 1070175 bytes, use -b option to extract)
[EXIF] Other Image : (Binary data 496340 bytes, use -b option to extract)
[EXIF] Thumbnail TIFF : (Binary data 57816 bytes, use -b option to extract)
ā¦ plus some 82 more values
The last three lines here are actually where the JPEG images and thumbnails are stored and explain why you donāt need to shoot in RAW+JPEG, because the JPEG is already contained in the RAW.
Then there is the MakerNotes domain, which can be fairly massive and is specific to the camera manufacturerā¦
[MakerNotes] Maker Note Version : 2.11
[MakerNotes] Quality : RAW
[MakerNotes] White Balance : 5600K
[MakerNotes] Focus Mode : Manual
[MakerNotes] Flash Setting :
[MakerNotes] Flash Type :
[MakerNotes] White Balance Fine Tune : 0 0
[MakerNotes] WB RB Levels : 2.02734375 1.26171875 1 1
[MakerNotes] Program Shift : 0
[MakerNotes] Exposure Difference : 0
[MakerNotes] Compression : JPEG (old-style)
ā¦
[MakerNotes] ISO Step Size : 1/3 EV
[MakerNotes] Exposure Comp Step Size : 1/3 EV
[MakerNotes] Center Weighted Area Size : Average
[MakerNotes] Fine Tune Opt Matrix Metering : 0
[MakerNotes] Fine Tune Opt Center Weighted : 0
[MakerNotes] Fine Tune Opt Spot Metering : 0
[MakerNotes] Multi Selector Shoot Mode : Select Center Focus Point (Reset)
[MakerNotes] Multi Selector Playback Mode : Thumbnail On/Off
[MakerNotes] Multi Selector : Do Nothing
[MakerNotes] Exposure Delay Mode : Off
[MakerNotes] CL Mode Shooting Speed : 3 fps
ā¦
[MakerNotes] Roll Angle : 90.1
[MakerNotes] Pitch Angle : -40.3
[MakerNotes] Yaw Angle : -0.1
[MakerNotes] Noise Reduction : Off
[MakerNotes] Color Balance Version : 0223
[MakerNotes] Lens Data Version : 0204
[MakerNotes] Exit Pupil Position : 68.3 mm
[MakerNotes] AF Aperture : 3.8
[MakerNotes] Focus Position : 0x81
[MakerNotes] Focus Distance : 4.47 m
[MakerNotes] Focal Length : 35.6 mm
ā¦ plus another couple of hundred other values generated from the camera.
And thereās more, depending on the camera model.
So, before you download a RAW image from your camera, it already contains a whole bunch of metadata, in XMP format, including an XMP section.
My DAM tool allows me to write searchable attributes like keywords, ratings, etc directly to the XMP section in the RAW file but PhotoLab opts for using an XMP format sidecar file instead of modifying the RAW file itself.
So, we have XMP format metadata in the RAW file and we can write all sorts of stuff into that file but, in order to avoid tying the RAW file to one editing tool, most editors separate out editing āhistoryā into a sidecar.
Adobe uses a sidecar with the XMP extension, as does DarkTable,
DarkTable stores some of their editing metadata in a custom darktable domain, which maps internally to XMP tags.
But, straightaway, you find that the Darktable XMP sidecars are namedā¦
Filename.FileExtension.XMP
ā¦ which is different from the industry standard XMP file that record file information metadata like keyword, IPTC data, etc., which are namedā¦
Filename.XMP
PhotoLab uses a similar arrangement so you get three filesā¦
Filename.FileExtension
Filename.XMP
Filename.FileExtension.DOP
And, with DarkTable, it can seem more confusing, because you getā¦
Filename.FileExtension
Filename.XMP
Filename.FileExtension.XMP
When you export a file edited in PhotoLab, the generated JPEG or TIFF file will contain XMP (non-image-editing) metadata, embedded in it.
Confused? When it comes to metadata, it is all too easy to become so 
Now, I need to lie down and let the throbbing in my brain dissipate 
1 Like
Thanks Joanna, very detailed explanation.
@Joanna, for someone who wants to fully understand what is going on behind the scenes, what you wrote sounds very complex.
Just one question - for someone who just wants to edit the best possible images using PhotoLab, how essential/important is XMP?
For me, I put my memory card in whatever camera I am using, capture images, use PhotoMechanic to load them into my computer, and do my editing mostly with PhotoLab.
(But for your understanding of all these data files, I might never have solved my problems from a month ago, but by following your advice, that issue is now under control.)
Most of the time I will shoot in RAW, likely creating Nikon RAW files (but I still own my Leica and Fuji and Canon cameras, which create their own RAW files.
I still have access to Lightroom, but I expect to leave it alone on my older Mac mini computer, which Iām no longer using very much, and to not use Lightroom on my MacBook Pro, my main computer as of now.
Forgetting DAM, and only considering image processing, is there any valid reason why I need to pay attention to my .XMP files? (for images which will be processed in PhotoLab)?
(I donāt want to discourage anyone else from discussing XMP, but Iād like to stop thinking about it, and just ignore those files if possible.
Unfortunately, it is, but only for those who need to make use of it.
If you are not at all interested in metadata, it is not at all important. But you have decided to ingest all your images using PhotoMechanic, whose sole aim in life is to create and manage metadata. Therefore, you make life more complicated than just editing images.
On the other hand, if you want to find an image or images by keywords or other attributes, then you need to write those metadata, either to the RAW file, or to an XMP sidecar file - something that Photo Mechanic can do for you, although there are simpler apps out there.
You no longer need Lightroom unless you are planning on re-editing files already edited in it.
As I have said previously, XMP files are optional sidecar files, usually only needed if you want to manage metadata for indexing and searching for files by their metadata, instead of visually.
To repeat, as long as you donāt want to index or search your files, or ingest them using Photo Mechanic, you donāt need XMP sidecar files.
I donāt know for sure if Photo Mechanic can write directly to RAW files but, if you switch to that option, then it should no longer generate XMP sidecar files.
If you do anything with metadata in PhotoLab, then that will produce XMP sidecars as it doesnāt write directly to RAW files.
The size of the embedded JPEG can vary and is usually quite small compared to the full size image.
I shoot RAW+JPEG because the JPEGs are full size and allow me to quickly examine and compare composition and focus in an image browser as a first pass and cull of RAW/JPEG pairs. I then further examine the remaining RAWs in Fast Raw Viewer. For panoramas, Iāll occassionally throw the JPEG set at my stitching program just to see if there are any issues or indeed, if it is worthwhile developing the RAWs.
For those who like to examine their images on the camera screen, the larger JPEGs allow zooming in on the detail, which is not possible with the RAW embedded JPEGs.
Just a thought, shared for anyone contemplating switching to RAW only.
Otherwise, though our workflows may differ, I have learned a lot from your posts here.
Thank you.
Not sure if this is still true, but there used to be two versions of PhotoMechanic, one that just did what PhotoMechanic has always done, and a more expensive version that included DAM technology. (Being foolish, as I usually am, I bought the better version, just as I always buy the most powerful version of PhotoLab.)
For me, until lately, itās sole purpose in life was to ingest my images into my computer in whatever form I feel is most useful to me, as it renames images, and renames folders, in a useful manner. Thatās what bugs @Wolfgang, as it uses the ābarā character as a separator: | You will see this character as youāre on a Mac. Windows doesnāt allow this character for reasons I never really understood.
Currently, I have no need for Lightroom, but if I want access to PhotoShop, both come in Adobeās āphotography planā. So yes, I have it, and itās presumably still working on my Mac mini, but not on my MacBook Pro.
I have no plans to stop using PhotoMechanic. It and PhotoLab co-exist, and this is how I will be moving my images from memory card to computer. I just ignore the XMP sidecar files for the most part, other than when weāre discussing them, like now.
Iām not knowingly doing anything with XMP files while using PhotoLab. If PhotoLab writes data to the XMP files, I have been ignoring them.
I have my D780 configured such that if I click on the center button on the back of my camera, it shows the center part of the image at 100%. Ditto, if Iām viewing images. My camera is almost always set to RAW capture only. I switch to ājpgā when Iām shooting give-away-photos, like a kidās birthday party.
Question for @Joanna - how large are the embedded jpg images? I can easily zoom into the embedded jpg images on my D780, and see the fine detail. I donāt think what @David_McA wrote applies to Nikonās D780.
For the Nikon full size and a reasonable quality. Watch the NEF in an image browser like Irfanview. It shows you the embedded JPG in its size. No reason to shoot RAW and JPG.
Just another question. If you look at your images in the D780 with the histogram on and zoom in, what happens with your histogram?
George
Mike, my recommendation is, harmonize your workflow and your data sources. Otherwise you will lose the overview. To work with DxO, the dop files are essential because everything is in there, including the advanced history. Only if you want to exchange META data with other tools do you need XMP files. You can define how DxO should handle XMP files in the settings.
I think the topic of XMP should not be neglected, since it does exist. You can prioritize it down for yourself. In any case, I would like to have a complete overview of these mechanisms and @Joanna has provided a lot more clarity.
As always, you write nonsense.
Windows simply removes the incompatible character when reading and everything works fine.
IF you want to work with it afterwards, youāll have to rename the file manually - YOUR problem as long as you insist on using incompatible filenames.
I would never allow any application to write to my RAW files. Itās like changing the constitution.
Or is it?
Nikonās own NX Studio can and does write to NEF files. I just did a test and it adds a couple of hundred tags in the NikonApp domain when it edits such a file, with every single editing parameter recorded. Then it changes the edited date on the original file.
So, Nikon doesnāt have any problems with writing to RAW files.
But why? Because the RAW file you see on disk is actually a container, with several compartments inside.
We have already seen that it contains one or more JPEG images, one of which is a full-size interpretation of the RAW data and that is what you see when looking on the back of the camera or at the file in most file managers or browsers.
But, most importantly, in a separate compartment, we find the actual undecoded image data. This is āring-fencedā and is not touched by any editor. Whereas, the metadata is found in a totally separate compartment and editing it doesnāt go anywhere near the RAW data compartment.
So, how safe is editing the metadata block in a RAW file?
The DAM tool that I wrote uses ExifTool to talk to the file and, with the right parameters, calling ExifTool first makes a copy of the original file and places it on the disk next to the original. Then it makes any necessary changes to the metadata block in the original file, verifies the file for integrity and then, optionally, erases the āsafetyā copy. if the edit corrupts the file in any way, the safety copy is left on disk, where it can be renamed to replace the failed edited original.
The fallacy that is promoted by āpuristsā is that editing an original RAW file violates its integrity. Whereas, as I have just described, the original image block inside the file is not touched in any way, shape or form - it is only the metadata block that is affected in a safe and verified way.
Which is why I have been editing metadata in my RAW files for years without having ever damaged a single one.
3 Likes
To you, that is normal. To me, Windows is still ābrokenā. I donāt expect to ever again buy a Windows computer, for many reasons. Current choices include Mac, Windows, and Linux.
When I call Apple about an issue, they will talk to me, connect to my computer, and eventually find a solution.
When I call Microsoft/Windows (as I did last year for my brother), they wanted me to pay just to talk to them, even though the problem was caused by a Microsoft update.
If I was a programmer, I might enjoy using Linux (like UNIX that I used to use), but Iām not - Iād be out on my own.
To me, the āpipeā character ( | ) is just as valid as any other ASCII character, and again, to me, Windows failure to accept it means Windows is broken. I know you disagree, but not even you could give me a reason as to why that character is blocked. Not that it matters - neither of us is going to change to the other. ā¦and I sill have my Lenovo W530 laptop in the other room, and one of these days Iām going to figure out how to use it again. Sorry for the diversionā¦
Maybe Iām wrong - maybe this is part of the reason?
" *
Related
Where is the pipe key on the UK keyboard?
*
Itās on the 102nd key.
Youāll often see keyboards described as 101/102-key. Thatās because they have 101 keys in the US and 102 keys in the UK. The extra key is between the shift key and the Z key.
There is only one character that we must have in the UK that the US doesnāt need: Ā£. Itās on shift-3. That moves # down to the key that would have been \ and | on a US keyboard. \ and | then becomes the 102nd key
That leaves shift-# without a character so it gets ~. The top left key then has the grave accent ` and the logical not operator Ā¬ that nobody ever uses, and for some reason, as one of only two keys to produce three characters, with Alt-Gr it produces the broken pipe Ā¦, which absolutely nobody ever needs.
We also swap ā and @ for some reason.
Since the keyboard layout was fixed the European Union created a new currency so now we also need a way to generate the Euro symbol ā¬. So that is on Alt-Gr-4."
You constantly entertain yourself with things that are absolutely unimportant.
Instead, focus on your photography and editing skills.
1 Like
Well, Iād use the word āincompetentā in that it is less well designed than macOS.
Windows used to have even more severe character restrictions and used ācode pagesā, which could differ according to the locale the computer was operating in. It was even more of a nightmare.
But you are already using UNIX! Or, at least a graphically wrapped superset of it. When you access the Terminal, you have access to the UNIX command line.
Except that both Windows and macOS now use Unicode, which is more comprehensive.
Unfortunately, unlike UNIX/macOS, which allows virtually all Unicode characters except
: (colon)
Windows reservesā¦
- < (less than)
- > (greater than)
- : (colon)
- " (quote)
- / (forward slash)
- \ (backslash)
- | (vertical bar)
- ? (question mark)
- * (asterisk)
- ^ (caret) *FAT
ā¦ for various purposes.
No that has nothing to do with it. It is purely at the OS level, not the keyboard.
Why on earth would you want to do that? Isnāt life complicated enough already?
Get on with making photos.
As @Joanna already pointed out, Nikonās NX studio writes data to Nikon NEF files. In addition, Canonās Digital Photo Professional (DPP), also writes data directly to Canon CR2 and CR3 raw files. Both of these software applications do not create sidecar files or have a database as a result.
Mark