Clipped highlights, but only in Photolab?

[noname (Captain Obvious)]

@Wlodek

Do you have an idea of what setup to use?

well - if you can get at least one 1 or better 2 channels within <= 1/2 stop to clipping in raw ( but do not clip ) across some relevant part of the frame ( like OP got that over kid’s facial skin and that was clearly an important part of the picture , not to be discounted as some irrelevant specular highlights )

I can make a synthetic raw raw file for D780 with any kind of content, but some people think that 2 + 2 apples = 5 apples when you tell them that 2 + 2 oranges = 4 oranges … even when in this case what matters is just the math : 2+2 = , so the file like the excellent raw supplied by OP surely will incite some more fire in our old bones / obvious reference to the day today /

so if a valuable part of the frame has at least one raw channel close to clipping ( denoted w/ EV0 ) like “(G1+G2) / 2” here ( better even a bit closer )

that is an excellent test

[noname (Captain Obvious)]

@Wlodek

I wouldn’t have spent my money

one can hope that not all is lost with DxO - and a proper sustained crusade shall turn them over and make them see the light of the Saint Raw Data …

don’t mess with my [ unclipped ] raw data and make a use of the data that is available in raw files ( focusing distances and manufacturer’s supplied optics correction data ) - I do not ask for more ( for now , there is a matter w/ DCP profiles, but several steps at a time )

[john4pap (John)]

@noname

it will be then interesting to compare one from ones previous to see how different the sensor saturation was in that same area ? do you mind to post the raw and we shall investigate the difference ( or absence ) in raw channel saturation… thank you for making our Presidents’ day less … / obviously struggling to find a politically correct epithet / … boring

Certainly. From the looks of it I’d say this one is just a couple of seconds before the one from my original post. Same settings.

_PAP2058.NEF (30.7 MB)

[noname (Captain Obvious)]

@john4pap

From the looks of it I’d say this one is just a couple of seconds before the one from my original post. Same settings.

something was different - see statistics across the frame in this 2058 NEF

and skin area =

and in 2059 NEF

and skin area =

sensor saturation did not change for skin area, but something else ( ambient light, framing of the area around the face, etc ) did change so some other areas in the sensor outside of the skin area in question got more saturated and that presented a problem for DxO code

so in 2059 DxO code calculates apparently data as clipped and in 2058 as not ( because it thinks that it is way down below clipping point - which of course it is , but in BOTH raw files )… as usual - somebody in DxO either has a math difficulties or was not aware how and where to get clipping level for raw files

so this where the bug probably lies more precisely , it is even worse it seems : (A) DxO code can’t decide where actual clipping point is and then (B) still clips the UNCLIPPED data that lies close to ( properly or incorrectly calculated ) clipping point

even you convert NEF to DNG using Adobe DNG Converter and kind Adobe engineers calculate that for DxO ( while level tag :: 0xc61d WhiteLevel = 15892 ), DxO still ignores that and calculates clipping incorrectly and then ( icing on the cake ) still clips unclipped data close to incorrectly calculated clipping point …

---

in 2058 NEF skin area lies ~0.8 stop below clipping if DxO calculates clipping point based on the max DN found in the frame

in 2059 NEF skin area lies ( it is NOT, but we about DxO point of view ) ~0.3 stop below clipping if DxO calculates ( incorrectly ) clipping point based on the max DN found in the frame

so for 2059 NEF DxO decides to clip unclipped data because it calculates white point incorrectly and then does the bad thing and for 2058 NEF it does not because god bless something that saturated the sensor further somewhere else in the frame

PS: that something for in 2058 NEF was the front/light/window facing side of the box… it is fully saturated in greens and almost in blue channels in the frame , so gives imperfect DxO code a hint where white level is

[noname (Captain Obvious)]

President’s Day was productive… we got a better picture of DxO multiple sins in the code

[sgerman (Steven)]

It may be a problem in DxO’s camera profile for Nikon Zf. Just as an experiment I changed the camera model in the NEF file to Nikon Z 6, and
the DxO PL v7.4 renders the image without blown highlights.

_PAP2059_renamedZ6.NEF (30.6 MB)

[eriepa (WIN11, PL7, FP7, VP4, NIK1)]

@john4pap

Certainly. From the looks of it I’d say this one is just a couple of seconds before the one from my original post. Same settings.

_PAP2058.NEF (30.7 MB)

I had a feeling you would be making noname’s day! Thanks for providing.

[noname (Captain Obvious)]

@sgerman

It may be a problem in DxO’s camera profile for Nikon Zf.

dear - test were executed also by exporting to linear DNG w/ NR and optics correction applied… if you are not aware that is before any color transform ( that is what camera profile is , happens AFTER demosaick ) …

so if anything it is not the camera profile - but DxO coding the processing differently for different camera models ( YOU → camera model in the NEF file to Nikon Z 6 ) and may be they fixed that matter for Z6 … as I noted - they need to fix it for ALL CAMERA MODELS instead of taking a piecemeal approach to shut up the person who files a bug report about a specific camera !

[noname (Captain Obvious)]

@eriepa

making noname’s day!

obviously (c)

[noname (Captain Obvious)]

@sgerman

to Nikon Z 6

BTW - thank you for the tip, now we can investigate this camera model on synthetic raw file just to be sure that DxO fixed white point detection/calculation for that camera model AND also does not clip unclipped data close to it !

also we shall test a theory about wrong WP calculation by filling the ( ALL of it ) frame w/ data well below clipping point for the sensor and then filling with patches from black level up to that level for a Zf … will see if we can tip DxO code to make an error

if I have time more time this evening

[Wlodek]

As you can see from the attached examples, I have no real issues with overexposed photos up to about +2EV. One shouldn’t expect medium level consumer camera to have more headroom in highlights.

Test:
PL7.4/Win11.
Nikon D780 FW 1.10, AFS Micro 60/2.8G (non-cpu 50/1.8 in OP), @f/4 (f/1.8 in OP), ISO 3200 (like in OP), AWB, matrix metering, A-mode.
Room lighting, 2700K 95 Ra LEDS (but never seen spectral data), some color casts.
Subject: one of those color checker gadgets I’ve never taken seriously, 10y+ old, so colors may be quite off the original. Perhaps too flat in DR and unreal scene for a decent test, but still I found it useful.
The jpegs were generated by PL7.4 from NEFs taken with 0 or positive exposure compensation given in the filenames (EC=0, EC=+1EV, …), exposure corrected by the same amount in PL (although the units might slightly differ). WB was corrected by probing the square below the black one in the top-right corner. No other PL settings were changed, in particular no NR or optical corrections were applied. The jpegs were made with 50% quality but that shouldn’t matter.

RAW files reported LightValue 5.6, 4.6, 3.6, 2.7, 1.6 respectively (in OP it was 3.7),
shutter speeds starting with 1/100 sec and changing as expected, RedBalance=1.2, BlueBalance=2.2. (in OP these were 1.726 and 1.336 respectively – daylight).
Hence the green channel was blown out first, then red, and the blue was quite persistent.

I have a habit of NOT taking photos with key information DR too wide,
but the OP photo certainly doesn’t fall in that category.
I hadn’t yet time to look at the “good” photo.

Bottom line:
The problem, which looks real for OP, seems to be specific to some camera/lens/settings combinations.

Side remark:
CO (short for Captain Obvious, to be sure) enthusiastic generalizations seem to be too premature, if not simply wrong, but he’s obviously right in the OP case, at least to me.

[noname (Captain Obvious)]

OK, here is ZF synthetic raw = https://app.box.com/s/qg4ji6cmpayii8plqgxqkoebtj1pdu51

I intentionally make maximum raw values in raw file even less than in OP’s raw file ( _PAP2059.NEF ) =

as you can see max raw values are whole 1+ stop below true clipping for Zf

and what do we have for this camera model Zf in this raw ? DxO as expected promptly ( A ) makes an error in estimating what is the white level / clipping point for raw data, assumes wrong number for that and then promptly ( B ) destroys raw data even further below this wrongly calculated white level / clipping point ( destroyed data is located 2 stops and onwards below actual white level / clipping point )

[noname (Captain Obvious)]

@Wlodek

I have no real issues with overexposed photos up to about +2EV

first thing first… if no clipping in raw channels the shot is NOT overexposed

and then OP has real issue with ~2/3 stop underexposed raw shot

[noname (Captain Obvious)]

@Wlodek

As you can see from the attached examples

I do not see raw files … no raw files → empty words … are we in a violent agreement on that part ?

[anon78744791]

@noname, I can’t because I don’t know enough, but perhaps you can explain the following:

When using the colour picker under HSL in PhotoLab on the area that is of interest here and looks white on screen, it selects a red/yellow range.

If the image is opened in Darktable and:

• demosaic is set to “photosite color debug” (EDIT: just noticed also happens with other methods. Darktable was not rendering the whole image sometimes, so I didn’t see anything)

• “Visualize clipped highlights in a false color representation…” enabled

• threshold set to around 0.6 or lower (with the default method or segmentation based)

you can see something happening in the problematic area (on the marker pen it starts with a higher threshold):

[noname (Captain Obvious)]

@anon78744791

When using the colour picker

(A) I pray to RawDigger - it something is not clipped in raw channels then it is not clipped in raw channels and whoever clips and destroys that data shall be promptly tarred, feathered and burned on the nearest stake ( slowly )

(B) I export from DxO PL after applying NO CORRECTIONs ( for the purpose of this investigation ) preset to DNG with NR+Optics Corrections only ( and I disable them both for the purpose of this investigation ) - if the data present in the source file is destroyed in that linear DNG ( see screenshot below ) then whoever clips and destroys that data shall be promptly tarred, feathered and burned on the nearest stake ( slowly )

(C) I have no clue what our talented open-source komrades do in DarkTable with raw data, but if they do the same as DxO PL code does then whoever clips and destroys that data shall be promptly tarred, feathered and burned on the nearest stake ( slowly )

[anon78744791]

@noname

I have no clue

Perhaps the fact that enabling “Visualize clipped highlights in a false color representation” under “highlight recovery” shows this area on the skin (and marker pen) provides one.

[noname (Captain Obvious)]

@anon78744791

Perhaps

I do not know what they do , so no comment … the data in OP’s original raw was not clipped not in a single raw channel on the face - so whatever they do if they intend to say that there was any clipping in raw in that facial skin area then they shall stop (writing the code)

[sgerman (Steven)]

Yes I can see that the problem is triggered by the camera model declared in the NEF file, and not by the color profile used for processing.

For example, if I processs the original _PAP2058.NEF file in PL, it shows blown highlights. I can change the DxO PL setting Color/B&W Rendering
to another camera model. Changing to Nikon Z6 does not remove the blown highlights.

Whereas, presenting PL with the edited NEF file where camera model is
declared Z6, does remove the blown highlights.

[noname (Captain Obvious)]

@sgerman

Changing to Nikon Z6

for Z6 (mk1) raw files DxO might be properly calculating where the clipping is but still does clip/destroys unclipped data below that point ( last 1/3 stop of data below clipping is eliminated )

Z6 synthetic raw file to illustrate that DxO PL still clips = https://app.box.com/s/tff3qibgs2mxh7yxd8x1u61rz9d9akix ( maiden file = https://www.dpreview.com/sample-galleries/8048435192/nikon-z6-sample-gallery/2515224383 )