I did this long ago, back when I wanted to calibrate the iMac. I gave up.
The iMac is operating in whatever OS it came with when new. If I update the OS on the iMac, I can’t use it as a remote display as I’m doing.
I connect a video signal from my Mac Mini to the HDMI connector on the back of the iMac, then forget about it. When I simultaneously press “F2” and “Command” on the keyboard permanently connected to the iMac, both of my screens switch off, and then return with the iMac showing an output from my Mac mini, meaning I have two screens from the Mini, on which to do whatever I want. …but doing this disconnects me from being able to control anything on the iMac. Oh, and I have to manually adjust the brightness on the iMac to whatever I want before they are connected. The iMac does nothing from then on, other than act like an external display from my Mac mini. Sometimes like when I reboot the Mac mini, I forget to adjust the screen brightness on the iMac before connecting them.
I’m not even close to being an expert at any of this - I just know if I do what I just described, I have my two displays. I don’t know if anything is changing the brightness of the iMac, automatically or otherwise.
Next time I do this, I will see if I can do what you suggested. Sometimes my iMac looks much too bright, and other times it looks much too dark. It’s annoying, but lots of things in life are annoying. :-). Thank you - next time I’m connecting them, I’ll try your idea.