Add ability to transform scene color in shaders.

[sfence]

Add a color transform matrix to shaders, which can be used to transform scene color.

• Goal of the PR
  Allows modders to apply custom color transformation matrices for visual effects such as color blindness simulation and night vision.
• How does the PR work?
  player:set_lighting method table now reads table vision_effects. See more in the doc.
• Does it resolve any reported issue?
  It should help with the possibility of implementing Feature Request: Night Vision. #6509.

To do

This PR is Ready for Review.

How to test

Get server privilege and use the command player_editor lighting to change vision_effects.

[sfence]

Maybe, it will be better to add a new player method set_visual_effects/get_visual_effects instead of extending the player:set_lighting method to add this feature. Or make set_lighting/get_lighting obsolete and replace it with set_visual_effects/get_visual_effects.
What are the opinions of others?

[appgurueu]

Personally I think this is fine in set_lighting, though it may feel like a minor misnomer. Maybe we should have [sg]et_visual_effects, of which lighting would just be a part?

[appgurueu]

Needs to be more precise. What I imagine given this description is something like:

{
  a, b, c,
  d, e, f,
  g, h, i,
}

but this also does not seem to match the code, which expects

{
  {a, b, c},
  {d, e, f},
  {g, h, i},
}

Personally I would prefer the first format (the second may be more convenient if you have rows lying around, but you might as well concatenate them).

Either way from this documentation it is unclear whether this is a matrix that deals with row or column vectors. In other words, what is the image of red = (1, 0, 0)? Is it the first row or the first column?

As said, an example would help here (but this should also be stated explicitly).

[sfan5]

^ this is not adressed

[sfence]

I personally prefer the version with vectors, because it nicely separates constants for each color.

[sfence]

Personally I think this is fine in set_lighting, though it may feel like a minor misnomer. Maybe we should have [sg]et_visual_effects, of which lighting would just be a part?

Yes, something like make lighting part of [sg]et_visual_effects.

So when something like horizontal/vertical scene flip will be added as a vision effect or something like that, it will be under one table.

[lhofhansl]

Can/should we retrofit the saturation setting into this?

[sfence]

@lhofhansl
Saturation is still applied in shaders after this effect.

[HybridDog]

Clamping the values in [0,1] often causes hue distortions, see for example https://bottosson.github.io/posts/gamutclipping/

I think with gamut clipping in a perceptual colour space, e.g. OKLab, the clipped result would be much more similar to how it would look like if it clamping was not needed.

Even a simple clamp in YCbCr may look better than clamping in RGB, but I haven't thoroughly tested it:

// RGB to YCbCr, ranges [0, 1]
vec3 rgb_to_ycbcr(vec3 rgb) {
    float y = 0.299 * rgb.r + 0.587 * rgb.g + 0.114 * rgb.b;
    float cb = (rgb.b - y) * 0.565;
    float cr = (rgb.r - y) * 0.713;

    return vec3(y, cb, cr);
}
// YCbCr to RGB
vec3 ycbcr_to_rgb(vec3 yuv) {
    return vec3(
        yuv.x + 1.403 * yuv.z,
        yuv.x - 0.344 * yuv.y - 0.714 * yuv.z,
        yuv.x + 1.770 * yuv.y
    );
}
// Clamp the saturation in YCbCr before a clamp in RGB for a better chroma
// preservation
vec3 clamp_saturation(vec3 color)
{
	vec3 yuv = rgb_to_ycbcr(color.rgb);
	yuv.yz = clamp(yuv.yz, vec2(-0.5), vec2(0.5));
	return ycbcr_to_rgb(yuv);
}

[sfence]

Without clamp:

With clamp:

[HybridDog]

With certain matrices, hue distortions are currently very noticeable.
/lua local t={1,0,8} me:set_lighting({color_transform_matrix = {t,t,t}})
Pull Request (commit e6b59de):

With clamp_saturation() applied:

Other examples with noticeable distortion (values for t in the above command): {1,4,0}, {1,2,0}, {3,8,1.5}

In my opinion it looks better with clamp_saturation(). As far as I know, if light of any colour reaches the human eye and is too bright for the cone cells, the human usually perceives it as white light.

This discussion could be related: w3c/csswg-drafts#10579

[HybridDog]

Should the documentation mention that the matrix is applied before the saturation and tone mapping?

And should it mention the behaviour with out-of-bounds colours and if this behaviour is subject to change?

[HybridDog]

What is the meaning of "vision" in the function name applyColorVision?
Can these two lines be inlined into main()?

[HybridDog]

I think it currently behaves like transformed_color_RGB = color_transform_matrix^T * color_RGB

[ryvnf]

A more general solution would be to support color lookup tables (LUT) supported by engines such as Unreal Engine.

It is simply a texture (256×16 in Unreal) which is indexed by the RGB components of colors and maps them to a different color.

https://dev.epicgames.com/documentation/en-us/unreal-engine/using-lookup-tables-luts-for-color-grading?application_version=4.27

Considering that can be used for the same things but also lots of other effects it may be worth considering as an alternative. I am pretty sure what you can do with a 3×3 matrix is very limited by comparison.

[sfence]

Yes, color lookup tables sound like a more flexible solution.
I will return to this later. Now I do not have time for it.