I'm running KDE Wayland Plasma 6.7.3 in HDR mode with an AMD RX 9060 XT on Gentoo (FF 153.0.3).

When the following prerequisite are met, SDR youtube video have flashy/oversaturated colors.

Prerequisites:

• Wayland compositor with HDR mode enabled
• Firefox with HDR enabled
• Hardware video decoding disabled
• "Open previous windows and tabs" enabled

Steps to reproduce:

• Open a YouTube HDR video.
• Close the browser before the end of the video.
• Re-open Firefox (the browser will restore the YouTube tab with the HDR video).
• Play any SDR video from this restored page.

The SDR video displays flashy/oversaturated colors like it's being played in HDR.
After closing and reopening the browser (on a SDR video or on another site than youtube) colors are normal.

I reproduced this behavior on Firefox 153.0.3 up to mozilla-central build.

Depending of the order (or another factor), there are 2 problematic patterns. Only software decoding mode is impacted.

-> Render SDR and HDR video in NV12
The videos SDR and HDR are rendered to a NV12 (3231564e) surface.

Below a trace of a HDR video rendered in NV12, it shows banding
[15:07:55.463077] {Default Queue} -> zwp_linux_dmabuf_v1#52.create_params(new id zwp_linux_buffer_params_v1#3661)
[Parent 291482: Renderer]: D/Dmabuf [7f0fa4ebb300]: layer [0] modifier 200000000082305
[15:07:55.463113] {Default Queue} -> zwp_linux_buffer_params_v1#3661.add(fd 313, 0, 0, 2560, 33554432, 533253)
[Parent 291482: Renderer]: D/Dmabuf [7f0fa4ebb300]: layer [1] modifier 200000000082305
[15:07:55.463123] {Default Queue} -> zwp_linux_buffer_params_v1#3661.add(fd 341, 1, 0, 2560, 33554432, 533253)
[Parent 291482: Renderer]: D/Dmabuf [7f0fa4ebb300]: zwp_linux_buffer_params_v1_create_immed() [2560 x 1440], fourcc [3231564e]
[15:07:55.463131] {Default Queue} -> zwp_linux_buffer_params_v1#3661.create_immed(new id wl_buffer#3685, 2560, 1440, 842094158, 0)
[Parent 291482: Renderer]: D/Dmabuf [7f0fa4ebb300]: created wl_buffer [7f0faa7bd900]
[15:07:55.463142] {Default Queue} -> wl_surface#834.attach(wl_buffer#3685, 0, 0)
[15:07:55.463149] {Default Queue} -> wl_surface#834.commit()
[15:07:55.463165] {Default Queue} -> wl_surface#527.damage_buffer(0, 256, 1024, 256)

-> Render SDR and HDR video in P010
The videos SDR and HDR are rendered to a P010 (30313050) surface.

Below a trace of a SDR video rendered in P010, it looks in this example oversaturated
[15:11:58.893420] {Default Queue} -> zwp_linux_dmabuf_v1#52.create_params(new id zwp_linux_buffer_params_v1#2231)
[Parent 296409: Renderer]: D/Dmabuf [7fd3da745800]: layer [0] modifier 200000000082305
[15:11:58.893427] {Default Queue} -> zwp_linux_buffer_params_v1#2231.add(fd 279, 0, 0, 5120, 33554432, 533253)
[Parent 296409: Renderer]: D/Dmabuf [7fd3da745800]: layer [1] modifier 200000000082305
[15:11:58.893434] {Default Queue} -> zwp_linux_buffer_params_v1#2231.add(fd 288, 1, 0, 5120, 33554432, 533253)
[Parent 296409: Renderer]: D/Dmabuf [7fd3da745800]: zwp_linux_buffer_params_v1_create_immed() [2560 x 1096], fourcc [30313050]
[15:11:58.893440] {Default Queue} -> zwp_linux_buffer_params_v1#2231.create_immed(new id wl_buffer#2232, 2560, 1096, 808530000, 0)
[Parent 296409: Renderer]: D/Dmabuf [7fd3da745800]: created wl_buffer [7fd3d06ab620]
[15:11:58.893446] {Default Queue} -> wl_surface#2080.attach(wl_buffer#2232, 0, 0)
[15:11:58.893449] {Default Queue} -> wl_surface#2080.commit()
[15:11:58.893453] {Default Queue} -> wl_surface#1890.damage_buffer(512, 256, 512, 256)

It could be only on my configuration

Given that this is not pref'd on by default, this would be a shipping blocker but we haven't shipped this yet.

Which of the 2 pattern is chosen seems to depend on whether the last played video before closing is SDR or HDR.
I will investigate the technical root cause.

(In reply to Louis-Philippe Manette from comment #3)

Which of the 2 pattern is chosen seems to depend on whether the last played video before closing is SDR or HDR.
I will investigate the technical root cause.

If you are wanting to take a look at fixing this in the code, make sure that https://searchfox.org/firefox-main/rev/ad5db99e4d2e00e9b5db2700c69050d709c7f340/widget/gtk/WaylandSurface.cpp#1522 is working correctly when going from HDR to SDR, I suspect it may not be called when the new video is SDR here: https://searchfox.org/firefox-main/rev/ad5db99e4d2e00e9b5db2700c69050d709c7f340/gfx/layers/NativeLayerWayland.cpp#1073

I'd like to double-check my understanding of the flow here, just to be sure I'm following correctly.

Looking at NativeLayerWaylandExternal::AttachExternalImage, mIsHDR is set based on surface->IsHDRSurface() (which checks for PQ/HLG transfer functions). If it's a BT.709 surface, mIsHDR evaluates to false, meaning WaylandSurface::EnableColorManagementLocked() won't be called for it?
This would align with your suspicion.

The initial problem leading me to this bug was that I encountered HDR playback resulting in severe banding when the surface format ended up being NV12 (this happened when closing a YouTube tab in SDR and reopening it to play an HDR video). Since NV12 is 8-bit, I assume this NV12 + HDR scenario is a separate bug?

You can get layer log by running with MOZ_LOG="WidgetCompositor:5" - that reveals which layer is directly exported. MOZ_LOG="WidgetWayland:5" shows you how WaylandSurface handles it, so maybe run as:

MOZ_LOG="WidgetCompositor:5,WidgetWayland:5,Dmabuf:5" ./firefox

to have a complete log.

Thanks a lot for the hints!

BT709 is properly colored when rendered as NV12. HDR doesn't have banding when rendered as P010.
The format variable in VideoFramePool::GetVideoFrameSurface was declared as static, causing the surface format to be evaluated only once during the initial call. This prevented the surface format from updating after the first image rendering.
Removing the static seems to fix the bug. I will submit in phabricator.

https://hg.mozilla.org/mozilla-central/rev/a7a9f20cfd58