Integrate sub-pixel morphological antialiasing (SMAA) in the Vulkan renderer

[Calinou]

Related to #3401.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Right now, we have MSAA and FXAA available in both 3.x and master. These algorithms serve their intended purposes, but each of them has their own limitations:

• MSAA looks good, but it is very expensive on master due to the added complexity of shaders. This is especially noticeable when using GIProbe or SDFGI. It also doesn't smooth out aliasing that originates from fragment shaders such as specular aliasing. (Alpha-tested surfaces are now well-handled thanks to alpha antialiasing and alpha-to-coverage support.)
• FXAA is cheap and looks decent on 1440p and higher. Unfortunately, the added blurriness makes it hard to use on 1080p and below. Post-processing sharpening algorithms make it possible to recover some of that lost sharpness, but they also add more aliasing on their own, so it's a tradeoff.

It's possible to use both MSAA and FXAA at the same time, but this has a significant cost and it won't improve the image's sharpness after it has been reduced by FXAA.

It's great that MSAA is still supported in master (thanks to the clustered forward renderer), but we have to admit that it's difficult to use in production with today's shader complexity. Therefore, this makes MSAA mostly suited to games that use few visual effects, which generally implies a stylized art direction.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Sub-pixel morphological antialiasing (SMAA) has been around since 2011 and is now a proven post-processing-based antialiasing algorithm. It's still used in a lot of AAA games.

Performance-wise, SMAA is more expensive than FXAA, but it does a good job at antialiasing despite introducing less blurriness. Either way, SMAA is likely to be less expensive than even 2× MSAA on the Vulkan renderer (despite providing better edge smoothing overall).

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

One downside of SMAA is that it's notoriously difficult to implement. Not only it's a multi-pass algorithm (unlike FXAA which is single-pass), there are also many quality settings to tinker with.

We can probably go with the recommended SMAA settings (luma-based edge detection), and only see if there are real benefits to exposing quality knobs in the project settings (such as changing the edge detection algorithm).

SMAA also offers an optional temporal component, but that comes with added complexity and motion trails commonly associated with temporal antialiasing algorithms. Therefore, I suggest we leave out the temporal antialiasing part of SMAA and focus on spatial-only SMAA (also called SMAA 1×). SMAA won't look as smooth without this temporal component, but SMAA still looks better than FXAA without it.

I've looked around a bit and couldn't figure out how to add a multi-pass post processing shader in Godot's GLSL core, so feel free to give it a try.

Question: Can this be implemented in 3.x?
I haven't seen any implementation of SMAA on OpenGL ES 3.0 (and even less OpenGL ES 2.0), so probably not.
Edit: It's possible to implement SMAA in WebGL 2.0 (and therefore OpenGL ES 3.0): https://github.com/dmnsgn/glsl-smaa

If this enhancement will not be used often, can it be worked around with a few lines of script?

Since real-time multi-pass shaders aren't possible in "userland" Godot yet, no.

Is there a reason why this should be core and not an add-on in the asset library?

See above.

[clayjohn]

For reference, the original SMAA code can be found here and it is MIT licenced.

[LiveTrower]

I found this: https://github.com/dmnsgn/glsl-smaa, i hope it helps.

[Calinou]

This porting work may also be useful as a reference: libretro/slang-shaders#189

[jntesteves]

This porting work may also be useful as a reference: libretro/slang-shaders#189

Hey, if you want to take any reference from my work there, please check the aftermath: libretro/slang-shaders#192

I can't agree with the docs on luma by default. I tested it extensively and color based edge detection always gave me better results, with less aliasing but not blurrier. In the end, I exposed every config knob, so I could more easily tweak settings at runtime. Tweaking it for your content is worthwhile, this AA is powerful when properly setup.

SMAA_PRESET_ULTRA + SMAAColorEdgeDetectionPS seemed like a good default to me. I increased AA strength a nod (SMAA_LOCAL_CONTRAST_ADAPTATION_FACTOR 2.5) where I knew there would be sharpening in a following pass, like SMAA+FSR.

My code is in the public domain, no restrictions apply. Feel free to copy anything.

[Calinou]

I've started work on integrating SMAA: https://github.com/Calinou/godot/tree/add-smaa-antialiasing

It's far from functional yet, but feel free to take a look nonetheless.

[Lasuch69]

Is there any update on this?

[Calinou]

Is there any update on this?

I haven't managed to get it working, as I have very little time to work on this.

Note that Godot 4 now offers temporal antialiasing, which you could combine with sharpening to counteract the added blurriness. This would still result in some amount of visible ghosting depending on the scene, but TAA handles specular and shader-induced aliasing much better than SMAA (which barely has any positive effect on this kind of aliasing).

TAA and FXAA sharpness can also be improved by adjusting mipmap LOD bias automatically.

I still think supporting SMAA is valuable for at least 2 scenarios:

• You want to maximize antialiasing quality (even in motion) by using both SMAA and TAA at the same time, at the cost of some blurriness. This is fairly expensive, but it can look very good in my experience, with nearly no noticeable aliasing on moving edges.
• You can't afford MSAA, you find FXAA too blurry but you still have enough GPU time available for SMAA. In this case, SMAA is a good compromise. (FXAA + sharpening can counteract the blurriness issue, but it will not resolve the issue where FXAA makes sharp edges appear rounded.)

[jntesteves]

I don't have any stake in Godot at the moment, so my opinion might not be too relevant, but I think working on the multi-pass shaders proposal mentioned in the description is a more worthwhile use of time. With that, anyone could easily do SMAA or whatever other post-process effects they want, not limited by what the engine offers.

Edit: Maybe these kinds of shaders should be on the Asset Library.

[Jamsers]

CMAA2 may be a great alternative to consider, it's been used to great effect in the recently released Counter Strike 2 and seems to work well as an MSAAesque cheap post process fallback.

The original repo published by Intel: https://github.com/GameTechDev/CMAA2

Intel's article on CMAA2
An article describing a hybrid CMAA implementation for mobile

A ReShade implementation: https://github.com/LordOfLunacy/Insane-Shaders/blob/master/Shaders/CMAA_2.fx

[jams3223]

How is this coming along? I was going to open a proposal until I saw this one. SMAA is the perfect middle ground in terms of performance, quality, and artifacts.

Q: Describe the project you are working on
A: 3D games that make use of realistic art styles and complex shapes

Q: Describe the problem or limitation you are having in your project
A: TAA tends to create ghosting artifacts while in motion, and while TAA is better at removing aliasing than SMAA, it still does a good job without having these issues; it's a good middle ground between TAA and FXAA. TAA is more resource-intensive and prone to artifacts, while FXAA is less effective.

Q: Describe the feature / enhancement and how it helps to overcome the problem or limitation
A: Add it to the list of already-available anti-aliasing algorithms.

Q: If this enhancement will not be used often, can it be worked around with a few lines of script?
A: No, it can't be used with a few lines of code; it needs to be implemented into the core engine.

Q: Is there a reason why this should be core and not an add-on in the asset library?
A: It sits right along with the other anti-aliasing methods that Godot ships with, so it helps extend the user's preferred choice and ensure a balanced level of quality.

[Calinou]

How is this coming along? I was going to open a proposal until I saw this one. SMAA is the perfect middle ground in terms of performance, quality, and artifacts.

SMAA is generally considered to be a stale approach by 2023 standards, so I wouldn't hold my hopes for it being added as a core feature (in the age of ever-improving TAA solutions). However, the rendering engine will eventually be made more flexible so that SMAA can be implemented by an add-on.

Modern games are often full of geometric and specular detail that SMAA can't do anything about. The only real use case I can see for SMAA nowadays is antialiasing an image when you have no access to motion vectors (such as running old games, antialiasing screenshots, etc). Similar concerns apply to other techniques like CMAA.

Being able to use TAA + SMAA at the same time isn't so important anymore, now that we have access to FSR2 at native resolution. This provides high-quality antialiasing with a sharper result compared to native TAA.

TAA is pending optimizations in the motion vector generation part, which will make it significantly less demanding.

[Jamsers]

Modern games are often full of geometric and specular detail that SMAA can't do anything about

That's fine - the users wanting to use SMAA aren't expecting it to match or even come close to the antialiasing capability of TAA - the point is to get the best possible antialiasing possible today that isn't TAA (due to ghosting and artifacting that will never be solved no matter how improved TAA gets) or MSAA (due to prohibitive performance cost)

SMAA is generally considered to be a stale approach by 2023 standards

Irrelevant. Progress and development on non-temporal post process antialiasing methods has practically halted, so unless the Godot team is planning to create a bespoke new technique, SMAA is still the best there is. (again, if you don't want TAA or MSAA)

If anything, FXAA is the stale, outdated, irrelevant technique - why is that still in core?

[jams3223]

I have tested the anti-aliasing algorithm with every kind of game art style, and I can say that it all depends on the art style or visual style. Godot will benefit by giving people choices in how they want to customize their game.

FXAA:
(+) Better with cel-shaded and anime art styles
(-) Lower Quality
(+) Better Performance

MSAA:
(+) Better with realistic art styles
(+) Higher Quality
(-) Lower Performance

TAA:
(+) Better with realistic art styles
(+) Higher Quality
(+) Balanced Performance
(-) Visual Artifact

SMAA:
(+) Better with realistic, anime and voxel art styles
(+) Balanced Quality
(+) Balanced Performance