I'm new to the engine and want to write custom shaders

Graphics
#1

Hi,

After spending a few months with Unity, I’m now considering switching to a different engine.
I tested Flax and got a positive first impression.
So far, I have only written an FPS character controller as a test.

Specific shaders are essential for my project, as they need to create a specific aesthetic.

I haven’t yet figured out how to write custom shaders. Ideally, I would like to write my own HLSL shaders with vertex and fragment stages.

Maybe someone experienced can help me understand how to achieve the following features:

  • Vertex-Snapping (PSX like)
  • Vertex Colors (from mesh)
  • Vertex-Lighting
  • Affine Texture Mapping (also PSX like)
  • Scalable Pixelation¹
  • PSX-like dithering and color space transformation²
  • Quantized Light-/Shadowmapping³
  • User-adjustable fullscreen shader (parameters like scanline-strength, etc.)

¹ Scalable Pixelation

The resolution limit is scalable and done this way in my Unity shader:

// Apply resolution limit to the base texture.
int targetResolution = (int)log2(_ResolutionLimit);
int actualResolution = (int)log2(_BaseMap_TexelSize.zw);
int lod = actualResolution - targetResolution;

half4 baseColor = SAMPLE_TEXTURE2D_LOD(_BaseMap, sampler_PointRepeat, i.uv, lod) * i.color;

I don’t know if there’s a similar LOD texture sampling system in Flax?

² PSX-like dithering and color space transformation

I won’t use dithering on single textures since my textures are already have dithering applied.
However, I’m using an user-adjustable fullscreen dithering shader:

col is the fullscreen blit, colLow is a downsampled (pixelated) version sampled with Unity’s LOD texture sampling as seen above. This results in a very nice dithering effect without calculating colors.

float3 PsxDitherFullscreen(float3 col, float3 colLow, float strength, uint2 p)
{
    col    *= 255.0;
    colLow *= 255.0;
    uint dither_u = psx_dither_table[p.x % 4][p.y % 4];

    col   += ((dither_u / 2.0 - 4.0) * strength);
    col    = lerp((uint3(col)    & 0xf8), 0xf8, step(0xf8,col)); 
    colLow = lerp((uint3(colLow) & 0xf8), 0xf8, step(0xf8,colLow));

    col    = lerp(col, colLow, step(0xf8, (dither_u / 15.0 * strength)));

    col /= 255.0;
    return col;
}

You can see my fullscreen shader in action here:
https://youtu.be/yauQqGNjKvA

For the colorspace transformation (on textures, not fullscreen) I’m using modified parts of this code:
https://github.com/Mortalitas/GShade/blob/master/ComputeShaders/PSXDither.fx

// truncate to 5bpc precision via bitwise AND operator, and limit value max to prevent wrapping.
// PS1 colors in default color mode have a maximum integer value of 248 (0xf8)
col = lerp((uint3(col) & 0xf8), 0xf8, step(0xf8,col)); 

// bring color back to floating point number space
col /= 255;

³ Quantized Light-/Shadowmapping
I already have shaders in Unity which support quantized mapping. It’s done using this technique:
https://www.reddit.com/r/Unity3D/comments/1gdj7ik/comment/lu4csgp/

The image of the initial reddit post shows the desired effect. This is, of course, only intended for use with fragment stage lighting, not vertex lighting. It is a nice solution to simulate baked shadows on lowres textures like some PS1 games did. Metal Gear Solid and Alien Resurrection come to my mind.

Thanks a lot for reading!

#2

Go ahead and see our docs about shaders: Shaders | Flax Documentation

In that sections, you can find code examples for custom geometry drawing (custom Vertex and Pxiel shaders too) or Compute Shaders. You can use typical HLSL code with some Flax-specific macros. Code from Unity should be rather easy to transfer.

For example, that SAMPLE_TEXTURE2D_LOD is Unity-specific macro which could be done as:

_BaseMap.SampleLevel(MySampler, uv, lod), where you need to define that texture and sampler to be bonded from code int oa specific registers. See HLSL official docs to learn more.

#3

I figured out how to import HLSL files and implemented a first post processing shader.
Now I’m trying to implement a custom surface shader but as far as I see, there’s no documentation on it?
This is something really essential, so I guess it’s possible, somehow?
Where can I find some information on that topic?

#4

We have an in-built material generation that uses templates depending on material domain:

You can output material properties from custom shader to be passed there (as MaterialLayer connection).

#5

Thanks! This looks like a solution I’d get along with very good.

Is there a way for me to create my own material template within my project? Like a modified version of the surface material template? This would be very nice. Can you please provide a more detailed advice on how to do this?

#6

You would need to modify the engine code (fork engine on Github). and:

  • add new MaterialDomain (ideally use higher value of enum to leave some space for in-built engine future domains)
  • implement it in all switches across the engine and editor (eg. displaying proper material inputs or properties (in material editor)
  • implement custom MaterialShader class just like others in Engine\Graphics\Materials folder
  • implement custom shader template just like in Content\Editor\MaterialTemplates folder
  • you can define own Constant Buffers, Pixel and Vertex Shaders and so on.
  • I would use Flax 1.10 (use branch) because there were changes to GPU Vertex Buffers binding and it will be better to start off
  • use material on objects (your void MaterialShader::Bind(BindParameters& params) will do the binding)

Have fun! In case of questions feel free to ask them here.

1 Like
#7

Thanks a lot for your reply.

I’m currently looking into this. While it seems doable it is a lot of effort for me since I’m not very experienced with larger C++ projects. Also, I realized, implementing a variation of the Surface shader to achieve specific features would solve only parts of my problem.

What I really need in the end would be a solution which allows to easily implement a bunch of smaller shaders for different use cases. Modifying the engine for each shader really isn’t a option I’d take.
To be honest, I think not having a workflow like this is a big downside of the engine.

  • Create a material → Assign a HLSL shader file with needed passes
  • Then, create material instances which take textures and parameters per instance

Is there any chance, we’ll get a feature like this in the near future?

#8

You can use Material Functions to implement shared chunks of material features (incl, shader code inlined there): Material Functions | Flax Documentation
Materials layering might also help: Layered Materials | Flax Documentation

Custom Material Domain (as described here) is complex feature but allows to do anything with shader drawing setup. Some of the features you’ve listed can be done already via Material Functions (they can include shader code from project source: Shaders | Flax Documentation). The only missing bit from your list is Vertex-Lighting. Others can be done via current engine toolset.

For example:

  • fullscreen pixelation can be done like this: Custom Fullscreen Shader | Flax Documentation
  • Vertex-Snapping - use use World Position Offset input in material to control vertex placement
  • Vertex Colors (from mesh) - already there, sue Vertex Color node
  • Affine Texture Mapping - I suppose just use shader code to sample custom texture UVs
  • …
1 Like
#9

For Vetrex Lighting you could use VS to PS material node that passes data calculated in Vertex Shader to Pixel Shader.

Also, I wrote some basic docs about using shader sources in materials: FlaxDocs/manual/graphics/materials/shader-code-in-material.md at master · FlaxEngine/FlaxDocs · GitHub

1 Like
#10

Thanks again. :slight_smile:

Also, I wrote some basic docs about using shader sources in materials: FlaxDocs/manual/graphics/materials/shader-code-in-material.md at master · FlaxEngine/FlaxDocs · GitHub

This was also mentioned in the Discord and it is what I’m working with currently.

The fullscreen shader was not much of a problem.

I’m still struggling with Affine Texture Mapping. Of course, it’s possible to reconstruct the not perspective corrected UVs but I think that approach is a bit overhead.

Common.hlsl contains a struct containing a variable with the noperspective keyword. I’ve tried to implement a vertex stage via the global code node and save the noperspective UVs to a variable but this doesn’t seem to work since gobal vars are implicitly read only.

struct Quad_VS2PS
{
    float4 Position : SV_Position;
    noperspective float2 TexCoord : TEXCOORD0;
};

The VS to PS material node sounds promising, but as far as I see there’s no documentation for every single node? So, I’m not sure how to use it properly. I guess I also could use the node to manipulate vertex colors? That would be nice.

In the Unity version of my shaders, I sample the lights and shadows and use them to manipulate vertex color. So, sampling the light/shadow colors in the vertex stage would be also something I want to achieve. :slight_smile:

I’d still prefer a way to write entirely custom shaders as HLSL files. Maybe something like a “Custom Shader” Material Template which just takes a HLSL an then generates respective inputs would be a solution?

#11

I’ve got affine textures to work.

For those who find this and are interested in the solution, this is how it works:

image
image1396×426 81.9 KB

The custom node contains:

Output0 = mul(float4(Input0.xyz, 1), ViewProjectionMatrix);

You also need a Custom Global Code Node to import the hlsl file which contains ViewProjectionMatrix:

#include "./Flax/MaterialCommon.hlsl"

As a little addition: When the texture warping gets to strong, you can lerp between the affine UVs aand the normal UVs to attenuate the effect.
image

1 Like
#12

@mafiesto4
Just a thought: Is it possible to implement new Material Templates in a plugin?

#13

It’s not possible right now but we could do it at some point. This would require exposing some internals from rendering backend and material editor/generator.