Thursday, November 14, 2013 Eric Richards

Adding Shadow-mapping and SSAO to the Terrain

Terrain DirectX 11 Screen-space Ambient Occlusion SlimDX C# SSAO Procedural Terrain Shadows Shadow Maps Ambient Occlusion

Now that I’m finished up with everything that I wanted to cover from Frank Luna’s Introduction to 3D Game Programming with Direct3D 11.0 , I want to spend some time improving the Terrain class that we introduced earlier. My ultimate goal is to create a two tiered strategy game, with a turn-based strategic level and either a turn-based or real-time tactical level. My favorite games have always been these kinds of strategic/tactical hybrids, such as (in roughly chronological order) Centurion: Defender of Rome, Lords of the Realm, Close Combat and the Total War series. In all of these games, the tactical combat is one of the main highlights of gameplay, and so the terrain that that combat occurs upon is very important, both aesthetically and for gameplay.

Or first step will be to incorporate some of the graphical improvements that we have recently implemented into our terrain rendering. We will be adding shadow-mapping and SSAO support to the terrain in this installment. In the screenshots below, we have our light source (the sun) low on the horizon behind the mountain range. The first shot shows our current Terrain rendering result, with no shadows or ambient occlusion. In the second, shadows have been added, which in addition to just showing shadows, has dulled down a lot of the odd-looking highlights in the first shot. The final shot shows both shadow-mapping and ambient occlusion applied to the terrain. The ambient occlusion adds a little more detail to the scene; regardless of it’s accuracy, I kind of like the effect, just to noise up the textures applied to the terrain, although I may tweak it a bit to lighten the darker spots up a bit.

We are going to need to add another set of effect techniques to our shader effect, to support shadow mapping, as well as a technique to draw to the shadow map, and another technique to draw the normal/depth map for SSAO. For the latter two techniques, we will need to implement a new hull shader, since I would like to have the shadow maps and normal-depth maps match the fully-tessellated geometry; using the normal hull shader that dynamically tessellates may result in shadows that change shape as you move around the map. For the normal/depth technique, we will also need to implement a new pixel shader. Our domain shader is also going to need to be updated, so that it create the texture coordinates for sampling both the shadow map and the ssao map, and our pixel shader will need to be updated to do the shadow and ambient occlusion calculations.

This sounds like a lot of work, but really, it is mostly a matter of adapting what we have already done. As always, you can download my full code for this example from GitHub at https://github.com/ericrrichards/dx11.git. This example doesn’t really have a stand-alone project, as it came about as I was on my way to implementing a minimap, and thus these techniques are showcased as part of the Minimap project.

Basic Terrain Rendering

Shadowmapping Added

Shadowmapping and SSAO

Adding Shadow Mapping to the Terrain

To support shadow mapping, our first step is to create a technique as part of our Terrain.fx shader effect to render the terrain depth to the shadow map. Fortunately, this is relatively easy, and we can reuse the vertex shader and domain shader that we already have implemented for rendering the terrain normally. We could potentially use the same hull shader as well, but I opted to write a new hull shader which tessellates the terrain fully when rendering the shadow map. I tried using the original hull shader initially, but found that I preferred the fully-tessellated look; because of the way we set up the light viewpoint when rendering the shadow map, the terrain would tend to be at very low levels of tessellation, so the shadows were simpler and didn’t match up exactly with the rendered terrain. Since we only care about rendering depth to the depth/stencil buffer, we can use a null pixel shader as well. The Depth RasterizerState that we use here was copied from our previous shadow-mapping example.

RasterizerState Depth
{
    DepthBias = 100000;
    DepthBiasClamp = 0.0f;
    SlopeScaledDepthBias = 1.0f;
};

technique11 TessBuildShadowMapTech
{
    pass P0
    {
        SetVertexShader(CompileShader(vs_4_0, VS()));
        SetHullShader(CompileShader(hs_5_0, NormDepthHS()));
        SetDomainShader(CompileShader(ds_5_0, DS()));
        SetGeometryShader(NULL);
        SetPixelShader(NULL);

        SetRasterizerState(Depth);
    }
}

Our new hull shader is the same as the previous hull shader, except that we have replaced the patch constant function with a new version which skips the distance-based tessellation factor calculation and simply applies the maximum amount of tessellation to the vertex patch.

PatchTess NormDepthConstantHS(InputPatch<VertexOut, 4> patch, uint patchID : SV_PrimitiveID)
{
    PatchTess pt;

    //
    // Frustum cull
    //

    // We store the patch BoundsY in the first control point.
    float minY = patch[0].BoundsY.x;
    float maxY = patch[0].BoundsY.y;

    // Build axis-aligned bounding box.  patch[2] is lower-left corner
    // and patch[1] is upper-right corner.
    float3 vMin = float3(patch[2].PosW.x, minY, patch[2].PosW.z);
        float3 vMax = float3(patch[1].PosW.x, maxY, patch[1].PosW.z);

        float3 boxCenter = 0.5f*(vMin + vMax);
        float3 boxExtents = 0.5f*(vMax - vMin);
    if (AabbOutsideFrustumTest(boxCenter, boxExtents, gWorldFrustumPlanes))
    {
        // do frustum culling…
    }
    else
    {
        // use maximum tessellation
        pt.EdgeTess[0] = gMaxTess;
        pt.EdgeTess[1] = gMaxTess;
        pt.EdgeTess[2] = gMaxTess;
        pt.EdgeTess[3] = gMaxTess;

        pt.InsideTess[0] = gMaxTess;
        pt.InsideTess[1] = gMaxTess;

        return pt;
    }
}

DrawToShadowMap()

With the shader effect technique in place, we now need a method to render the terrain to the shadow map. Ultimately, this method looks very much like our normal Draw() call, save that we can avoid setting some shader variables and textures that are only used for rendering. This method also handles binding the shadow map, provided that we pass in a valid one, so the only pre-processing that we need to perform in our application code is to calculate the view/projection matrix for the light source casting the shadow. All of the information that we would normally need to extract from the camera (camera position, frustum planes) can be retrieved from this matrix, with a little bit of work. Then we just need to select the new shader technique that we have just written (via the EffectTechnique handle that you’ll need to add and initialize in the terrain effect wrapper class), and draw the terrain geometry. Voila, easy as that we have drawn out the shadow map for our terrain!

// Core/Terrain/Terrain.cs
public void DrawToShadowMap(DeviceContext dc, ShadowMap sMap, Matrix viewProj) {
    sMap.BindDsvAndSetNullRenderTarget(dc);

    dc.InputAssembler.PrimitiveTopology = PrimitiveTopology.PatchListWith4ControlPoints;
    dc.InputAssembler.InputLayout = InputLayouts.TerrainCP;

    var stride = TerrainCP.Stride;
    const int offset = 0;

    dc.InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(_quadPatchVB, stride, offset));
    dc.InputAssembler.SetIndexBuffer(_quadPatchIB, Format.R16_UInt, 0);


    var frustum = Frustum.FromViewProj(viewProj);
    var planes = frustum.Planes;

    Effects.TerrainFX.SetViewProj(viewProj);
    Effects.TerrainFX.SetEyePosW(new Vector3(viewProj.M41, viewProj.M42, viewProj.M43));
    Effects.TerrainFX.SetMinDist(MinDist);
    Effects.TerrainFX.SetMaxDist(MaxDist);
    Effects.TerrainFX.SetMinTess(MinTess);
    Effects.TerrainFX.SetMaxTess(MaxTess);
    Effects.TerrainFX.SetWorldCellSpace(Info.CellSpacing);
    Effects.TerrainFX.SetWorldFrustumPlanes(planes);
    Effects.TerrainFX.SetHeightMap(_heightMapSRV);
            
    var tech = Effects.TerrainFX.TessBuildShadowMapTech;
    for (int p = 0; p < tech.Description.PassCount; p++) {
        var pass = tech.GetPassByIndex(p);
        pass.Apply(dc);
        dc.DrawIndexed(_numPatchQuadFaces * 4, 0, 0);
    }
    dc.HullShader.Set(null);
    dc.DomainShader.Set(null);
}

Adding Shadows to our Rendering Effect

Initially, I just converted the existing rendering techniques to all use shadowing, and just bound a white texture as the shadow map when I wanted to disable shadows, but I was not really happy with the result, so I implemented an entire extra set of techniques explicitly for enabling shadow-mapping. Shadow mapping is thus enabled or disabled by passing an additional uniform boolean to the pixel shader when defining the technique. All of these new techniques follow the same pattern, so I’ll only display one here:

technique11 Light1Shadow
{
    pass P0
    {
        SetVertexShader(CompileShader(vs_4_0, VS()));
        SetHullShader(CompileShader(hs_5_0, HS()));
        SetDomainShader(CompileShader(ds_5_0, DS()));
        SetGeometryShader(NULL);
        SetPixelShader(CompileShader(ps_4_0, PS(1, false, true))); // last parameter controls shadowing
    }
}

For shadowmapping, we need to add two additional global shader variables. We will need to add the shadow transformation matrix, in order to calculate the projective texture coordinates to sample the shadow map, and the shadow map texture itself. We also need to add the texture sampler that we will use for sampling the shadow map. All of these are exactly the same as we implemented to do shadow mapping before.

SamplerComparisonState samShadow
{
    Filter = COMPARISON_MIN_MAG_LINEAR_MIP_POINT;
    AddressU = BORDER;
    AddressV = BORDER;
    AddressW = BORDER;
    BorderColor = float4(0.0f, 0.0f, 0.0f, 0.0f);

    ComparisonFunc = LESS;
};

Texture2D gShadowMap;

cbuffer cbPerObject
{
    // other per-object shader variables...
    float4x4 gShadowTransform;
};

We will also need to modify our domain shader and the domain shader output structure to calculate and store the shadow map coordinates for each vertex generated by the tessellation stage.

struct DomainOut
{
    float4 PosH     : SV_POSITION;
    float3 PosW     : POSITION;
    float2 Tex      : TEXCOORD0;
    float2 TiledTex : TEXCOORD1;
    // new for shadowmapping
    float4 ShadowPosH : TEXCOORD3;
};

[domain("quad")]
DomainOut DS(PatchTess patchTess,
    float2 uv : SV_DomainLocation,
    const OutputPatch<HullOut, 4> quad)
{
    // normal domain shader code...

    dout.ShadowPosH = mul(float4(dout.PosW, 1.0f), gShadowTransform);
    return dout;
}

Finally, our pixel shader will need to be modified to accept the additional uniform boolean which controls shadowing, and to sample the shadow map and modify the lighting calculations accordingly.

float4 PS(DomainOut pin,
    uniform int gLightCount,
    uniform bool gFogEnabled, uniform bool gDoShadow) : SV_Target
{
    // normal pixel shader code
    float3 shadow = float3(1.0f, 1.0f, 1.0f);
    if (gDoShadow){
        shadow[0] = CalcShadowFactor(samShadow, gShadowMap, pin.ShadowPosH);
    }

    float4 litColor = texColor;
    if (gLightCount > 0)
    {
        // Start with a sum of zero. 
        float4 ambient = float4(0.0f, 0.0f, 0.0f, 0.0f);
        float4 diffuse = float4(0.0f, 0.0f, 0.0f, 0.0f);
        float4 spec = float4(0.0f, 0.0f, 0.0f, 0.0f);

        // Sum the light contribution from each light source.  
        [unroll]
        for (int i = 0; i < gLightCount; ++i)
        {
            float4 A, D, S;
            ComputeDirectionalLight(gMaterial, gDirLights[i], normalW, toEye,
                A, D, S);
            
            ambient += A;
            // modified for shadowmapping
            diffuse += shadow[i] * D;
            spec += shadow[i] * S;
        }

        litColor = texColor*(ambient + diffuse) + spec;
    }
    // normal pixel shader code
}

At this point, you should be able to run the example and see shadows on the terrain, like in the screenshot below.

Adding SSAO to the Terrain

To add ambient occlusion to our terrain, the first step is to create a shader technique that can render the normal/depth texture that the SSAO effect uses as input. Once again, we are going to use our new hull shader, so that the ambient occlusion is computed for the fully tessellated terrain mesh. We will need to create a new pixel shader to output the normal and depth information, as we did when implementing SSAO before.

technique11 NormalDepth {
    pass P0{
        SetVertexShader(CompileShader(vs_4_0, VS()));
        SetHullShader(CompileShader(hs_5_0, NormDepthHS()));
        SetDomainShader(CompileShader(ds_5_0, DS()));
        SetGeometryShader(NULL);
        SetPixelShader(CompileShader(ps_4_0, NormDepthPS()));
    }
}

float4 NormDepthPS(DomainOut pin) : SV_Target{
    //
    // Estimate normal and tangent using central differences.
    //
    float2 leftTex = pin.Tex + float2(-gTexelCellSpaceU, 0.0f);
    float2 rightTex = pin.Tex + float2(gTexelCellSpaceU, 0.0f);
    float2 bottomTex = pin.Tex + float2(0.0f, gTexelCellSpaceV);
    float2 topTex = pin.Tex + float2(0.0f, -gTexelCellSpaceV);

    float leftY = gHeightMap.SampleLevel(samHeightmap, leftTex, 0).r;
    float rightY = gHeightMap.SampleLevel(samHeightmap, rightTex, 0).r;
    float bottomY = gHeightMap.SampleLevel(samHeightmap, bottomTex, 0).r;
    float topY = gHeightMap.SampleLevel(samHeightmap, topTex, 0).r;

    float3 tangent = normalize(float3(2.0f*gWorldCellSpace, rightY - leftY, 0.0f));
    float3 bitan = normalize(float3(0.0f, bottomY - topY, -2.0f*gWorldCellSpace));
    float3 normalW = cross(tangent, bitan);

    float3 posV = mul(float4(pin.PosW, 1.0f), gView).xyz;
    float3 normalV = mul(normalW, (float3x3)gView);

    float4 c = float4(normalV, posV.z);

    return c;
}

This new pixel shader is essentially just the first stage of our normal terrain rendering pixel shader. We estimate the normal at the pixel, then transform it into light-space using the light view matrix (gView). Then we output the normal in the RGB channels of the output color, and the light-space depth in the alpha channel.

With the normal/depth technique created, our next step is to add a method to our Terrain class to use this technique to render the normal/depth map for SSAO and to compute the occlusion map. This method looks a lot like the method that we added to compute the shadow map texture; we start by binding the SSAO normal/depth target to our DeviceContext, and then draw the terrain geometry, ignoring all of the material and texture properties of the terrain. Finally, we compute the occlusion map using the newly created normal/depth map, and perform our blurring operations to smooth out the occlusion map.

public void ComputeSsao(DeviceContext dc, CameraBase cam, Ssao ssao, DepthStencilView depthStencilView) {
    ssao.SetNormalDepthRenderTarget(depthStencilView);

    dc.InputAssembler.PrimitiveTopology = PrimitiveTopology.PatchListWith4ControlPoints;
    dc.InputAssembler.InputLayout = InputLayouts.TerrainCP;

    var stride = TerrainCP.Stride;
    const int offset = 0;

    dc.InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(_quadPatchVB, stride, offset));
    dc.InputAssembler.SetIndexBuffer(_quadPatchIB, Format.R16_UInt, 0);

    var viewProj = cam.ViewProj;
    var planes = cam.FrustumPlanes;            

    Effects.TerrainFX.SetViewProj(viewProj);
    Effects.TerrainFX.SetEyePosW(cam.Position);
    Effects.TerrainFX.SetMinDist(MinDist);
    Effects.TerrainFX.SetMaxDist(MaxDist);
    Effects.TerrainFX.SetMinTess(MinTess);
    Effects.TerrainFX.SetMaxTess(MaxTess);
    Effects.TerrainFX.SetTexelCellSpaceU(1.0f / Info.HeightMapWidth);
    Effects.TerrainFX.SetTexelCellSpaceV(1.0f / Info.HeightMapHeight);
    Effects.TerrainFX.SetWorldCellSpace(Info.CellSpacing);
    Effects.TerrainFX.SetWorldFrustumPlanes(planes);
    Effects.TerrainFX.SetHeightMap(_heightMapSRV);
    Effects.TerrainFX.SetView(cam.View);

    var tech = Effects.TerrainFX.NormalDepthTech;
    for (int p = 0; p < tech.Description.PassCount; p++) {
        var pass = tech.GetPassByIndex(p);
        pass.Apply(dc);
        dc.DrawIndexed(_numPatchQuadFaces * 4, 0, 0);
    }
    dc.HullShader.Set(null);
    dc.DomainShader.Set(null);

    ssao.ComputeSsao(cam);
    ssao.BlurAmbientMap(4);
}

Adding the Ambient Occlusion Calculations to our Rendering Techniques

We will need to add a texture reference for the occlusion map to our Terrain.fx rendering shader, as well as the transformation matrix which converts world-space positions into texture coordinates to sample the occlusion map.

cbuffer cbPerObject
{
    // Other per-object variables...

    // new for ssao
    float4x4 gViewProjTex;
};

Texture2D gSsaoMap;

We will be computing these occlusion map lookup coordinates in our domain shader, so we will once again need to modify our DomainOut vertex structure.

struct DomainOut
{
    // other DomainOut members
    // new for ssao
    float4 SsaoPosH   : TEXCOORD2;
};

The calculation to generate these texture coordinates is fairly simple; we just need to multiply the computed vertex world position by the view-projection-toTexture matrix we added.

dout.SsaoPosH = mul(float4(dout.PosW, 1.0f), gViewProjTex);

Finally, we need to modify our lighting calculations in our pixel shader to sample the occlusion map and scale the calculated ambient color value.

    // new for ssao
    pin.SsaoPosH /= pin.SsaoPosH.w;
    float ambientAccess = gSsaoMap.SampleLevel(samLinear, pin.SsaoPosH.xy, 0.0f).r;

    float4 litColor = texColor;
    if (gLightCount > 0)
    {
        // Start with a sum of zero. 
        float4 ambient = float4(0.0f, 0.0f, 0.0f, 0.0f);
        float4 diffuse = float4(0.0f, 0.0f, 0.0f, 0.0f);
        float4 spec = float4(0.0f, 0.0f, 0.0f, 0.0f);

        // Sum the light contribution from each light source.  
        [unroll]
        for (int i = 0; i < gLightCount; ++i)
        {
            float4 A, D, S;
            ComputeDirectionalLight(gMaterial, gDirLights[i], normalW, toEye,
                A, D, S);
            // modified for ssao
            ambient += ambientAccess*A;
            // modified for shadowmapping
            diffuse += shadow[i] * D;
            spec += shadow[i] * S;
        }

        litColor = texColor*(ambient + diffuse) + spec;
    }

For completeness, I will include our final Terrain.Draw() method here. One thing to note is that you will need to set the occlusion map and shadow map shader resources within your application code, since the Terrain class does not have access to them, as well as the shadow transform, since that is dependent upon the entire scene. You’ll also notice that I’ve added a flag, Shadows, to control whether we use a shadow-mapping or non-shadow mapping rendering technique. This is just to make it easier to display the difference between enabling shadows and not, since we cannot simply bind a null texture as the shadow map when we want to disable shadowing; a null texture is treated as black, and so everything would appear in shadow, which is the opposite of what we intend.

public void Draw(DeviceContext dc, CameraBase cam, DirectionalLight[] lights) {
    if (_useTessellation) {

        dc.InputAssembler.PrimitiveTopology = PrimitiveTopology.PatchListWith4ControlPoints;
        dc.InputAssembler.InputLayout = InputLayouts.TerrainCP;

        var stride = TerrainCP.Stride;
        const int offset = 0;

        dc.InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(_quadPatchVB, stride, offset));
        dc.InputAssembler.SetIndexBuffer(_quadPatchIB, Format.R16_UInt, 0);

        var viewProj = cam.ViewProj;
        var planes = cam.FrustumPlanes;
        var toTexSpace = Matrix.Scaling(0.5f, -0.5f, 1.0f) * Matrix.Translation(0.5f, 0.5f, 0);

        Effects.TerrainFX.SetViewProj(viewProj);
        Effects.TerrainFX.SetEyePosW(cam.Position);
        Effects.TerrainFX.SetDirLights(lights);
        Effects.TerrainFX.SetFogColor(Color.Silver);
        Effects.TerrainFX.SetFogStart(15.0f);
        Effects.TerrainFX.SetFogRange(175.0f);
        Effects.TerrainFX.SetMinDist(MinDist);
        Effects.TerrainFX.SetMaxDist(MaxDist);
        Effects.TerrainFX.SetMinTess(MinTess);
        Effects.TerrainFX.SetMaxTess(MaxTess);
        Effects.TerrainFX.SetTexelCellSpaceU(1.0f / Info.HeightMapWidth);
        Effects.TerrainFX.SetTexelCellSpaceV(1.0f / Info.HeightMapHeight);
        Effects.TerrainFX.SetWorldCellSpace(Info.CellSpacing);
        Effects.TerrainFX.SetWorldFrustumPlanes(planes);
        Effects.TerrainFX.SetLayerMapArray(_layerMapArraySRV);
        Effects.TerrainFX.SetBlendMap(_blendMapSRV);
        Effects.TerrainFX.SetHeightMap(_heightMapSRV);
        Effects.TerrainFX.SetMaterial(_material);
        Effects.TerrainFX.SetViewProjTex(viewProj * toTexSpace);
                
        var tech = Shadows ? Effects.TerrainFX.Light1ShadowTech: Effects.TerrainFX.Light1Tech;
        for (int p = 0; p < tech.Description.PassCount; p++) {
            var pass = tech.GetPassByIndex(p);
            pass.Apply(dc);
            dc.DrawIndexed(_numPatchQuadFaces * 4, 0, 0);
        }
        dc.HullShader.Set(null);
        dc.DomainShader.Set(null);

    } else { // dx10 code branch}
}

Next Time…

You may have noticed that all of these screenshots have a quad missing on the bottom of the window… This is where my terrain minimap should be displaying, except that I commented it out for this post. Next time around, we’ll look at creating a minimap, by using an orthographic camera to show our entire terrain from an overhead view, then rendering to a texture. We’ll also figure out how to show our view camera’s frustum superimposed on the minimap, which is very handy for figuring out where in the world you are. If I have extra time to code it up between now and then, we’ll also look at moving the camera using the minimap.

Thanks for reading!