External Alpha masks
Above are the two links from the UDN on the topic. I've implemented the above using some external tools and quite a bit of experimentation so I want to outline the approach along with some of the pluses and minuses that it represents.
The approach is to create a heightmap using an external tool at a high resolution, along with all of the accompanying data. This includes high fidelity normal maps as well as texture maps that can exceed the normal amount of data a smaller map can contain.
As my example I'm using a 512*512 Set of data and applying it to a 64*64 terrain. This technique allows you to increase both the texture fidelity and the normal map fidelity by 4096(64*64) times.
Please keep in mind that my programmer art ranks poorly even among programmer art and I've been heavy handed with the artwork itself to highlight the detail. A proper artist using this set up will have better looking results with just a little bit of effort.
Creating the heightmap
I've used two different tools for creating the heightmap.
The first is L3DT for creating the shape and polygon level editing to make the map playable.
For additional detail and for their great visual effects editor I used World Machine 2
The reason I used both is that World Builder either lacks the ability to do polygon level editing or else I lack the knowledge to make it do so. What it does have is a great editor that implements an interfacer similar to kismet and the material editor. This makes it a breeze to capture all of the data you need for normal maps and supermasks, as well as some great procedural effects you can add to your terrain right out of the box.
The first gotcha is to make sure that your heightmap is a power of 2 +1, so in my case I created a 513*513 heightmap.
The heightmap I created using both tools is here.
Creating the normal map:
I'm not going to go in to great detail on setting up World Machine to create the maps you need but here is the general idea.
For the normal map you simply connect the normal map maker node to the output of your heightfield and then connect a bitmap output to the normal map maker.
Here's the normal map
That step yielded the normal map above. For the erosion detail you add an erosion node and grab the heightfield it creates with another normal map maker and then combine them with a combiner node. (Even as over-detailed as this erosion map I'm using is I culled out 85% of the detail with a selector node). As with all output you capture it in a bitmap output node.
This is what my normal map looks like afterward, note that I inverted the normal map because I wanted the erosion channels to be depressions instead of bumps.
Now we're adding some great detail. For a production quality map we'd definitely want to tone this down but for the purposes of showing off the technique I kept all of it.
Creating the supermask:
Creating the supermask in World Machine is essentially the same technique. You need to set up selector nodes to grab the parts of the terrain you want each texture to be on and then push them into a channel combiner. You then capture the output in a bitmap output node.
Here's what the packed supermask for my map looks like.
Each channel of the map is really just a heightmap that we'll later tell the engine to interpret as an alpha mask. Because this type of map is often called an alpha mask or alpha texture, it's important not to confuse the terminology. This texture is a 24 bit bitmap, an R8G8B8 texture, which means 8 bits per channel, with 3 channels 1 red, 1 blue, 1 green.
So despite all of the color, it's helpful to think of it as 3 grayscale textures all packed into one texture. Where white is 100% and black is 0%, with up to 8 bits(0 to 255) of grays in between. Later we'll set up the material so the engine knows which texture to draw at which color and how much to blend for each pixel.
For now all you need to know is that when you import the texture into Unreal you import it as a "CompressionNoAlpha"
The reason you may want to consider supermasks, especially for smaller maps is discussed in the UDN.
For my 64*64 vertex example choosing a single vertex to paint means the smallest detail, like the footpath example, has to be 14 meters(almost 46 feet) wide. This technique allows a detail level that brings the same footpath down to as small as .28175 meters( .717 feet)The current terrain layer painting system is limited to the resolution of the terrain mesh vertices. The number of terrain mesh vertices is the same as the underlying terrain heightmap, so each layer weightmap is therefore the same resolution as the heightmap. In other words, a 256x256 patch terrain which contains an underlying 257x257 heightmap allows for a layer painting resolution of 257x257 discreet blending pixels. While this is fine for the majority of terrain layer painting detail, there are instances where a finer degree of texturing may be desired. Note that a weightmap is often also referred to as an alphamap. It provides a similar functionality within the terrain layer blending system as an alpha channel in a 32-bit image or a paint application such as PhotoShop.
To put this into perspective and scale, many common two lane highways are constructed of two 3.5 meter lanes plus additional shoulder width. So in comparison, a single painted vertex on a 256 DrawScale terrain is approximately the same width as a two lane highway. This is often too wide for terrain layer detailing such as foot-paths, ditches, creeks, etc. Choosing to solve this layer weightmap size issue by decreasing the terrain DrawScale3D may result in a terrain that is too dense and impacts the framerate due to the greater number of patches being rendered in the frustum.
I didn't really exploit the texture detail as fully as it could be done but if you've tried painting detail on a 64*64 map you should be able to appreciate even my amateur texture maps using this technique.
Getting it to render:
Time for your next tool:
This is a great piece of software. It does just a couple of things and it does them well. It will let you downsample your large height map you created all of your detail on to whatever size you plan on making your terrain. Keep in mind that Unreal expects a power of 2+1. It also will convert your 8 bit grayscale heightmap to the G16 format that Unreal requires for building terrain.
In my case I went from 513*513 to 65*65.
It also allows you to constrain and adjust your altitudes. I found that using the toolchain I've outlined my altitudes were between 0 and 65000+ meters, which means your map is going to look like a donut unless you make some adjustments. The map in this example varies from 0 to 300 meters. You can make these adjustments at any point in the pipeline but I've found HMCS to be the easiest route.
Importing your textures:
Now we need to get all of our textures into the editor. There are a couple of things to watch out for here. If you're using the same toolchain I've outlined the first thing to know is that World Maker builds it terrain starting at a different corner than Unreal. This means you need to flip vertically either your heightmap or else the normal and super mask.
This can be done trivially in any pixel-pusher application. I used Gimp for mine.
In this example every texture I'm importing except for the heightmap is a 24bit bitmap. The normal map, super mask, and all of the terrain paint should be imported using the "CompressionNoAlpha" option enabled. If you want to use a 32 bit bitmap because you're packing 4 planes instead of 3 then you probably know how you want to import it yourself.
The second thing to look out for is the editor expects your normal map and super mask to be in a power of two while your toolchain has generated a power of 2+1 texture. Use your same pixel program to scale your maps down to a power of 2. Remember that your heightmap must be a G16 format and a power of 2+1.
At this point you may as well import the textures that are going to be painted on your terrain. In this example I used 256*256 textures. Upping the texture size to at least 512*512 for production quality maps may be a good idea.
I have 3 textures I'm painting for convenience sake. You could just as easily use 5 or 6 textures either packed or unpacked in a super mask. My textures are the grass, the mountainside, and the red dirt, which is supposed to look like a rust-colored sediment to detail the erosion layer.
Here's the material set up for the supermask shader.
The supermask link at the beginning of the tutorial presents the material set up and is easier to read so I'd suggest reading that article before you hurt yourself squinting at my screenshot.
You'll notice some connectors running off the screen below. I'm using the supermasks to blend some texture normals into the terrain normals. I'm not satisfied with the results though so for the purposes of this tutorial we're just going to plug our normal map above straight into the normals input.
Setting up the editor:
We need to change a few default settings in the editor. The first thing you should do is create a dominant directional light and change its settings. For this example I upped the brightness to 2.0 to show off the technique and added a tiny bit of gold color. The most important setting to change is the Shadow Projection Technique.
I'm using ShadowProjTech_PCF, for Percentage Closer Filtering with a low Quality Shadow Filter.
The next setting you need to change is under the Terrain sub-category of the terrain properties. It's called Normal Map Layer and has a default setting of -1. This is a flag that tells the terrain where you're feeding it the normal map for the terrain. -1 means to use the internally generated normal map. this setting is 0 for me, which is the index for the material where I'm feeding the engine the normal map. If you set your terrain up like I have you'll only have one material so the index will always be 0.
To be developed:
The shortcomings of this technique thus far seems to be the proper way to add texture normals to the pipeline. I'm currently using the supermask to paint texture normals that match whichever texture is being drawn at that specific point. For noisy textures this method adds some detail but if you have a clean uniform texture like a brick road I'm not certain how you'd line the texture normal maps up to match the diffuse texture.
Further, because the normal map you're blending with is for the entire terrain, you need a great deal of detail in the texture normal map in order for any of it to translate to the final normal map.
If anyone extends this technique and solves the problem I would appreciate you sharing it.
I went into a lot of detail and outlined some of the gotchas that cost me a few hours to figure out. The technique is pretty simple to implement and adds a great deal of detail to the terrain at a very low cost. The material set up uses 7 texture samples, although you would likely want to add a detail texture to the mountainside, so let's call it 8.
31 instructions in the vertex shader and 41 instructions in the LightEnv shader. The shader complexity map is a nice darkish green with no trouble spots.
For a real feel for the issue this technique solves create a quick 64*64 terrain and try to duplicate painting the sediment texture on a mountainside. For that matter try to duplicate the blend between the grass and the mountains. To achieve the same blend you're making a 512*512 map and sacrificing enough frames per second to run all of your game AI, or some really nice physics.
All of the screenshots use the editor settings I outlined above. Resolution is 1280*1024. Lighting quality is in preview mode.
If you look closely you can see the rivulets in the mountainside.
Here's a shot with the sun at a 45 degree angle to maximize the normal detail. I had to do a regular screen cap to get the right perspective.
I added some additional screenshots below which include my texture normal implementation.