First up, I used some open source art resources from the Torque3D Pacific Demo, which can now be found free for download with all art assets here http://wiki.torque3d.org/main:downloads. I amalgamated various terrain textures with their images marked _detail for overlays, such as grass, dry ground and volcanic rock to create a starting texture. I used the detail textures as specMaps and the original normalMaps, before testing the various slots of detailMap and detailNormalMap in the matieral.
Adding a detail map with a scale of 1 made the highlights and shadows of the diffuseMap stand out more. Hardly surprising as the detailMap is taken from the diffuse. Increasing the strength of the detailNM increased the grain of the image. Scaling the detailMap across the diffuse caused a more even grain without sharpening specific features. I thought that textures tended to look better close up the more sharp they were, and the opposite further away.
Next up I tested layering textures. The idea was to create new materials without having to manually merge two images in an image editing app, thus saving on texture memory in the long run. I took a grass texture for the base layer0, and overlayed it with the volcanic rock texture in layer1, to which I had added an alpha channel. The alpha channel was quite simple, I simply boosted contrast on the reference image and took highlights, mid, and shadows.
After intial testing I thought that the grass was somewhat washed out so I boosted the reduced the mid levels of the alpha channel so that it favoured the grass over 50-50 mixing. Originally I had based the alpha channel directly from the contrast of the texture, but found it better to boost the hightlights and then expand them for white, then expand a smaller area for mixed and have the remaining as fully translucent black. This let much more of both layers appear with only limited fade, making the whole thing sharper.
For the final test I added a mask onto the grass texture to simulate earth or dirt (though this could have done this with another layer between grass and rock to again reduce image quantity and thus texture memory, but these are only tests for visual effect). I hoped this would give a more naturalistic effect rather than just having rock sticking out of grass.
Rock and Alpha Channel (at 25% scale) Very 1-0 with only a little semi-transparent
I also noticed in previous tests that whilst detailMaps, detailNMs and specMaps worked on higher layers, standard normalMaps only used the base layer0. To give the rock layer some extra height, I used the volcanic Nm and Spec for layer0 with the grass texture. I had used the rocks alpha channel to paste dirt over the grass and now reduced the size of the white and grey levels so that more of the dirt would show around the edges. I proceeded to do various tests to see it looked, changing detailMaps, detailNMs, and even tweaking layer1 diffuse.
Eventually my eyes went all googly from looking at tiny changes.