EDIT: I've decided to expand my question to include other possible methods of muscle deformation. I've heard from somewhere that CAT Muscles can be used with UDK characters, but my attempts to do this myself don't work 100% (possibly due to "non-orthogonal" distortion). The deformation happens, but it seems to be misplaced (almost like the muscle is being moved downwards during export).
I'm attempting to rig a character for UDK, and i'm having a lot of trouble finding any information on how to get convincing muscle deformation.
My character uses a custom CATRig, and I need for him to have realistic muscle deformation in various parts of his body. I've already attempted to use 3DS Max's own muscle system, only to discover (after exporting it as an .fbx) that UDK does not support this (every segment of the body rigged to the muscles became incredibly glitchy and sunken in). After removing the muscles, I then attempted to use a Skin Morph as I have seen various people on the internet recommend this (without ever saying how to do it). Unfortunately this too doesn't work; the rig works fine, but the morphs have no effect on the model within UDK even if I bake the animation on export (.fbx). I even tried putting the Skin Morph modifier under the Skin modifier (as the .fbx exporter doesn't seem to like anything on top of the skin modifier), but that still didn't work.
So could someone please tell me the recommended way on including muscle deformation with my character? The only reference I can find to such a thing in the official UDK documentation is a brief mention that morphs could possibly be used for muscle deformation amongst other things.
I'm really thankful that you're trying to guide me, but that still doesn't answer my question. How would one go about "implementing it either in code or anim tree"? Does the anim tree contain nodes that allow you to dial in a morph depending on the angle of the bone? How does that work? Again, it seems that although basic character rigging for UDK is well documented, anything beyond simple deformations has been left unexplained or incredibly vague.
I've decided to expand my question to include other possible methods of muscle deformation. I've heard from somewhere that CAT Muscles can be used with UDK characters, but my attempts to do this myself don't work 100% (possibly due to "non-orthogonal" distortion). The deformation happens, but it seems to be misplaced (almost like the muscle is being moved downwards during export).
Well, I managed to find my own solution. Basically I ditched the muscle strands/CAT muscles and went back to a morph based solution. I already had an older save file where I attempted to use a Skin Morph modifier. What I did, was that I extracted the shapes made with the Skin Modifier, and reapplied them to the figure as regular old blendmorphs (in this case, the right buttox and the left buttox). I then used the "wire parameters" feature of 3ds max to link the z-rotation of the left thigh with the left buttock morph (by a value of * -145), and did the same for the right side. Export as fbx, reimport to UDK, works perfectly fine without needing to fiddle with any coding or animtree nodes. I find this workflow should benefit those who are used to working predominantly within 3ds max.