Edge Detection or "Borderlands" style shader in UDK

[PrawnSkunk]

Many people are interested in having a Cel Shading/Toon/Borderlands-style shader effect as part of their level in Unreal Development Kit. It's a cool effect that really brings your level to life by making ordinary static meshes "pop out."
Here is my abridged version of the very lengthy tutorial found in another documentation source.

Click here for the tutorial on my blog!



After Sobel Edge


Before Sobel Edge

[Alex D]

Edge detection is not Cel-Shading as it does not change how the objects in the scene are shaded, it is just edge detection. On the other hand, Cel-Shading has nothing to do with edge detection in the first place, many games which make use of the effect do not make use of hard-edge outlines such as these.They are entirely separate techniques which produce entirely separate effects, they just happen to be used together more often than not. This is a very simple distinction which, it seems, is consistently misunderstood.

As far as constructive criticism of the actual tutorial goes, it would be more beneficial if you actually detailed the techniques used in the tutorial rather than just telling readers to download a file from another tutorial and use it.

[PrawnSkunk]

Thank you for your post.

Yes, I know Cel-Shading =/= edge detection, but I included that phrase as part of the title and description so that people looking for a toon/borderlands/edge detection/"cel-shading" looking post process chain, they could find it easily and it would pick up those key search terms in Google or the UDK Forums.

As far as the depth of explanation in this tutorial goes, I decided not just regurgitate an already made tutorial and claim credit for it. This is quick, concise - and has all the essentials explained in a few (abridged) simple steps.

If someone truly wants to read the documentation of the full tutorial, it can be found here. (link added to OP)

I look forward to any other comments you, or anyone else may have!

[Mougli]

To be honest, and I don't mean to offend, but a tutorial about how to use a tutorial sounds like taking people for idiots, especially when it originates from the official documentation. It will be more interesting for you and the community to make tutorials about things that aren't documented.

[Xenobite]

speaking of Borderlands stlye, don't forget they have unique texture style
they had some drawn ink lines on them exagerating some details and shading (hatched lines) that look like someone was using a marker pen or something like that that also added to "comic book" look

[SkullKing]

Thank you for your post.

Yes, I know Cel-Shading =/= edge detection, but I included that phrase as part of the title and description so that people looking for a toon/borderlands/edge detection/"cel-shading" looking post process chain, they could find it easily and it would pick up those key search terms in Google or the UDK Forums.

Here is my constructive criticism: Change the title.

You didn´t just put cel-shading in the tittle, you made it the tittle. That´s is misleading at best. you plain said you had a tutorial about something and post a tutorial about something else completely different, Edge detecting isn´t even mentioned in the tittle or the post.

Now people looking for a cel-shading tutorial will end-up wasting their time and getting their hopes up about your useless tutorial, and people looking for edge detection would have a hard time finding your post if not for the comments correcting your error.

If you wanted people to find your tutorial when looking for cel-shading you could easily have named it [Tutorial] Edge detection(frequently used with cel-shading) or something along those lines.

Thread does not deliver. Disapointing

[PrawnSkunk]

Thank you for your input on my thread, though I'm saddened you found it "disappointing" and of no value to yourself.

I've changed the thread title as suggested.
Even though I meant "Cel-Shading-style or "Borderlands" style shader" It's good to change these things early before they get picked up by search engines.

[SkullKing]

Don´t take me wrong, It is a good thing that you are trying to help the community, and it may very well help someone.

It just so happens that I want to make cel-shading in my game, and got exited that a tutorial was made available, just for it not to be about cel-shading at all.

[Xenobite]

there is some good cell shading tutorial on moddb:

http://www.moddb.com/games/unreal-to...shading-in-ut3

[PrawnSkunk]

Thanks everyone for their input. This tutorial seems to have gained popularity, so in the future I will definitely look to create more tutorials for the community

[RoizzeEntertainmen]

Hello, I'm getting a strange bug? Glitch? I don't know what to call it. So, whether or not, editor and in-game, the shader applies to a specific location, unless I move, everything else that I see in front of me, is BLACK. Any fixes?

[SkullKing]

Post a screenshot of the glicth, before and after moving(or link to a video)

Did you test it with the demo maps,or just in your maps? Does it happens in any map?

[PrawnSkunk]

Hello, I'm getting a strange bug? Glitch? I don't know what to call it. So, whether or not, editor and in-game, the shader applies to a specific location, unless I move, everything else that I see in front of me, is BLACK. Any fixes?

I've had this problem, where the ground appears to be black. If this is the same reoccurring bug, the ominous black shadow should move with you as you move forward/back up.

Look at this image, in the distance. Is this your same problem? If so, I will try my best to find a fix. It might not be fixable, due to the shading calculations in this post-processing effect.

[RoizzeEntertainmen]

Same problem as screenshot. And I got 1 more problem, with the console commands. Any console command I put, like.. open DM-Deck, it says command not recognized !! It's driving me insane! Plz help if anyone knows. D:

[Omar007]

And I got 1 more problem, with the console commands. Any console command I put, like.. open DM-Deck, it says command not recognized !! It's driving me insane! Plz help if anyone knows. D:

Unrelated to this. Make another topic.

EDIT: I see you did, check it.

[Alex D]

I've had this problem, where the ground appears to be black. If this is the same reoccurring bug, the ominous black shadow should move with you as you move forward/back up.

That is unfortunately a limitation of this method of edge detection. As I'm sure you are aware, this method relies solely on comparing depth values and the depth is not stored in linear values. This means there is less of a difference in depth values taken close to the camera than there are in values taken further away from the camera. In simple terms, if the difference between the depth samples is too great then the post process material decides it is an edge. At a certain distance the difference in depth values between adjacent pixels on a floor or wall plane is greater than threshold of the edge detection.

The way to get around this under normal circumstances would be to compare surface normals to find surface discontinuities as well as using depth, which would not only add edge detection on normal mapped surfaces, but also correct for those errors. However, UDK does not allow access to the scene's rendered normals through post process materials. I've suggested they add support for sampling scene normals like this many times since UT3 came out but it's not been added.

One option around this could be to use one of the depth samples to adjust the threshold based on the distance from the camera, however this may not end up with too great results.

[Chosker]

However, UDK does not allow access to the scene's rendered normals through post process materials. I've suggested they add support for sampling scene normals like this many times since UT3 came out but it's not been added.

it doesn't "allow access" to the rendered normals because UT3/UDK doesn't really have a deferred rendering pipeline. such a normal buffer simply doesn't exist (and Epic doesn't seem to be interested in creating it and making it available in a node)
however, recently I found out you can reconstruct the scene normals in view-space using the material editor with the depth node. I could write a tutorial if enough people are interested.
it won't account for geometry normalmaps though, just for the geometry's actual normals

[SkullKing]

You can get rid of those black patches by increasing the "minDistance" parameter, however it will also alter how it applys the edge to things that are working fine. I suggest you click the UDN link in the OP and delve a little deeper in it.

[PrawnSkunk]

I found a solution to the black shadows created as a result of this shader:

http://forums.epicgames.com/threads/...ading-material

[MrVyktys]

Many thanks