Outline [NPR] Post-Process issues.

[Ikono]

Hmmm...
So, I'm attempting to use a Sobel filter to detect edges based on the SceneDepth in a post-process material.
Beginning from a SceneDepth sample, and then dividing that by 10000 to get a decent range I plug it into the Sobel filter, offsetting the depth sample and such.
Anyway, material stuff aside, I'm getting some interesting and frustrating issues.
Basically, any details close to the player are getting completely lost, and the base of the character model is fading out too.
And anything too far away from the player, or sometimes too close, is being painted pure black.

Here are some examples:

http://i769.photobucket.com/albums/xx334/nekoikono/outline_wip1.png
Basically, the "sea" area of the DM-Sanctuary map is being painted pure black at the extreme depths toward the edge of the map.

http://i769.photobucket.com/albums/xx334/nekoikono/outline_wip2.png
And here, all detail near the player is washed out in white.

I really am not sure why this is happening...
So, just wondering if anyone has any clues what's causing this?

Thanks. ^^

Update:

Just changed the filter around a bit so it uses less operations and determines edges based on a diagonal offset. The edge detection is working MUCH better now, but the issue with flat surfaces is still there and it still removes detail at close range.

http://i769.photobucket.com/albums/xx334/nekoikono/outline_wip3.png
As you can see here~

Blargh~! ^^';;

[KazeoHin]

when I created a toon effect, I didn't divide the scene depth. This reduces the accuracy, and since outlines compare values, you don't really need a 'range' anyway.

[neai]

Aside from precision issues and possible far view clipping, this looks about right for a sobel filter. It's approximate. Check the gradient of scene depth, in case it's not what you assume.

[Ikono]

Thank you, Kazeo and neai.
Eliminating the divide command has helped correct the view clipping at close and long range in certain cases.
I've been testing what value to use as the final comparison after the sobel filter and I'm unsure, because all options seem to generate insane amounts of "noise", and so I assume this might be noise within the scene depth itself. Though again, not entirely sure.

If I compare the final result of the sobel against the base scene depth value, I get this:
http://i769.photobucket.com/albums/xx334/nekoikono/outline_wip6.png

This noise is present regardless of the comparison I use, I can't seem to filter it out without inadvertantly also causing the issue of clipping the close details again.

I'll take a look at the scene depth itself, try and find where the noise is coming from, and see what I can work out.

Update:

Go figure, it's actually part of the Scene Depth node's value.
After plugging the base scene depth into a ramp texture sample, this is what I get.
http://i769.photobucket.com/albums/xx334/nekoikono/outline_wip7.png

So, the noise is coming from the scene depth itself.
Any suggestions on ways to clean this up at all?

[KazeoHin]

you can add in some math so that at grater distances it takes a greater difference in depth to render an outline. I remember having to do that with my Toon shader way back.

[Ikono]

Okietay~
Finally done, I had to settle for some of the black "noise" still being there but for the most part the depth check to increase the value of comparison has worked wonders.
So, many thanks Kazeo!

[sueds]

Ikono could you enlight us about your method to create a sobel filter withing the material editor ?

[crash0999]

I saw what you were doing with post processing and i was wondering if thats the approach to take when trying to create a game in black and white. basically im part of a group making a mod for unreal and we want everything to be black and white and or sepia to match the style and setting of the game. any ideas on how difficult it might be and/or where to start?

[Alex D]

My previous experience with doing post process outlines in UT3 made me really wish they had a SceneNormals node as that would allow much greater control over edge detection and further allow normal-mapped details to contribute to the outlines. It seems a bit limitting only having the SceneDepth to work from.

[Oniram]

Ikono could you enlight us about your method to create a sobel filter withing the material editor ?

x2. id really like to see your material creation method.

[CZorg]

x2. id really like to see your material creation method.

x3 that. This effect is so awsome. Maby you can post screenshots of how the shader is built?

[vizay]

I must say that it looks really awesome what you've done!
I've been fiddling around myself a bit with this kind of style for my current project, however I'm getting a more sketchy style, mainly due to a less advanced algorithm for the edge detection.

Here's what I've done so far
https://lh6.googleusercontent.com/_Zh_YxaPjjvc/TVuexVfxInI/AAAAAAAAAi0/lWIRD7zE5Qw/Shader-1.jpg

I'm wondering what references you used for building the sobel filter (mathematical wise), would be interesting to try and implement it myself but I've got no clue where to start out :)