Landscape drawscale crash 2013-07 build

[Farseer]

Another curious issue, encountered while migrating AFF to the latest UDK build - the 2013-07 one:

We have a map with a 509x509 landscape. Admittedly it's probably a bit too dense because it's been scaled down to DrawScale = (48.0, 48.0, 18.0), so every tile is just 48x48 uu. However it used to work perfectly in the previous build we were on - 2012-07.

Now in 2013-07 the presence of that landscape crashes the whole engine on collision checks performed in the map. Be it moving vehicles or just tracing (and not even tracing directly at the landscape surface, it's enough to just point the trace at a staticmesh near the landscape, perhaps in the same octree region or however it works under the hood), everything results in an immediate crash with the callstack pointing to some PhysX guts:

[0097.86] ScriptLog: VOIP| SetupPlayerMuteList:  Player [ Player257 ] doesn't have a UniqueNetId.  Nothing to do.
[0109.20] DevGarbage: Collecting garbage
[0109.26] DevGarbage: 57.192288 ms for realtime GC
[0109.27] DevGarbage: 12.375299 ms for unhashing unreachable objects
[0109.33] DevGarbage: GC purged 1604 objects (172758 -> 171154)
[0138.07] Log: === Critical error: ===
Fatal error!

Address = 0x7155a386 (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\PhysXCore64.dll]
Address = 0x71776c55 (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\PhysXCore64.dll]
Address = 0x716b60e5 (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\PhysXCore64.dll]
Address = 0x884f9fc  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7b00  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f7e87  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x83f8fc9  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x82dffc9  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x82e0716  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x8446cda  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dc7079  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dbc20e  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcedde  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dc7079  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dbc20e  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcedde  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcedde  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dc7079  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcedde  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dc7079  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcedde  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcedde  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dcbff0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7dd6c37  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x80af849  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x841f658  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x81903b8  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x82558ed  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x8256cca  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x9338ba0  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x933deee  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x933df5a  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x125616d4 (filename not found) 
Address = 0x12511ee8 (filename not found) 
Address = 0x71b5255a (filename not found) [in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll]
Address = 0x1256124e (filename not found) 
Address = 0x15b2758  (filename not found) 
Address = 0x7bc0000  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]
Address = 0x7bc0000  (filename not found) [in D:\UDK\affJuly13\Binaries\Win64\UDK.exe]

[0138.08] Exit: Executing UObject::StaticShutdownAfterError
[0138.08] Exit: Executing UWindowsClient::ShutdownAfterError
[0138.12] Exit: Executing UWindowsClient::ShutdownAfterError
[0138.40] Log: Shutting down FaceFX...
[0138.42] Log: FaceFX shutdown.
[0138.42] Log: appRequestExit(1)

Obviously can provide dump file along with that if needed.

We did some research and it seemed the issue only occured if the DrawScale on the terrain got dropped below the defaut 128 on X or Y axes. I wasn't sure if that's a valid assessment, could be more related to the fillrate of those landscape tiles when they get smaller or any number of things instead of directly to the DrawScale value.

So then I tried making the landscape DrawScale power of 2. And that surprisingly fixes it. So terrain with DrawScale 48x48 crashes, but with 32x32 or even 16x16 doesn't. Landscape specification never said anything about the need to make tile sizes ^2.

As I said, the issue occurs in 2013-07 but not in 2012-07, anything changed in the engine regarding landscape collision between those builds?

For the record, this is not an issue with any landscape properties - only DrawScale affects that and it happens regardless of number of components, regardless of heightmap or foliage or anything. Happens on any landscape no matter if it comes from a map from a previous build or is newly created. The only other thing that seemed to fix it was cranking up CollisionMipLevel - I presume it, when set to high enough exponent, did the same for collision as increasing the DrawScale beyond the default 128.0.

[DemoMan2]

I don't think your problem is with landscape size. I just tested a 4033 x 4033 level in the July 2013 version of UDK and it worked fine.

Are you sure that you copy of UDK is installed correctly? In looking at those error messages in the log file it seems to indicate that UDK.exe cannot be found.

[Farseer]

It's not with the landscape size, it's with the landscape's DrawScale, number of components is just one of the factors that help reproduce the issue. Under the parameters I provided the crash is 100% reproducible in the 2013-07 64bit build, not sure if it's the same on 32bit, will try later.

The error message has nothing to do with missing files (the UDK callstack you see on crash never reports actual missing files despite saying so), it's the physx DLL falling apart at some physical calculation done on the terrain.

[DemoMan2]

Well, I see what you mean.

I took one of my maps (i.e. it's a 1009 x 1009) and changed the landscape properties of the map from (256 x 256 x 64) to (48 x 48 x 18). Interestingly enough, the editor seemed fine with it but when it was run with UDK I got a crash just as you described. So is this just one map that you are working on or is it a whole series of maps that need this fix? Also, do these maps already have static meshes, pathnodes, playerstarts etc on the maps? The reason I ask is to give you some help in finding a fix to your problem.

[DemoMan2]

BTW, that's some impressive work you guys have done with Angels Fall First.

[DemoMan2]

Here's something you should try. I know I was having some problems with recent Nvidia drivers with unreal engine games (i.e. UDK, UT3, Borderlands 2 etc). If you look at the Nvidia 320.xx driver series logs, they did update the physicx stuff and perhaps that's whats giving you and me our problems. So here what I propose.

1.) Try running your level in UDK on a machine with an ATI video card. Do you have the same issue?
2.) Try running your level in UDK on a machine with an 314.xx Nvidia driver. Do you have the same issue?

This is just to find out if the problems on the UDK side or the Nvidia driver side of things.

[Farseer]

Thanks, we're trying our best to make AFF into a proper game at last.

So, despite suspecting PhysX doesn't run the essential calculations on GPU anyway, I tried with different cards and drivers. Nope, no difference, crash still 100% reproducible.

However, I've finally got some time to test it and it seems the issue only occurs on 64bit builds of UDK. Run the 32bit version of the game, crash is gone. Appears something between the 2012-07 and 2013-07 builds got messed up with the integration of the 64bit PhysX libs...

[DemoMan2]

How interesting.

Who would have thought that the 64 bit version of the .exe would be the problem. Keep us informed on what you find.