Navigation mesh stops producing valid paths in Multiplayer.

[BooleanFiasco]

Another day, another strange MP-related issue!

I have a simple AI class that periodically spawns and paths toward a preset destination. This AI class spawns in waves every 10-20 seconds or so as long as the game is running, and doesn't do much besides walk toward its point and attack other enemies and players that it sees along the way (think DOTA).

The problem I'm having is that in multiplayer only, after about 15 seconds none of the AIs are able to generate valid paths anymore. They go from walking around no problem to all freezing in place and having NavigationHandle.FindPath() return false regardless of what location you give it. It seems like the entire nav mesh is getting corrupted but for the life of me I can't figure out how or why. The number of AIs spawned doesn't seem to matter - I tried spawning more than 50 and less than 10 and in both cases the nav mesh starts failing at exactly the same time (about 15 seconds after the first AI spawns).

One of the first things I considered was that the constraint list was filling up, but after double checking I definitely call NavigationHandle.ClearConstraints() before creating a new path. Are there more aspects of the nav mesh that need to be periodically cleared? And if so, why would this behavior be different between single and multiplayer?

Quite stumped! Any advice, experience or suggestions you guys can provide will be very much appreciated.

[RNG]

I tried using NavMeshes on multiplayer, used this tutorial as a starting point. It works on multiplayer right out of the box, and its easy to expand upon. You can compare your own pathfinding with this. Its quite basic, but it functions well.

I stopped using navmeshes after a while though, the generated paths are absolutely trash when it comes to tight places (tight for nav meshes, pretty wide in reality). Doing it manually is either a pain, or it doesn't work at all. Ended up using good old navigation points.

[BooleanFiasco]

My pathfinding is pretty similar to that, as well as a number of other tutorials on basic navmesh pathfinding in UDK. The AIs do path properly - it's just that after about 15 seconds of running, the entire navmesh becomes invalid in MP.

I've got no idea how or why it happens, and even less why it would happen only on a dedicated server. I'm going to keep hammering at it until I eventually sort it out.

[BooleanFiasco]

Still slogging away on this bug with no positive progress. I've had number of false fixes that didn't end up working - I tried giving the AIs shorter paths (in case some path cache was overflowing or something), tried spawning more and less AIs, made it so that nothing could die (to see if something wasn't getting destroyed correctly), tried running the server without any players connected and watching the logs...nothing.

Nobody else has ever seen similar behavior? For whatever reason, when running a dedicated server the navmesh will eventually become invalid and all actors that try to path on it will fail - even if the point is right next to them and directly reachable!

[Jetfire]

Two things have caught me out in the past-
Make sure your scout has a pathsizes entry for the smallest navigating pawn, including crouching- my ai were failing to nav when crouched because the navmesh generation hadnt defined walkable edges at that size;
Make sure you're not exceeding MaxPathVisits in navmeshgoal_at, subclass it if you need to to increase hte number allowed, I have it at 65536 with no discernable negative effect

Enable verbose path debugging, it should give you a cluie as to what constraints are throwing out all of your edges

[BooleanFiasco]

Thank you so much for the suggestions, Jetfire. I really appreciate your help!

Make sure you're not exceeding MaxPathVisits in navmeshgoal_at, subclass it if you need to to increase hte number allowed, I have it at 65536 with no discernable negative effect

Tried that, no dice unfortunately.

Enable verbose path debugging, it should give you a cluie as to what constraints are throwing out all of your edges

Thanks for this suggestion, I didn't know about that option! Unfortunately, the results of that aren't very helpful - the path search fails because InitializeSearch() returns false, and that's a native C++ function so I can't debug it any further than that.

It really seems like I must be hitting some sort of constraint limit, but I'm calling ClearConstraints() and I can't understand why that limit would be different between a dedicated server and a single player game. The code I'm using to create the paths is very simple, taken more or less directly from a number of different tutorials on the net:

function bool FindNavMeshPath()
{
	// Clear cache and constraints (ignore recycling for the moment)
	NavigationHandle.ClearConstraints();
	NavigationHandle.PathConstraintList = none;
	NavigationHandle.PathGoalList = none;

	// Create constraints
	class'NavMeshPath_Toward'.static.TowardPoint( NavigationHandle, NavigationHandle.FinalDestination.Position );
	class'NavMeshGoal_At'.static.AtLocation( NavigationHandle, NavigationHandle.FinalDestination.Position );

	//`log( "PathCache size:"@NavigationHandle.GetPathCacheLength() );

	// Find path
	return NavigationHandle.FindPath();
}

[Jetfire]

What does `log(NavigationHandle.LastPathError); give you after your pathfinds?

[BooleanFiasco]

Seems to give PATHERROR_GOALPOLYNOTFOUND.

The position that the AIs are trying to path toward is definitely inside the navmesh - it works initially, and then starts failing later on (they're always trying to path to a point on the other side of the map around 3000 units away).

[cargarage22]

Seems to give PATHERROR_GOALPOLYNOTFOUND.

The position that the AIs are trying to path toward is definitely inside the navmesh - it works initially, and then starts failing later on (they're always trying to path to a point on the other side of the map around 3000 units away).

I had a vaguely similar issue to this but it's probably not the same issue as it happened for me in single and multiplayer. So, sorry if this isn't much help.

Nav meshes didn't seem to work well with outdoor terrain that is hilly or even small bumps, and sometimes the AI would just freeze once it reached one. I think I fixed this by adjusting some values either on the collision of the pawns or the pylon, it's been a few months so I can't remember.

Other than that, I'm guessing that you've verified via logging that the destination position is correct while it's returning false?

[BooleanFiasco]

Yeah, I log the position when the call to FindPath fails, and the position is correct. I don't think this is a terrain issue since I'm using a perfectly flat brush floor, heh. The value you're probably referring to is MinWalkableZ, which I'm pretty sure isn't my problem due to the previously mentioned flat floor.

[cargarage22]

Yeah, I log the position when the call to FindPath fails, and the position is correct. I don't think this is a terrain issue since I'm using a perfectly flat brush floor, heh. The value you're probably referring to is MinWalkableZ, which I'm pretty sure isn't my problem due to the previously mentioned flat floor.

Have you tried making the aicontroller use the Scout pawns instead of your custom pawn to see if it makes a difference? This guy had similar issue, pawn moves for a while then stops, (http://forums.epicgames.com/threads/...vmesh-problems see bottom post).

[BooleanFiasco]

I don't think that the scout class he's referring to in that thread is meant to be spawned in the game - it's used by Recast to generate the navigation mesh at map compile time. I'm not using a custom class for that, just the default setting in DefaultEngine.ini (which is UTScout).

I went ahead and tried it anyway since I don't have any better ideas but it just results in the pawns falling out of the world (since Scout.uc calls SetCollision(FALSE,FALSE) in PreBeginPlay).

[Jetfire]

Scout is used for navmesh generation AND pathfinding at runtime, if you're using a custom scout, it needs to be set in two places in Engine.ini, for editor and runtime. Its not to be used AS your pawn.

[BooleanFiasco]

Scout is used for navmesh generation AND pathfinding at runtime, if you're using a custom scout, it needs to be set in two places in Engine.ini, for editor and runtime. Its not to be used AS your pawn.

That's what I figured. I haven't made any scout changes, I'm just using what UDK defaults to (which seems to be UTScout).

After further testing I can say that the problem is definitely tied to spawning. The number of AIs spawned doesn't seem to matter, but the time since the first AI was spawned does. I can spawn a dozen or more AIs just fine if I do it quickly and they'll carry on pathing around happily forever. But, spawning just two will cause the navmesh to start failing if I wait 15-20 seconds before spawning the second AI. As soon as the second AI spawns BOTH of them will start having PATHERROR_GOALPOLYNOTFOUND every time they try to generate one.

Going to continue poking at it and try to uncover more, but perhaps this will remind somebody else of something they've once encountered. Thanks for all of your help so far!

[BooleanFiasco]

You are never going to believe this, but I am going to tell you anyway because it has completely blown my mind. Executive summary: there is some sort of memory alignment issue in UDK when running the 32-bit executable in a 64-bit environment.

I have been pounding on this bug for several weeks and, in a final act of desperation, I burned everything down and started copying my child classes over line by line trying to identify what could possibly be causing this bug. I eventually narrowed it down to something in my Pawn class (and not in the Controller class, which was quite a surprise) and continue the process of adding one line at a time, recompiling, testing, ad infinitum.

After a few hours I finally narrowed it down to the presence or absence of some floating point variables declared in my Pawn class. I'll repeat that, because it bears repeating: the bug is caused by having too many member variables declared in your class.

I narrowed it down further to the exact byte by declaring a bunch of padding bytes, one at a time, until I got the bug to occur. I could declare a maximum of 3 bytes, and upon adding byte #4 and running my game the previously discussed nav mesh problem would occur. The header portion of my class is included at the end of this post for posterity, though its exact nature probably won't be helpful since the actual manifestation of this bug will depend on what your inheritance chain is, the number and nature of the interfaces you're implementing, etc.

After spending several minutes laughing like crazy person I realized that this was probably the result of running a 32-bit executable on my Windows 7 64-bit installation, and sure enough when I checked the shortcuts I use to run my game they were all pointing at the Win32 directory in my UDK install. Switching these to use the 64-bit version completely solved the bug.

If I may wax poetic for a moment, this is possibly the most amazing debugging effort of my programming career (I'm a professional game programmer in real life too). Managing to track down and debug a memory issue without any of the usual debugging tools is something I never, ever want to do again, but I feel like I can wear it as some sort of freakish medal.

Thank you to everyone here who tried to assist me - there's absolutely no way anybody would have guessed that this was the issue, and I'm grateful for all the very helpful suggestions I received along the way. I still learned a number of things about UDK and UnrealScript in the process of fixing this so I'm going to chalk it up as a learning experience.

The previously mentioned class header is below. The nav mesh failure will occur 100% of the time if the fourth padding byte (PaddingByte3) is uncommented and UDK is run using the 32-bit executable (or,for that matter, if you uncomment the other 3 actually useful floating points).

class MobaTestPawn extends Pawn implements( MobaTargetableInterface, MobaStatusInterface, MobaHUDInterface );

var() MobaSelectionComponent SelectionComponent;

var DynamicLightEnvironmentComponent LightEnvironment;

/** Slot node used for playing full body anims. */
var AnimNodeSlot FullBodyAnimSlot;

/** Overhead status bar values */
var float OverheadBarWidth;
var float OverheadBarHeight;
var float OverheadBarYOffset;
var float OverheadManaBarHeight;

/**
 * Attack and ability related values
 */
var float AutoAttackRangeSqr;
var bool bCanAutoAttack;
var() MobaHomingProjectile AutoAttackTemplate;

/** Will only be valid on AI controller minions */
var MinionReplicationInfo MinionReplicationInfo;

/** How much experience and gold this pawn is worth when killed */
var() int ExperienceValue;
var() int GoldValue;

/** Basic combat values all pawns have - these level up with the pawn (via time, exp, etc.) */
var() float BaseGroundSpeed;
var() int BaseHealth;
var() float BaseArmor;
var() float BaseAttackDamage;
var() float BaseAbilityPower;
var() float BaseResist;
var() float BaseAttackSpeed;        // Attacks per second

/** Used for displaying damage, healing, status effects, etc. */
struct DamageDisplayInfo
{
	var float       LifeTime;
	var float       StartTime;
	var string      Text;
	var LinearColor Color;
	var Vector2D    Direction;
	var float       Distance;
	var Controller  Instigator;
};

var array<DamageDisplayInfo> DamageDisplays;

struct MobaTakeHitInfo
{
	var float               Damage;
	var class<DamageType>   Type;
	var Controller          Instigator;
	var byte                UpdateStamp;        // Used to force the struct to replicate
};

/** Stores the most recent hit that the player has taken */
//var repnotify MobaTakeHitInfo LastHitInfo;
//var() float HitInfoDisplayLifetime;
//var() float HitInfoDisplayMinAlpha;
//var() float HitInfoDisplayMinDistance;
var() float HitInfoDisplayMaxDistance;

var byte PaddingByte0;
var byte PaddingByte1;
var byte PaddingByte2;
//var byte PaddingByte3;

[VendorX]

Tip: Use // instead of /**...*/... First, because you using only 2 not 5 and it looks better. Second, sometimes engine have problem with that too - especially if is used default properties.

[BooleanFiasco]

I only use that style for commenting member declarations at the top of a class, and then it's just a "when in Rome" situation since that's how the built-in UDK classes are commented (Pawn.uc, Controller.uc, etc). I use // comments in normal code and the defaultproperties block.

[b3h47pte]

That's absolutely bizarre haha. This is only a bug in multiplayer? I'm assuming then that you had to run the server in 64-bit instead of 32-bit? Or did you have to run the client in 64-bit too?

[BooleanFiasco]

That's absolutely bizarre haha. This is only a bug in multiplayer? I'm assuming then that you had to run the server in 64-bit instead of 32-bit? Or did you have to run the client in 64-bit too?

Yep, multiplayer only. The server needs to be run in 64-bit but the client can be 32 or 64, it doesn't matter. I'm not sure if the same problem occurs with a listen server (I'm running a dedicated server) but I imagine that it probably does.

My guess is that this is garbage collection related. The bug occurs very reliably 10-20 seconds after spawning an AI and doesn't seem to be affected by the number of AIs you spawn. You can spawn 30 or 40 of them just fine as long as you do it quickly, but spawning even 2 of them will cause the crash if you wait a while in between. I guess at some point the GC goes through and ends up freeing or moving the NavigationHandle in a way that the 32-bit executable simply can't handle, at least on a 64-bit system.

[GigaHobbit]

BooleanFiasco, your efforts sound truly heroic, hehe. Is this something that could be reproduced in the stock UDK or are some of your scripts required? I'd like to get a repro case setup for programmers here to take a look at.

Best.