Gradually degrading performance [possible memory leak?]

[ThraxxMedia]

Hi there Epic Games dudes,

I'm posting this into the UDK forums because of a simple reason: the following problem/bug/issue which I am going to describe in detail applies to ANY game/product created with (a certain version of) UDK and/or the Unreal Engine 3.

I'm well aware that Epic Games will not (or cannot) offer support for specific game related questions in many cases, but since I strongly believe this is engine related, I'll try my luck here.

Please let me quote my initial post from the BioShock Infinite technical support forum (BS:I was created with UE3):

 

Spoiler

Hello there,

first off, I really enjoy playing BioShock Infinite as well as a couple of other 2K Games but unfortunately some of them (including BSI, that's why I'm opening this thread) are packed with a weird and seemingly - as of yet - unresolved performance issue.

I will go into all the details to make it perfectly clear what I'm talking about, since I believe this issue I'm having is very often mistaken with other ones (maybe because they're somewhat similar in nature), and thus the offered solutions do not apply.

I am well aware of the already ongoing discussions about stuttering and performance drops, but if you read on, you'll see that this is a different problem. For a summary of my system specs, please scroll down to the bottom of this post.

Short description
The game plays fine without any flaws, hiccups or signs of performance loss for about 1-2 hours (sometimes less). After that, if you have something to monitor the current frames per second, you'll notice it fluctuates slightly, starting to degrade. The game is still playable at that point, framerate-wise, but the first noticeable thing is mouse input getting more and more unresponsive (as if slowed down) and the audio begins to crackle [slow mouse movement seems to be tied directly to your screen FPS, so it only affects you IF you have things like bSmoothFramerate or VSync enabled].

This effect is getting worse over time, dropping the game's FPS to as low as 15-20, resulting in massive stuttering (video and audio!), rendering it completely unplayable. The only current way to get rid of the issue is to restart the game client. This will temporarily solve it, until performance degradation kicks in again as aforementioned.

Elaboration
Some further facts about the issue:

• Even if you do not move your character and face a wall or something (so the rendered image is in no way graphically intense) the performance will still drop over time.
• GPU usage noticeably drops from about 50-60% (normal load for my case in situations like the above) to ~20% or less.
• CPU usage doesn't seem to be affected since it never changes in unusual ways.
• Both GPU and CPU temperatures are absolutely fine (GPU at a steady 70-73°C, CPU varies from ~45-60°C depending on load)
• Slowdown of mouse input affects the whole system, even if you Alt+Tab into Windows [tied to VSync options, as stated above].
• Performance will not drop to zero, degradation stops at mentioned 15-20 FPS (tested over the course of 5-6 hours), though it doesn't matter because the game will be unplayable at that point.

This specific kind of performance drop occurs to me in the following games: BioShock Infinite, XCOM: Enemy Unknown, Aliens: Colonial Marines, Remember Me, TERA and a fan-made game in the Warcraft universe created with UDK. They all share the common fact that they're based on Unreal Engine 3.

At first thought, this would lead to the conclusion that all UE3 games may potentially be affected by this issue, but that's proven to be untrue. I'm also playing games like Mass Effect (the whole trilogy), Mirror's Edge, Batman: Arkham City or Dishonored, which are based on the very same engine, but never gave me such problems.

Any other game, be it a resource hog like Crysis 3/Far Cry 3 or whatever, is not affected by this! (aside from the possibility each game brings its own specific bugs, but that's another story) - they play smooth and stable on my system, mostly with maxed out settings.

With that in mind, and after all the research I've done on the internet, I'm led to believe the issue might just be caused by poorly modified/non-optimized versions of the Unreal Engine. Unfortunately, any attempt on my behalf to fix it via config modifications, driver updates/settings, use of third party tools etc. has been unsuccessful.

A list of what I've tried so far:

• Lowering graphic details or other performance tweaking options ingame doesn't do anything good.
• Neither enabling/disabling vertical sync through the game nor forcing its state through drivers or tools makes any difference, except giving a more or less smooth playing experience as long as the overall performance hasn't dropped yet.
• Any combination of turning vsync on/off and setting the config parameters of bSmoothFrameRate to false/true doesn't make a difference either.
• Any other config tweak, from increasing pool sizes, over various ISVHacks, to enabling/disabling features like background streaming, texture file cache, OneFrameThreadLag etc. either crashed/messed up the game or simply didn't work as well. To clarify: I'm not editing the wrong config files, for I am seeing my changes taking effect (not the desired one, though).
• Reading the pool size from config file (via -ReadPoolSizeFromIni switch) instead of letting the game do the calculation either results in massive texture pop-in problems or overall lowering of texture quality.
• Update to the latest version of audio/video/chipset drivers (no difference if beta or stable release is used) doesn't help.
• Neither setting driver parameters to fixed values nor letting the application control them helps.
• Turning off overclocking features like AMD OverDrive doesn't help.
• Turning off power saving features like AMD Cool'n'Quiet doesn't help.
• Using tools like RadeonPro and creating a custom profile with recommended settings from people who had their stuttering issues solved doesn't help.
• Some people pointed out issues with the CEG protection of Steam, which might lead to stuttering in some cases. Using a recommended tool called Orochi-CEG to fix this didn't help.
• Disabling the ingame overlay of Steam/Origin doesn't help.
• Updating the game itself doesn't help (most of them are self-updating through Steam/Origin anyway).
• Switching from fullscreen to windowed or vice versa doesn't help.
• Updating BIOS to the newest version doesn't help.
• Memtest doesn't give me any errors. My system doesn't suffer from BSODs or anything like that.
  
• Updating to the latest game version (patch level 5) doesn't help. *edit 13/07/09*
• Spoofing video device ID of the graphics card to another device/manufacturer doesn't help (RadeonPro). *edit 13/07/09*

If you ever encountered this exact issue once, you may reproduce it any time again on the same system, no matter what. Just start up one of the affected games, sit there and wait. Problem is not related to what you're doing ingame. I am totally helpless here, it destroys my gaming experience. Too bad, because especially BSI and XCOM were two of my latest favourites.

System details
OS: Windows 7 x64 SP1
Graphics: AMD Radeon HD 7970 @Catalyst 13.5b2
Sound: Realtek HD Audio (ALC892, onboard) @v6.0.1.6642
Mainboard: ASUS M5A97 PRO @BIOS v1604
CPU: AMD FX-6100 Six-Core 3.30 GHz
RAM: 16 GB of AMD Memory (Performance Edition)
HDD: 1 TB Samsung HD103UI (OS/pagefile is running on a separate partition/HDD)
Desktop properties: 1920x1080 @60Hz (that's how I usually play my games, too)

If you have any further questions regarding specs, symptoms or things I tried etc. or if you have the opinion that I missed something else, please feel free to leave a comment below. Needless to say, if you happen to know a working solution, your reply is most welcome.

The whole conversation can be followed here:
http://forums.2kgames.com/showthread...ng-performance

As you can see, I'm totally helpless with this issue. It does occur once in a while for other people, too - if you google specific phrases (describing the exact problem) you'll find a couple of posts on different game forums, but there's never any working solution (sometimes not even any reply at all).

An interesting fact might be, that this only occured for me in games AFTER (or around) the release of XCOM/Dishonored (which were released in the same week iirc), so it may or may not be tied to a version of UDK/UE3 past that date.

If anyone has insight or clues to this, I'd be most thankful.

/edit
There's a video showcasing the exact behavior of this issue, here:
http://www.youtube.com/watch?v=g2zzouC0V1E

Please skip directly to 08:26 and don't mind the rest of the video. You'll see what I'm talking about.
Thanks

[nonder]

Try to watch your RAM. Maybe it can be somehow connected with huge memory leak in last UDKs (It will be fixed in upcoming beta).

Doesn't leak
2011-05

Does leak (older versions are not tested)
2012-05 - till now

http://forums.epicgames.com/threads/...ht=memory+leak

[ThraxxMedia]

Well, I thought of that, too. Strange thing is though, that the issue I described also happens when no new data (be it from level, NPC, whatsoever) is loaded. Meaning: you can start up a new game, sit there in your starting area and wait for certain amounts of time. If you happen to be one of the unlucky people who is affected, your performance will drop.

I read that changing levels (and thus loading more data into the RAM frequently) does speed this process up, but still, even without any loading it happens.

I watched my RAM, I got 16 GiB of physical memory and it never reaches a cap where it would run THAT low to have such a heavy impact on the game. But then again, are UDK/UE3-based game executables even able to use that much memory? Maybe it's capped somewhere else (I read something about 32bit applications only having a maximum address space of 4 GiB if some "large address aware" flag is set), and once it reaches 4 GiB it hits the ceiling.

But wouldn't the game crash at that point? For me, it just keeps on going, but with horrible audio-visual and input lag.

Another thing which comes to my mind - if this bug/leak would be true for anyone who uses UDK or a game based on that engine, wouldn't that mean, anyone playing the mentioned games would have this problem? How come only a small amount of people seem to encounter this? My best guess would be, UE3 has some poorly written code targetting some very specific hardware components. Maybe it hates 100%-AMD-systems or something like that, and if your rig happens to fit into the picture, you're screwed like me. :/

(for my full system specs you may check the quoted post from above)

[cube2222]

Well, new udk version out today. You can check whether it's fixed :P

[ThraxxMedia]

Since I'm only playing games based on this (and not using UDK as a developer myself), I'm not really able to check right now. But thank you very much for the info.

[cube2222]

Fixed, at least release notes say that.

[ThraxxMedia]

I finally found a working solution for the issue, and apparently, it's not a memory leak. The Unreal Engine seems to struggle badly with certain multi-core CPUs (such as mine), thus producing the described symptoms. If the affinity is manually set to just the first core only, the issue disappears.

Again, this only happens for me in games based on UE3, so the developers might want to look into this and see if there are any discrepancies regarding multithreading/multi-core CPU handling (especially aimed at AMD processors since this is what I got; updated the post on the 2k forums which I linked above accordingly).

On my rig, this issue is reproducable at any given time. I'm glad I don't have to worry about it anymore, it was game breaking.

Thanks for hearing me out.

[cube2222]

The new UDK patch fixed it. But the games made with older versions, will still have the issue propably.

[ThraxxMedia]

Just to be on the safe side - you're not talking about this:

At a quick glance, these were the issues tackled:
-Resolved the streaming memory leak issue.

But about the issues with multi-core CPUs (which I was describing in detail above), right?

If that is so, I'll see what can be done to animate the game's developers to put this onto their lists for future patches.

Thanks again.