The Fourth Benchmark: Unity 4.x

Awhile back, I did an entry I called, "A Tale Of Three Benchmarks" where I compared GameMaker Studio, Clickteam Fusion 2.5, and Construct 2 on a simple test to see which could push the most FPS under the following conditions:
  • 1028x800 resolution, windowed.
  • Bullets made up by 2x2 pixel sprites of a single color (black).
  • The bullets would bounce off each other.
  • The bullets would bounce off of the edges of the window.
  • There will be two text displays.  One displaying the frames per second, the other displaying the number of active bullets.
  • Clicking the mouse would generate a projectile at that point of the screen that would careen off at a random angle.
  • I will generate as many projectiles as I can until I hit an average of 50 frames per second, highest projectile count wins.   (This was because a lot of IDE would cap the FPS at 60, so I needed to find a point where the computer was visibly under load.)
  • The build target is always Microsoft Windows, if possible.  That's important, because it's entirely possible that the results would be completely different on something like an Android.
Despite its simple rules, this is a ruthless game logic benchmark.  In your average shmup, the bullets will only be tested against what they can hit.  But this is a basic 2D physics benchmark, the point is to generate load, and this can be accomplished quickly by testing everything against everything else.

In practice, each bullet is being compared against sides of the screen to see if they will bounce, but also every single bullet already in play.  The first bullet fired (before any other bullets were spawned) would be comparing its position to 4 things (the four walls surrounding the screen) to see if it needs to bounce off.  By the time we reach bullet number 100, it needs to compare itself to 103 things (four walls and ninety-nine other bullets) and so would all the other bullets already in play.  These calculations are run for every object in every frame.  In this way, it gets incrementally more expensive which each bullet you introduce to the scene.

If the engine being tested automatically optimizes out some of the calculations involved with a shortcut that has the same result, that's actually a good thing.  It means the engine is smart enough not to do literal busywork.  There are many such behind-the-scenes optimizations that will work for this situation.

My results were:
  • Construct 2 managed about 170 active projectiles.   (Note it operates on HTML5 regardless of export platform.)
  • GameMaker pulled about 600 projectiles.  I later came back when the YoYo compiler was released for free and was disappointed to discover it did not impact this benchmark.  (It's probably really more about optimizing needless calculation that can be done during compilation.)
  • Clickteam Fusion pulled about 900 projectiles.  You could say it "won."
Now that I have finished a few lectures on a related Udemy course, I am repeating this simple benchmark using Unity3D operating in 2D mode.  My CPU has not changed since I did the previous tests, although I did upgrade to Windows 10 and get a better graphics card.  Given that this is a game logic test, I believe that the bottleneck was always in the CPU (in my case, an AMD FX-8120) and your results would vary based on yours.
The main script on Monodevelop superimposed over my Unity 3D editor.
To make this test work in Unity 3D, I found that a few minor concessions had to be made:
  • A 1028x800 resolution was not permitted, so I am doing a (more standard) resolution of 1024x768.
  • The Box Collider 2D component could not recognize anything smaller than 5x5 pixels on the default 100 pixels per world unit sprite interpretation, so I upped the size of the sprite to the minimum size needed (5x5 px).
I could call this "close enough," but honestly Unity 3D's engine is radically different in philosophy from the others.  This is primarily because it is built around the idea of world space being modeled separately from pixels, whereas pixels and world space are fairly closely bound in the other three engines being compared (especially GameMaker). 

I am not saying either approach is superior, just that Unity 3D's method of representing space makes this somewhat apples to oranges.  In the end, I suppose it's the results that matter.  This is a simple test of heavy 2D projectile physics resolution.  How will Unity3D fare?

My general assumption here was that Unity 3D would do worse because it maintains much of the logic of a 3D engine even when running in 2D mode.  Theoretically, this should introduce quite a bit of overhead.  In fact, the executable file for this simple benchmark was almost 15 megabytes in size despite the only asset being a 5x5 pixel PNG file!

My chosen method of implementing this test was to use Unity's built in 2D physics system.  Part of this is because it is easier than coding my own scripts.  However, when I create these benchmarks, I try to use the method that seemed most native to the IDE.  In the other IDE, this meant sticking to drag-and-drop code methods whenever possible, and these are often the most optimized on the lowest level by the engine.  Unity3D's built-in preferred method seems to be the use of Colliders and physics materials.

The actual project was set up like this:
  • The camera was set to 1024x768 size.
  • The camera's viewport was manually surrounded by four "bumpers," Box Collider 2Ds that simply have a physics material 2D attached to them with fiction 0, bounciness 1.  They are of sufficient thickness so as to prevent any projectiles from escaping.  (Because the bumpers were placed manually, they don't really scale with resolution, but all tests were done at 1024x768.)
  • A simple canvas was added with two counters, one for FPS and another for Entities.  Like with the other benchmarks, the text for these UI controls were updated every frame whether they needed it or not.
  • The projectiles themselves had (besides the sprite) just a Rigid Body 2D component with no gravity and a Box Collider 2D component with the same physics 2D material as the bumpers.  The sprites were set to point filter which was probably less work for the renderer, and left my sprites looking as crisp as they could.
  • The projectile spawner object simply took my left mouse click to spawn a projectile where I was pointing and then imparted a random vector2D to its velocity.  (I later scaled this back to a range of 2.5f, but this made little difference.)
Long story short: no scripts were involved in simulating the physics; the movement and bouncing of the projectiles is 100% handled by Unity3D's own collision system.  The scripts that were in use were only for the text elements ("Frames Per Second" and "Entity Count") and the GameObject that contained the code used to instantiate projectiles.
It's not much to look at, but a whole lot of math is going on!
So here's the results:
  • On windowed mode with "good" graphics selected, about 1300 projectile entities were created before FPS dropped to 50 FPS.  (This is simply spawning them as fast as possible before they "settle," more on that phenomenon coming up.)
  • The same results seemed to apply regardless of graphics settings ("fantastic" and "fastest" settings were tested) although the UI reaction time was quicker on the lower settings and the "fastest" setting did not cap the FPS at 60.
  • The same results seemed to come from full screen mode.
  • It does not seem to matter if I built it for x64 or x86, the FPS would be the same.  Chances are this simple application does not require the memory that a true x64 application would benefit from. 
  • Playing the game from the editor was noticeably slower.  (In benchmarking all four IDEs, I tried to get the numbers from stand alone apps when possible.) 
  • When I enabled SLI mode on my video cards, I managed about 1650 projectile entities before the FPS dropped to 50.  Unless something else happened in the transition to SLI mode, it seems that the Unity3D engine actually leverages graphics card processing for its physics.
Color me surprised: Unity3D just dethroned the fastest 2D-dedicated game IDE in this benchmark test, Clickteam Fusion 2.  In fact, it seemed to do significantly better than the dedicated 2D IDEs I tested, and with fairly advanced physics components operating to boot!  (Now if only I could do something about that 15 megabyte executable!)

An interesting thing happened if I waited long enough: the FPS would first take a bit of a dip, down to about 10 FPS, and then it would raise and stabilize back to 60 FPS.  This did not really happen when I tested the other IDE.  I eventually figured out that the FPS was taking the largest hit from when collisions were actively happening.   There was a spike of collisions as spent projectiles that had tranferred their momentum to other projectiles began to settle in the corners.  Heaps of settled projectiles in close proximity to other projectiles created a potential for a collision to set off dozens, and there was where the lower FPS cane from.  Afterward, with less actively moving projectiles, there was simply less colliding going on, and so the FPS would climb.

Throughout the test, my CPU use never exceeded 17%, so it appears Unity3D is not leveraging all my cores for these physics.  You would think dedicating a CPU or two towards resolving collisions would make a big difference.  To those ends, I did a simple Google search and determined that multi-core physics optimizations were implemented in Unity 5.  Perhaps I should do another test to do in order to see if these optimizations would apply to this benchmark, but the tutorial series I am currently following wants me to stick to 4.x for code compatibility reasons.

But then, I suppose it's fair to say that optimization is always a thing, and in all of my benchmarks it's possible to improve the results further.  The important thing I learned here is that Unity3D does not actually lose 2D performance to its 3D engine.  That's pretty much all I needed to know.  Discovering that it actually seems to be preforming better than a dedicated 2D IDE is, in all honesty, a bit of a shock.


Popular Posts