Skip to main content

A Tale Of Three Benchmarks

Yesterday had me perform an experiment.  Scientifically speaking, it was a lousy experiment, with no real control factor and a thousand and one different ways to interpret the results.  However, I learned something nonetheless.

The goal of the experiment was to confirm whether or not my assumptions about three popular game engines were correct, at least in one important factor: processing speed.  So I set the specifications of a benchmark to create in each that had the following specifications:
  • 1028x800 resolution, windowed.
  • Bullets made up by 2x2 pixel sprites of a single color (black).
  • The bullets would bounce off each other.
  • The bullets would bounce off of the edges of the window.
  • There will be two text displays.  One displaying the frames per second, the other displaying the number of active bullets.
As both Clickteam Fusion 2.5 and Construct 2 attempt to cap themselves at 60 FPS by default, the goal is to see how many projectiles bring the engines down to 50 FPS on the same platform: my Windows 7 OS personal computer.

Experiment 1: Construct 2.
Having just recently finished an introductory tutorial on Construct 2, I was familiar enough with it to easily cobble together a 2x2 "bullet"-type movement object and put it against a black background.   I am using the free version, so the HTML5 default target platform was the most suitable for my Windows 7 target platform... why no standalone .exe, I wonder?   As with the other tests, unrelated settings were left at default.

The HTML 5 target export worked like a charm, but it only took about 170 active bullets to get my frame rate down to 50.   This was in Firefox; it was slightly faster in Chrome, but not by much (about 200 bullets).  This is the limitation of a javascript-centric engine, and I suspect that the frame rate would not improve even if I used their Windows.exe exporter (unfortunately not available in the free version, although there was an Windows 8 Store App exporter).

Experiment 2: Clickteam Fusion.

I had completed all of the Clickteam Fusion tutorials a couple weeks ago, but apparently that was not enough for it to move from my short term memory.  I ran into three problems replicating the code I made in Construct 2:

  • How does one spawn an object where the mouse is clicking?  It seems I only had a choice of spawn an object at a fixed coordinate or relative to something already in the screen, and there was no "mouse" object.
  • How does one keep the projectiles on the screen?   "If bullet is outside layout, bounce it (off of itself) back into the layout" was difficult to find, but I eventually recollected it was under the "test position" section.
  • How does one perform a textual display of FPS and entity count (from an incremented global variable)?  Turns out that the "text" objects were bad at this, there seems to be no means to do it, or else Clickteam Fusion's lack of Intellisense made it unable to point it out to me.  The answer is that you use "counter" objects, instead.
It's boggling, really - both Clickteam Fusion and Construct 2 do most things quite similarly, but there's just enough "French logic" in the wording of Clickteam Fusion to make performing the most rudimentary operations require some reinterpretation.

I eventually cobbled together "close enough" code above, and the results were that I could pull 900 moving projectiles before the FPS fell to 50 with my hardware. (Funny enough, this was only in the Windows-engine driven "preview" mode.  The free version of Clickteam Fusion only allowed me to build HTML 5, and their HTML 5 optimization is currently awful: it could only put out about 25 projectiles before the FPS had already fallen to 50!  Way to put your worst foot forward, guys.)

Experiment 3: Game Maker.

Alright, so I haven't dabbled with Game Maker Studio in months.  I tried to put together the benchmark in an identical manner to the other two, but found myself floundering because it has been so long since I used it.  Game Maker is simple compared to coding from scratch, but significantly more complicated compared to either Clickteam Fusion or Construct 2, it is just that much less visual even if they did bend over backwards to add "drag and drop" methods to nearly everything.

Where the latter two go out of their way to differentiate different types of objects, Game Maker treats all objects as the same and has you define how they behave.  I have a certain respect for that, I like an engine that lets you get at the nuts and bolts of things.  (In fact, if you want to get really technical, you can even see the memory addresses of your various elements!  I am guessing certain functions can make use of those addresses, because otherwise there would be no point in revealing them.)
I had forgotten how to do something as rudimentary as display my frames per second and entity count, and I could not figure out how to add watch variables to Game Maker's debugger.  Fortunately, I already created a similar benchmark project in the past.

Oddly enough, I could never get those projectiles to stay on the screen, even when I resorted to trying to wrap them around instead of bounce off the edge.  It's probably possible, I'm just doing a lousy job of coding in Game Maker.

Even if the projectiles were not on screen, the engine was still simulating their movement, so this would work for benchmarking purposes.  Game Maker could pull around 600 projectiles before the FPS fell to 50 on my hardware.  This is without the benefit of the Yoyo Compiler, which is advertised as providing 100x the speed, but I do not yet have access to it.


It turns out my initial assumptions were correct.  Clickteam Fusion can produce the most optimized products for Windows, Construct 2's javascript dependency made it the slowest, and Game Maker had some optimization but was not quite as fast as Clickteam Fusion (unless you get that Yoyo Compiler, although I cannot verify this firsthand).

I would like to reiterate what was saying earlier: these findings lack sufficient controls to be considered scientifically accurate.  The main problem is that there is probably quite a bit of black box mechanics going on in each engine, and all I did was test one facet of that engine (projectiles) and possibly did not even code them in the most efficient way possible.  It's possible each engine has strengths and witnesses in the performance of different kinds of operations.
Considering the kinds of games that tend to come out of these engines, and I am inclined to believe this frame rate is not really sufficient for truly impressive games.  When I look at something like Freedom Planet, or Noitu Love that is the very best that the engines seem to be doing, two Clickteam engine products that actually look like pretty good games for early 1990s consoles like the Sega Genesis.  All these engines really need a lot more power to produce some truly impressive games, it's really no wonder a lot of serious developers have to resort to developing their own engines.  Power equals brains, and this is what PC games need the most.

It's a pity, because the worst-performing engine, Construct 2, is an absolute joy to develop in.  Coding this benchmark on that IDE went a lot smoother than the others, and this is not completely because I had completed its tutorial so recently.  It just has a radically improved GUI over the product it was initially created to supersede, and the implementation of intellisense auto-completion works well and made a big difference.  Despite not being a very good performer, Construct 2 is a very tempting purchase for its incredibly user-friendly interface and an extremely attractive pricing structure, and it is probably going to be the choice for any 2D product that does not require much in the way of power, especially on the HTML 5 platform where speed really is not much of a possibility anyway.

Coincidentally, I was just poking around Yoyo Games's website and it looks like they're including the Yoyo compiler for free soon, targeting November 6th.  This could radically alter the face of homebrew games, as its advertised performance gains would be adequate to make some really good PC games, and many people (myself included) would be willing to humor a tougher learning curve if it means getting performance power severalfold that of the alternatives.  I look forward to giving it a try with the new compiler and seeing if the advertised performance is as good as they say, even 1.5x would put Game Maker neck and neck with Clickteam Fusion, but 10x-100x would blow everything else away.
Post a Comment

Popular posts from this blog

Ancient Warfare - What Is It Good For?

The Ancient Warfare mod for Minecraft threw me for a loop.  I was looking for "villagers" that would perform useful tasks while simultaneously resolving the glut of food with a need to eat, thereby turning Minecraft into a bit of 4X game you can play from the inside.  Millenaire wasn't quite there, partly because recent updates to Forge had broken its compatibility with Minecraft 1.7.10, and Minecolony's development is not quite fast enough to keep up with the state of mods in general (they probably need to make a core API).
In comes Ancient Warfare, which does indeed provide workers and soldiers who need to eat, you can even order around a little army of them to defeat your enemies.  It has working waterwheels and windmills, something I thought was awesome in Resonant Induction.  It has a warehouse with a built-in sorting system, as well as courier NPCs that can move things from building to building, and crafting NPCs that can create things for you automatically - w…

Resonant Induction Really Grinds My Gears... In A Good Way

From about 2pm yesterday until 8pm today, I've been dabbling with my latest custom mod mix for Minecraft 1.6.4, which is this time very much Universal Electricity focused.
Aside from the usual GUI enhancers and Somnia, the primary contenders in this mix were:
Calclavia Core - Of course: this is the base of the Universal Electricity system.Resonant Induction - This seems to be largely focused on increasingly more advanced methods of refining ores divided across 4 ages of technological progression.  It also includes some really cool things such as assembly lines.  I'll primarily be talking about just a few blocks out of this mod today.Atomic Science - A mod dedicated to generating more of those lovely universal electricity volts via the power of splitting the atom.  Build your own nuclear reactor!  Deal with nuclear meltdowns!  You maniac!ICBM - A mod dedicated to generating more destruction using those lovely universal electricity volts (and more than a little gunpowder), it cer…

Not Getting Over It

One of the recent fads to go through YouTube recently is Getting Over It With Bennett Foddy, a game about a man sitting in a cauldron filled with his own sweat dragging himself up a mountain of societal detritus with nothing more than a huge staking hammer.
As can be expected from the creator of QWOP, the game is extremely hard to control, requiring great precision on behalf of the player's mouse skills in order to propel the man up the mountain.  It is known as a "rage game" because it is deliberately engineered to punish the player.

I think I figured out a significant chunk of the source of that rage.  Basically, by investing our time and energy into trying to figure out how to get the little man on the screen up the mountain, we come to care about it.  Even though there's nothing important waiting for us at the top of our trash mountain, even though it's a ridiculous little game about mountain climbing, we care about it because we are trying.

By wanting to win…