Skip to main content

A Tale Of Three Benchmarks

Yesterday had me perform an experiment.  Scientifically speaking, it was a lousy experiment, with no real control factor and a thousand and one different ways to interpret the results.  However, I learned something nonetheless.

The goal of the experiment was to confirm whether or not my assumptions about three popular game engines were correct, at least in one important factor: processing speed.  So I set the specifications of a benchmark to create in each that had the following specifications:
  • 1028x800 resolution, windowed.
  • Bullets made up by 2x2 pixel sprites of a single color (black).
  • The bullets would bounce off each other.
  • The bullets would bounce off of the edges of the window.
  • There will be two text displays.  One displaying the frames per second, the other displaying the number of active bullets.
As both Clickteam Fusion 2.5 and Construct 2 attempt to cap themselves at 60 FPS by default, the goal is to see how many projectiles bring the engines down to 50 FPS on the same platform: my Windows 7 OS personal computer.

Experiment 1: Construct 2.
Having just recently finished an introductory tutorial on Construct 2, I was familiar enough with it to easily cobble together a 2x2 "bullet"-type movement object and put it against a black background.   I am using the free version, so the HTML5 default target platform was the most suitable for my Windows 7 target platform... why no standalone .exe, I wonder?   As with the other tests, unrelated settings were left at default.

The HTML 5 target export worked like a charm, but it only took about 170 active bullets to get my frame rate down to 50.   This was in Firefox; it was slightly faster in Chrome, but not by much (about 200 bullets).  This is the limitation of a javascript-centric engine, and I suspect that the frame rate would not improve even if I used their Windows.exe exporter (unfortunately not available in the free version, although there was an Windows 8 Store App exporter).

Experiment 2: Clickteam Fusion.

I had completed all of the Clickteam Fusion tutorials a couple weeks ago, but apparently that was not enough for it to move from my short term memory.  I ran into three problems replicating the code I made in Construct 2:

  • How does one spawn an object where the mouse is clicking?  It seems I only had a choice of spawn an object at a fixed coordinate or relative to something already in the screen, and there was no "mouse" object.
  • How does one keep the projectiles on the screen?   "If bullet is outside layout, bounce it (off of itself) back into the layout" was difficult to find, but I eventually recollected it was under the "test position" section.
  • How does one perform a textual display of FPS and entity count (from an incremented global variable)?  Turns out that the "text" objects were bad at this, there seems to be no means to do it, or else Clickteam Fusion's lack of Intellisense made it unable to point it out to me.  The answer is that you use "counter" objects, instead.
It's boggling, really - both Clickteam Fusion and Construct 2 do most things quite similarly, but there's just enough "French logic" in the wording of Clickteam Fusion to make performing the most rudimentary operations require some reinterpretation.

I eventually cobbled together "close enough" code above, and the results were that I could pull 900 moving projectiles before the FPS fell to 50 with my hardware. (Funny enough, this was only in the Windows-engine driven "preview" mode.  The free version of Clickteam Fusion only allowed me to build HTML 5, and their HTML 5 optimization is currently awful: it could only put out about 25 projectiles before the FPS had already fallen to 50!  Way to put your worst foot forward, guys.)

Experiment 3: Game Maker.

Alright, so I haven't dabbled with Game Maker Studio in months.  I tried to put together the benchmark in an identical manner to the other two, but found myself floundering because it has been so long since I used it.  Game Maker is simple compared to coding from scratch, but significantly more complicated compared to either Clickteam Fusion or Construct 2, it is just that much less visual even if they did bend over backwards to add "drag and drop" methods to nearly everything.

Where the latter two go out of their way to differentiate different types of objects, Game Maker treats all objects as the same and has you define how they behave.  I have a certain respect for that, I like an engine that lets you get at the nuts and bolts of things.  (In fact, if you want to get really technical, you can even see the memory addresses of your various elements!  I am guessing certain functions can make use of those addresses, because otherwise there would be no point in revealing them.)
I had forgotten how to do something as rudimentary as display my frames per second and entity count, and I could not figure out how to add watch variables to Game Maker's debugger.  Fortunately, I already created a similar benchmark project in the past.

Oddly enough, I could never get those projectiles to stay on the screen, even when I resorted to trying to wrap them around instead of bounce off the edge.  It's probably possible, I'm just doing a lousy job of coding in Game Maker.

Even if the projectiles were not on screen, the engine was still simulating their movement, so this would work for benchmarking purposes.  Game Maker could pull around 600 projectiles before the FPS fell to 50 on my hardware.  This is without the benefit of the Yoyo Compiler, which is advertised as providing 100x the speed, but I do not yet have access to it.

Conclusions.

It turns out my initial assumptions were correct.  Clickteam Fusion can produce the most optimized products for Windows, Construct 2's javascript dependency made it the slowest, and Game Maker had some optimization but was not quite as fast as Clickteam Fusion (unless you get that Yoyo Compiler, although I cannot verify this firsthand).

I would like to reiterate what was saying earlier: these findings lack sufficient controls to be considered scientifically accurate.  The main problem is that there is probably quite a bit of black box mechanics going on in each engine, and all I did was test one facet of that engine (projectiles) and possibly did not even code them in the most efficient way possible.  It's possible each engine has strengths and witnesses in the performance of different kinds of operations.
Considering the kinds of games that tend to come out of these engines, and I am inclined to believe this frame rate is not really sufficient for truly impressive games.  When I look at something like Freedom Planet, or Noitu Love that is the very best that the engines seem to be doing, two Clickteam engine products that actually look like pretty good games for early 1990s consoles like the Sega Genesis.  All these engines really need a lot more power to produce some truly impressive games, it's really no wonder a lot of serious developers have to resort to developing their own engines.  Power equals brains, and this is what PC games need the most.

It's a pity, because the worst-performing engine, Construct 2, is an absolute joy to develop in.  Coding this benchmark on that IDE went a lot smoother than the others, and this is not completely because I had completed its tutorial so recently.  It just has a radically improved GUI over the product it was initially created to supersede, and the implementation of intellisense auto-completion works well and made a big difference.  Despite not being a very good performer, Construct 2 is a very tempting purchase for its incredibly user-friendly interface and an extremely attractive pricing structure, and it is probably going to be the choice for any 2D product that does not require much in the way of power, especially on the HTML 5 platform where speed really is not much of a possibility anyway.

Coincidentally, I was just poking around Yoyo Games's website and it looks like they're including the Yoyo compiler for free soon, targeting November 6th.  This could radically alter the face of homebrew games, as its advertised performance gains would be adequate to make some really good PC games, and many people (myself included) would be willing to humor a tougher learning curve if it means getting performance power severalfold that of the alternatives.  I look forward to giving it a try with the new compiler and seeing if the advertised performance is as good as they say, even 1.5x would put Game Maker neck and neck with Clickteam Fusion, but 10x-100x would blow everything else away.

Comments

Popular posts from this blog

Empyrion Vrs Space Engineers: A Different Kind Of Space Race

In my quest for more compelling virtual worlds, I have been watching Empyrion: Galactic Survival a lot this bizarro weekend, mostly via the Angry Joe Show twitch stream.  What I have concluded from my observations is Empyrion is following in Space Engineers' shadow, but it is nevertheless threatening the elder game due to a greater feature set (the modding scene notwithstanding).

Empyrion is made in Unity, whereas Space Engineers is built on a custom engine.  While this does put Empyrion at a disadvantage when it comes to conceptual flexibility, its developers nevertheless have a substantial advantage when it comes to adding features due to a savings of time spent that would have gone into developing their own engine.  Examples include:
Planets.  Empyrion already has planets and space to explore between them, whereas in Space Engineers planets are in the works but still awhile away (so you just have asteroid fields to scavenge).Enemies.  Space Engineers' survival mode boasts onl…

Resonant Induction Really Grinds My Gears... In A Good Way

From about 2pm yesterday until 8pm today, I've been dabbling with my latest custom mod mix for Minecraft 1.6.4, which is this time very much Universal Electricity focused.
Aside from the usual GUI enhancers and Somnia, the primary contenders in this mix were:
Calclavia Core - Of course: this is the base of the Universal Electricity system.Resonant Induction - This seems to be largely focused on increasingly more advanced methods of refining ores divided across 4 ages of technological progression.  It also includes some really cool things such as assembly lines.  I'll primarily be talking about just a few blocks out of this mod today.Atomic Science - A mod dedicated to generating more of those lovely universal electricity volts via the power of splitting the atom.  Build your own nuclear reactor!  Deal with nuclear meltdowns!  You maniac!ICBM - A mod dedicated to generating more destruction using those lovely universal electricity volts (and more than a little gunpowder), it cer…

Sometimes, The Cat Skins You

The formula for cat girls is simple enough: young girls are cute, cats are cute, so young girls who are also cats are cute times two, right?  ... Cat Planet Cuties (a.k.a Asobi ni Iku yo) does not stop there, it includes girls with guns, nudifying weaponry, and even failed romantic comedy that shows lots of skin.  Cat's out of the bag: Cat Planet Cuties is basically straight up wish fulfillment.

The overall plot is pretty simple.  The Catians are a race so advanced that they have not changed in over 70,000 years.  They discover Earth, a planet whose people have uncanny resemblances, right down to speaking the same language!  Desiring an escape from their cultural stagnation, they rename their own planet from Earth to Catia and dispatch a scout to assess the viability of first contact with this new Earth inhabited by humans.  High jinks ensue.

Other than an excuse to see some fun sci-fi devices, the plot sucks. Let me count the ways:
Kio Kakazu - The male center of our harem, a 1…