So this year ... it's been a huge one. Playtomic has grown an insane amount in so many ways. At the end of last year I wrote about what 2010 was like, looking back it's hard to believe once upon a time we only did 300 million events a day and that I was the only developer.

We closed a tidy round of funding from our awesome investors and friends Daniel McNeely from Armor Games, Jameson Hsu and Bob Ippollito from Mochi Media and the founder way back when of Tiger Electronics Randy Rissman who pioneered casual games back when they required AA batteries!

This money has let us put together a small team of great developers who make my life easier every day, it let us shift the platform to an array of beefy servers - way back when the API servers were just four virtual machines, and it let me sleep at night knowing where my rent check was coming from because I was busting my ass all day and night keeping this ship floating instead of making games I could sell the licensing for through our friends at Flash Game License, mostly to Martine Spaans from SPIL Games although she's now with Blue Byte, makers of Settlers!

At the start of this year we supported just Flash games and right at the end of the year a JavaScript version of our API for the then budding HTML5 game movement. We've now added Unity3d, Objective C for iOS, Java for Android, Mono.NET for various mobile platforms, C++ for Windows, Mac, iOS, Android and Linux soon. One of our awesome users half-ported the API to Lua which we're hacking away on in our spare time, and we've got a HaXe version that is almost ready to release as well. Someone even made it work in Construct 2 which is just awesome. I don't even know most of the platforms you can use Playtomic in these days which is kind of a funny feeling.

These days we are doing well over a billion events a day! The other week I was looking in our admin dashboard and we'd done 1.7 billion the day before ... almost 6 times what we were doing this time last year, and on our way to doubling what we were doing just back in August. At times we get almost 1/2 a million people are playing your games at the same time which is just incredible.

We finally got rid of our pricing page by renaming it "premium". Tricky stuff, but a lot of people would click through to that page to see what's up. What's up is we have premium plans lurking just around the corner with a bunch of awesome new features. What we currently offer is going to stay free although I will be carrying a list of non-paying users around at Flash Gaming Summit in March so I can tell my real friends apart from the rest of you!

For our indie developers we have a plan that'll probably be called Plus, Indie, or something else entirely which introduces some cool new stuff - an API for exporting your data or building great tools on like Chrissie Ellis's Unity3d Heatmap emitter which lets you import heatmap data directly into the editor, this new API is versatile enough that it could power our entire dashboard, and that's something we're looking at doing to reduce our code size and of course it's going to be awesome seeing what people build on top of this - mobile and tablet alternatives to our dashboard would be great to see, I'd certainly pay for it. This plan also includes stuff like chart notes, chart comparisons so you can look at the same data over different dates or levels, daily heatmaps, logging exceptions that occur in your games, and anything else we can think of that we're missing.

For the bigger developers we have a powerful new offering about to launch - non-aggregated events that can be used for great stuff like cohort analysis, the paths players take through games, virality, revenue, and more. Most of you probably won't get to play with this but it's super cool, if you have an on-going investment and revenue stream in your game (aka social and bigger mobile games) you're going to love it.

We also began exploring building others' platforms into Playtomic this year starting with our friends at Parse who have a great, complimentary platform that lets you have a custom database you can do whatever you want with in your game - read, write, update, delete data. The potential applications there for games are crazy awesome ... storing replays, inventories, persistant worlds etc.

Playtomic got an office this year as well. Up until July the 'office' was my spare bedroom in a tiny little house in Nicaragua. The commute now is a little bit further - 7 blocks instead of 7 metres - and it still feels funny getting up every morning to go to work!

The internet has exploded a few times this year ... Adobe ditched the Flash Player on mobile which is overall a great move for Flash game developers. Online privacy became a bigger issue than ever, and Do Not Track came into existence, way back in February I posted my thoughts on it and for a while now we've had full support for the initiative although we've always taken online privacy very seriously.

We also saw some absolutely amazing new games this year. Personal favorite is of course Kingdom Rush from our friends at Ironhide Games. They just launched their iPad version so show them some love, it's a terrific game. Other awesome new titles that caught my eye include Raze 2, Burrito Bison, Rebuild and Rebuild 2 and so many more.

So, thank you all for a great year and for helping Playtomic become and grow into what it is today, for reporting bugs or being patient while they were addressed, for opening up Playtomic to platforms we didn't support, and for making the game development community so much fun to work in and with.

When we moved down here to Argentina I discovered that the developers Ironhide Games are just a few hundred miles away in Uruguay, so over the weekend other Ben and I jumped on a boat and cruised over for a visit.

This is us at some cool little bar near their office. We had a great night, although people down here like to start and finish ridiculously late ... we turned in a bit after 1am, which is when Uruguayans and Argentinians start!

This is me playing the very soon to be released iPad version of Kingdom Rush upside down, it's coming out hopefully this month and it's even better than the Flash version because tablets are so much fun to play on. Any game developer who wants to see just how much polish and attention to detail they need to put in a game should check it out, this is one of the benchmarks you should be measuring yourself against.

Today we went to a buffet parilla restaurant called Siga la Vaca, a parilla is an Argentinian excuse to eat several pounds of grilled animals.

From left to right ... Ben who does HTML5 and biz dev and stuff, Javier whose work includes the iOS and Android APIs and backend stuff like the heatmap archiving software, me, and Matias who works on our iOS and C++ stuff. Behind us is some guy and behind us on the right is a waitress.

Get it from the Asset Store!

Our level metrics (available in all apis) are a very simple, very powerful tool you can quickly use to find where players fall out of love with your game. They allow you to track a variety of information across levels and you can use that individually or cross-reference them to pinpoint problems.

Here are four simple strategies you can use in your game to spot the problems that are going to hurt its ratings in iTunes or Android App Stores, Kongregate or anywhere else.

Track how many people begin each level

This is probably the most valuable thing you can track to improve your engagement. By just logging when each person begins a level you can see two things:

1) how far into your game people are playing.

2) how your traffic drops off across your levels. This is inevitable, 100% of people are not going to play your game all the way through. But it's really easy to make a game where you lose 80% of players in the first few levels.

AS3: Log.LevelCounterMetric("started", levelnameornumber);

AS2: Playtomic.Log.LevelCounterMetric("started", levelnameornumber);

iOS: [[Playtomic Log] levelCounterMetricName:@"started" andLevel: name andUnique:NO]; 

iOS: [[Playtomic Log] levelCounterMetricName:@"started" andLevelNumber: number andUnique:NO]; 

Android: Playtomic.Log().levelCounterMetric("started",  "name", false);

Android: Playtomic.Log().levelCounterMetric("started", levelnumber, false);

C++: Log()->LevelCounterMetric("started", levelnumber);

C++: Log()->LevelCounterMetric("started", levelname);

Unity3d: Playtomic.Log.LevelCounterMetric("started", levelnumber);

Unity3d: Playtomic.Log.LevelCounterMetric("started", levelname);

HTML5: Playtomic.Log.LevelCounterMetric("started", levelnameornumber);

When you're logging this stuff you can easily see it visualized in a chart that will tell you straight away your game is a trainwreck on levels x, y and z.

Track achievements

If your game hands out bronze, silver and golds or three stars or whatever rating system that judges how the player completed each level, use a Ranged level metric to track each value. That way when you look at the "started" graph and see you lost 50% of players on level 4 (happened to me once) you can come over to this chart and start looking at how the players are actually performing - you lost 50% of players, and 90% of players couldn't get gold!

(couldn't find exactly the image I wanted for this one .... haven't made any games in too long)

AS3: Log.LevelRangedMetric("achievement", levelnameornumber, achievement);

AS2: Playtomic.Log.LevelRangedMetric("achievement", levelnameornumber, achievement);

iOS: [[Playtomic Log] levelRangedMetricName:@"achievement" andLevel: name 

            andTrackValue: achievement andUnique:NO]; 

iOS: [[Playtomic Log] levelRangedMetricName:@"achievement" andLevelNumber: number 

           andTrackValue: achievement andUnique:NO]; 

Android: Playtomic.Log().levelRangedMetric("achievement",  "name", achievement, false);

Android: Playtomic.Log().levelRangedMetric("achievement", levelnumber, achievement, false);

C++: Log()->LevelRangedMetric("achievement", levelnumber, achievement);

C++: Log()->LevelRangedMetric("achievement", levelname, achievement);

Unity3d: Playtomic.Log.LevelRangedMetric("achievement", levelnumber, achievement);

Unity3d: Playtomic.Log.LevelRangedMetric("achievement", levelname, achievement);

HTML5: Playtomic.Log.LevelRangedMetric("achievement", levelnameornumber, achievement);

Track performance

The Average level metric is used to track the average, minimum and maximum of whatever event. For games that use time, clicks, moves or some other measurement to to judge the player's quality you can use the Average level metric to help interpret those dropoffs, for instance in a racing game maybe track 3 actually takes 3 minutes when you guessed it should take 2, most people are failing and abandoning your game. Or one of those puzzle games where you have to move from x to y in the fewest moves, it's really easy to get these things wrong and create a situation where we think the level and the information flow and everything else are aligned just right, and players think you're a jerk.

AS3: Log.LevelAverageMetric("moves", levelnameornumber, moves);

AS2: Playtomic.Log.LevelAverageMetric("moves", levelnameornumber, moves);

iOS: [[Playtomic Log] levelAverageMetricName:@"moves" andLevel: levelname 

            andValue: moves andUnique:NO]; 

iOS: [[Playtomic Log] levelAverageMetricName:@"moves" andLevelNumber: levelnumber 

            andValue: moves andUnique:NO]; 

Android: Playtomic.Log().levelAverageMetric("moves",  "name", moves, false);

Android: Playtomic.Log().levelAverageMetric("moves", levelnumber, moves, false);

C++: Log()->LevelAverageMetric("moves", levelnumber, moves);

C++: Log()->LevelAverageMetric("moves", levelname, moves);

Unity3d: Playtomic.Log.LevelAverageMetric("moves", levelnumber, moves);

Unity3d: Playtomic.Log.LevelAverageMetric("moves", levelname, moves);

HTML5: Playtomic.Log.LevelAverageMetric("moves", levelnameornumber, moves);

With this you can see first hand that players are requiring more time, more moves, more clicks, more energy to complete your levels than you expected, and than they want to give you.

Heatmaps

Heatmaps remain the most awesome invention in the world - they can reveal so much data in such a stunning way. They're not universally applicable to all games but if your game suits them, you're going to love them. You can use them to track things like where people die, where they explore, where they click, if they collect powerups, if they collect necessary items to complete a level. You can be hugely imaginative with these things, anything that has an x and a y (and with a z you just have to look top down ... for now) is a candidate for a heatmap. Heatmaps are also super easy to use. This is using a special exporter to load them into the Unity3d editor, more news on that later.

AS3: Log.Heatmap("level1died", "level1", x, y);

AS2: Playtomic.Log.Heatmap("level1died", "level1", x, y);

iOS: [[Playtomic Log] heatMapName:@"level1died" andGroup: @"level1" andX: x andY: y];

Android: Playtomic.Log().heatMap("level1died",  "level1", x, y);

C++: Log()->Heatmap("level1died", "level1", x, y);

Unity3d: Playtomic.Log.Heatmap("level1died", "level1", x, y);

HTML5: Playtomic.Log.Heatmap("level1died", "level1", x, y);

These 4 metrics can improve your engagement and make your game more fun than alcohol.

And they're all available for free.

API docs: Android, iOS, C++, Unity3d, HTML5, AS3, AS2

If you're making C++ or iPhone games then check them out. Feedback is always welcome, you can email Matias at any time at matias@ this website.

For Android developers you can email Javier at any time at javier@this website.

Android API / Github

C++ API / Github

This puts us up to 7 different API versions, with a couple more on the way for Windows Phone and C# guys. Words cannot express how fun it is working in 2, 3, 4, and on some magical days 5 IDEs and languages at the same time!

If we're missing support for what you build games with then let us know and we'll see if there's space for it on our roadmap.

Today Adobe announced they were ditching Flash on mobile which really only matters to the game developers making the thousands of Flash games out there. Adobe abandoning mobile Flash is a good thing for casual game developers, Flash on mobile ultimately meant continuing the portal model of publishing Flash casual games - someone else handles distribution, getting you players, and to a large extent monetization of the players. It's the easy path and one favored by a lot of developers who just want to have fun, make games, and hopefully not starve too much along the way.

The portal method can work well when it's pulled off properly like Kongregate's Android app which makes tons of great Flash games available on your tablet or phone but this model is not being widely adopted, the monetization efforts on them are limited and the games are typically just switched to touch interfaces rather than taking advantage of the benefits mobile platforms have to offer.

This move forces us to adopt the app model of games, and that's a good move for game developers for a bunch of reasons.

Monetization

The biggest reason is monetization on mobile favors the developer. Flash casual games are a curious model, for those of you who aren't familiar with it developers traditionally sell licensing rights and launch their games with someone like Armor Games branding and support, publishing first on that website and then having the game spread virally to potentially thousands and thousands of different websites where they can get a bajillion plays. The developer essentially is paid in advance for how much traffic the licensor thinks they're going to get, the majority of the sales go through Flash Game License where the companies and websites licensing games bid on them and if you're lucky there's enough competition to drive the amount you're offered high into the thousands or tens of thousands of dollars. There is essentially no concept of ARPU because on-going monetization is usually nothing or just advertising while the game downloads through companies like Mochi Media. In a very small number of cases games launch with virtual goods through Mochi and GamerSafe but these games are a tiny minority of the market.

Branding

Creating your own personal brand is also more viable because you're not selling the rights to promote a game portal which in almost all cases is going to significantly overshadow your own branding. The players associate you with your game exclusively unless you work through a publisher.

Owning the user

Outside of licensing you are able to 'own the user' which is virtually impossible when you are selling the licensing - companies like Armor, Addicting, Kongregate etc don't want their users being poached by whatever game dev for a variety of reasons; age and privacy concerns, security, and generally not wanting to fragment their user base by having their website logins + a different 3rd party login inside whatever games.

Keep developing on Flash

The most important part of Adobe's decision is it doesn't kill Flash as a development environment - you can still develop in ActionScript and on Flash, which is a ridiculously easy, powerful and creative platform to work on, and then use Adobe's AIR stuff to get your game packaged into a native application. AIR is really starting to come together well with high performance and their next release includes a thing they call Native Extensions which allows you to tie into 3rd party services available on native platforms which is great, it means you can use the iPhone version of Playtomic for instance but also stuff like Game Center from Apple.

The catch

The only real negative is on mobile the onus is on you to get traffic to your game and make them fall in love with your game (by using our game analytics of course!). But this is not necessarily a bad thing - it forces developers to get more serious about the business side of their games and if you can do it there is massive earning capabilities which just blow Flash away - it's futile trying to measure revenue per user on casual Flash games and it's standard outside of them, and if you can figure out how to get people to play your game you're going to be in a much stronger position than making an "Android Flash" version of your game hosted on whatever website.

This is essentially the kick in the pants Flash game developers need to start taking mobile seriously and look beyond the easy 'money now, nothing later' model of licensing which is so convenient and for experienced developers has limited risk for web based games.

Additions:

• renowned Flasher Grant Skinner has a nice post mortem on the mobile Flash Player.

• ZDNet has an interesting look at what this means long term for Flash's future

On the right you can see indisputable proof that I was working at least when I stopped to take the photo. On the left is Matias' desk and monitors and in the middle left is Javier's. We each have two chairs because that's how we roll.

Javier and Matias

I have 2 new developers down here in Buenos Aires with me! Javier is an iOS and .NET developer who's just done an awesome new version of the iOS api, and Matias is a C++ and iOS developer who's currently putting together a C++ version and then taking over the iOS one. It's great to have them on board and it's already making a huge difference because I can focus on specific things instead of having to worry about everything.

The Bug Bounty

The bug bounty I ran on the forums the other week was a huge success, a whole lot of problems and bugs and stuff were fixed over a grueling week and a bit. There's a small handful of things that are still being sorted out (engagement reports for instance) but they are moving up my to do list.

New Game Conf

On the 1st and 2nd of November I'm going to be in San Francisco at the New Game Conf which is an HTML5 game developer conference, hopefully some familiar faces will be there but if not I'll just make new, better friends.

Some dashboard updates

I got rid of the old edit custom metrics, edit level metrics and edit heatmaps pages and rolled them into a single, more efficient and easier to use Metrics & Heatmaps page. It includes filtering and showing displayed, hidden or all metrics which will help a lot managing your stuff if your game has a lot of items.

It also reveals the "track name" for your metrics and groups. This is how our servers see your metric, in general it's going to just be a lowercase version of what you're using, but in some cases it might vary if you're using certain characters.

It also separates the heatmap images, more news on that soon.

For most of us this is going to be late but when you add a game for the very first time it's going to email you your credentials in case you click away from the api settings' page and don't know how to find it again.

Ben

Today we're tracking thousands of games by thousands of developers across 5 popular casual gaming platforms, and we are now consistently passing the big billion mark on daily events! To put 'events' into perspective, it's any piece of data coming from some guy or girl playing a game. It doesn't include the non-analytics services we provide like leaderboards, player levels, geoip lookups or GameVars, which incidentally are also growing like crazy too - we just recently outgrew the shared database hosting at MongoHQ and had to get upgraded to a dedicated plan!

My strategy for handling all of this data has evolved a lot over the last 2 years. A lot of ideas seemed great at the time but ultimately every little victory was short lived. At each step though the processing got stronger overall, the methodology more refined, and the throughput increased significantly.

All through this adventure the underlying platforms have stayed the same - C#, Windows Server, and Microsoft SQL Server. These are very forgiving platforms to work with. Our hosting has improved significantly along the way though, originally we started off borrowing space on a friend Brian's server (who later joined Playtomic as COO and President) to 6 servers of our own (or 7 if you count the new MongoHQ box).

The evolution of our log processing

In the beginning games would hit ASP.NET files there and write directly to the database. That did not last very long.

As we outgrew that I changed it so those ASP.NET files would write their own logs . My code sucks so I let the web server (IIS6 for the most part) take over writing the logs. This is far more efficient than me writing the logs, and by limiting the bog-ordinary web server log files to just the fields we need we were able to get tons of data logged efficiently. We're still actually using this approach today, also on IIS6 since it seems to be a lot more efficient at handling a ton of concurrent users - we typically have a quarter million people playing games at any given moment. This became the foundation of handling all of our requests - IIS generates lots and lots of log files for us, and we process them separately.

At that point in time we were not real time. I had things set up so the log files were set to be (I think) 20 megabytes each and when one was finished the server would send it off to be processed. This was good and bad - during peak times it would take about 10 minutes to complete a file but late at night and especially in our earlier days it could take up to an hour or two to generate that much data.

On the processing side we were handling all of the data with a single service that would unzip a file, scan through it line by line and create any missing metrics and whatnot it needed to, and update-or-insert the stats. Eventually three main issues surfaced with this overall approach:

1) Because the log completion and processing time was so variable users constantly asked when their data would arrive in reports.

2) Because the processing was a single, unthreaded service there were some horrible bottlenecks

3) If something went wrong (and a lot did!) the process would crash which could result in a large backlog

Solving these issues (except #3 I guess, which is probably always going to a PITA at our volume) took some time and a lot of work on the API servers, the log processing, the way we update the database, and of course going real time.

The API servers

On the API servers very little has changed since the early days once it was detached from the database and IIS took over handling the log files. The first step to speeding things up was making it handle incomplete log files. Eventually the zipping-and-sending became multithreaded because just FTPing the files became a bottleneck after a while. This is probably the oldest code in use at Playtomic.

Going real time

With some sleight of hand you can appear to be handling a ton of data in a near-instant fashion even if you're not actually. This addressed the #1 support issue I had back then ... people constantly asking me when their data would be in their reports, and the answer being a terrible "anywhere from 5 minutes to an hour". At its simplest going real time meant we needed a temporary location to house data before it was committed to the database, and reports would have to check two sources.

I looked at a lot of in memory databases and particularly liked the idea of Oracle's TimesTen database, but being bootstrapped and pretty poor at that time it was out of the question... it's ridiculously expensive. There was a nice C# embedded one somewhere whose name eludes me now but I had trouble with the implementation and couldn't get it to work for my use case. Eventually I settled on writing a very boring web service that basically just stores the data in the Application cache and has a very simple API for bulk-inserting, deleting, and querying.

Only a handful of gradual improvements have been necessary with that web service - bulk-adding data got optimized a little, and the way it was stored was adjusted a bit so that data could be bulk-removed more efficiently too. Eventually it was split off into multiple copies each dedicated to one section of data.

The log processing

The log processing has always been the hairiest part of Playtomic as the volume grew from megabytes an hour to up to around 500 megabytes a minute during the really busy times.

Originally it was just one service that did it all ... it would unzip a file, read through it and it would put the data in the database. The next step was making it do multiple files at once and aggregating the data but of course it still couldn't keep up after a while.

After that was multithreading. Taking the log processing software lasted a while longer but it just wasn't enough and of course it still had the problem where if the process crashed everything stopped which meant a backlog would occur unless it recovered in a timely fashion, and that meant users would be emailing because reports wouldn't be updating etc.

That concept evolved into what we have now - task-specific processes that collectively form a short but wide pipeline. Each API server:

1) receives requests

2) generates logs

3) has multiple threads that can zip up and send log files to the processing serve

On the processing server:

1) a process unzips files

2) another process scans through unzipped files and sends the relevant data off to each section

3) a section processor (eg custom metrics) does two things, thread 1 prepares them for the database (eg creating new custom metrics) and stores the data in our temporary database, and thread 2 shovels the prepped data into our real database and removes it from the temporary one.

The best part about this approach is nobody cares what anything else is doing. Either you have work in your queue or you do not, and because each process has some tight little purpose there's not a lot of ways any particular process can crash so the data is pretty much always flowing.

The most complicated part is of course the section processors, but each of the section processors is just a service wrapped over a common library ( I love C# generics ) which reduces them to maybe 100 independent lines of code. There's only two things that can really go wrong in them - connectivity issues (we didn't used to be able to afford a private network between our servers, and haven't yet rectified that now that we can), or malformed data. Malformed data at this point in the processing shouldn't be possible so we just discard it, and methods that have to complete just keep retrying till they do which does block the processing (a bad thing) but typically are very short lived.

Notifications were belatedly added to this software, it's something that was needed all along but was always de-prioritized since someone emails whenever reports stop updating anyway and there's always about 11 thousand other things that need doing. At this point all of the 'retrying' errors get logged and have a slightly longer delay between retries, and every 10th failure they shoot off an email. If the number of files waiting to be processed gets ugly it'll also shoot an email off since that's usually an indication of a problem too.

The database

Getting the data into the database has always been an adventure too. There was a time where it would literally do IF EXISTS(...) and then insert or update the row. That did not last for long but it was fun while it did. For every custom metric, level metric etc it would have to do a number of operations, first making sure the metric actually existed (mitigated by a cache of existing ones), then it would have to do the same for daily, monthly and total rows.

The next approach had some legs, it would create any missing data like custom metrics (still does now) and then it would create any missing stats rows too (no longer does this), so then getting all the data into the database was reduced to just a ton of UPDATEs instead of ghetto upserts. The processes would maintain their own caches of rows that existed so this approach lasted for a very long time. It was particularly great during backlogs since the more data it was committing the more the aggregation would reduce it down to fewer queries.

The current and semi-final approach uses SQL Server 2008's user defined table types and actually passes off C# DataTables to be MERGE'd into the SQL Server tables. These are basically a temporary table with a horribly rigid structure (you can't modify them) and also horribly, must be exactly mirrored in your C# DataTable with column ordering. They're not perfect but they're awesome and ridiculously fast.

The future

Whether the current architecture will survive another year is doubtful, but right now because everything is an independent step the throughput can be massively increased by breaking up a slow step into multiple queues that can reside on the same or other hardware which gives it a fighting chance as our load grows from a billion a day to some other ridiculous number.