Nginx + Sinatra + MongoDB

I created a document to help explain the how and why of my web app setup.

Document

I use this in production, so I guess you could say that I eat my own dog food on this one.

Gotta love optParse

I use a very simple deployment system which goes like this:

* export code base at HEAD
* tar export
* scp tar to host server(s)
* create directory for this version of the payload
* unlink 'current' directory ( /var/www/[project]/current )
* unspool tarball into newly created revision directory
* link 'current' to newly created directory

In this way I can roll the entire code base back using a simple unlink/link-to-old-build-id command. This keeps my environments nice and tidy, and safe from evil things like version conflicts.

To this day I don't know why anyone would actually think that 'svn update' is an OK thing to do in a live production environment. Ehh...guess that's just me being all anal again.

My favorite part about this is the deployment script, dead simple, and I did the opsParse thing to remind me of how it all works:

krogebry@krogebry-desktop:~/aws/deployment$ ./deploy.rb -h
Usage: deploy.rb --project [project] --environment [stage|prod] --verbose
-v, --verbose Run verbosely
-p, --project PROJECT Project name
-e, --environment ENVIRONMENT Environment name ( stage | prod )
-h, --help Show this message

EBS logging VS Scribe...

I was running some numbers trying to figure out which would be the better solution between using EBS volumes on the web front ends for logging versus using something like scribe. In either case the log data would end up in a database for analytics.

In my mind the choice is pretty dead simple, however, I've recently run into a case where and admin had put together the most bizarre little hack for dealing with logging data. They were writing the log data to an attached volume, then reading the data off the volume with a series of nearly indecipherable spaghetti code written entirely in bash. This meant that the web front end were horribly bloated with all of this extra work, not to mention the sheer fragility of it all.

At any rate, I was thinking about how much this is costing them to do as opposed to doing something more elegant like using Scribe...

Assuming 10 web servers taking an average rate of 5 hits per second, and assuming that each hit is logged to an EBS volume, then read from the same volume later on into a database:

5 hps * 10 servers = 50 hps
50 writes and 50 reads = 100 total IO operations per second.
360,000 ops/hr = 8,640,000 ops/day

Amazon charges $0.10 per 1M operations, so that works out to $0.664 / day = $25.92 per month.

So, at the end of the day they end up spending a very small amount for this. Even if you piled the expense of the web server instances themselves it probably won't add up to much more then $50/month. That's not bad at all, but it's still %50/month they don't have to be spending when they could be getting a better service for free. Call me overly pedantic, but I just hate spending money I don't have to, especially when I have a lot riding on a newly funded startup. Every penny counts right?

New updates.

I just pushed a new build up to production. This build shows off some new features, and begins to show some of the onclick actions for the name plates.

More tanking effectiveness.

Now with event goodness!

The little red x is a death event. Eventually you'll be able to hover over it and see who died and when.

The blue horizontal lines are bone spikes. I haven't quite worked out all of the details on this yet, but it's getting closer!

RR: Tanking effectiveness

Tanking Effectiveness is basically a comparison between damage input on the tanks, and healing output on the tanks from the healers.

There are 20 ticks on the page that represent the 20 samples taken from this encounter. The white dots on the top show the total amount of healing done to the tanks by the dedicated healers. The red dots on the bottom show the amount of damage taken by the tanks from all sources.

There are 3 things missing from this graph that I am working on now:
* Connecting the white dots together.
* Connecting the red dots together.
* Build the difference line.

The difference line shows the difference between the healing output and the tanking input. The thinking here is that you would use the difference line to get a sense of how effective your healers and tanks are. If the line is above the center line, then the output is being mitigated by the healer output ( which is good ), if it's below the line, then the damage input isn't being mitigated by the healing output ( bad ).

Fun with WoWCombatLog and mongoDB

I wrote a very basic parsing utility to take data from a WoWCombatLog.txt file and put it into a mongoDB document. Now I can use mapreduce to create a summary report of the data:

res.drop()
function fMap(){
emit( this.sourceName, {
cnt: 1,
sum: this.amount,
cntNormal: (this.critical == false ? 1 : 0),
sumNormal: (this.critical == false ? this.amount : 0),
cntCritical: (this.critical == true ? 1 : 0),
sumCritical: (this.critical == true ? this.amount : 0)
});
}

function fReduce( key,values ){
var r = { cnt: 0, sum: 0, cntNormal: 0, sumNormal: 0, cntCritical: 0, sumCritical: 0};
for ( var i=0; i r.cnt += values[i].cnt;
r.sum += values[i].sum;

r.cntNormal += values[i].cntNormal;
r.sumNormal += values[i].sumNormal;

r.cntCritical += values[i].cntCritical;
r.sumCritical += values[i].sumCritical;
}
return r;
}

function fFinalize( sourceName,stats ){
stats.avg = ( stats.sum / stats.cnt );
stats.avgNormal = ( stats.sumNormal / stats.cntNormal );
stats.avgCritical = ( stats.sumCritical / stats.cntCritical );
return stats;
}

res = db.raid_data.mapReduce( fMap,fReduce,{
query: { raid_id: "4b7c64da756f4e5799048fca", eventType: /_DAMAGE$/, sourceGUID: /^0x0600/ },
sort: "sourceName",
finalize: fFinalize
});
res.find();

Which produces:

{ "_id" : "Badjoojoo", "value" : { "cnt" : 9, "sum" : 66260, "cntNormal" : 9, "sumNormal" : 66260, "cntCritical" : 0, "sumCritical" : 0, "avg" : 7362.222222222223, "avgNormal" : 7362.222222222223, "avgCritical" : NaN } }
{ "_id" : "Endofire", "value" : { "cnt" : 2618, "sum" : 11396761, "cntNormal" : 1478, "sumNormal" : 3797148, "cntCritical" : 1140, "sumCritical" : 7599613, "avg" : 4353.231856378915, "avgNormal" : 2569.1123139377537, "avgCritical" : 6666.327192982456 } }
{ "_id" : "Guslado", "value" : { "cnt" : 5146, "sum" : 13370673, "cntNormal" : 2921, "sumNormal" : 3690768, "cntCritical" : 2225, "sumCritical" : 9679905, "avg" : 2598.2652545666538, "avgNormal" : 1263.5289284491612, "avgCritical" : 4350.519101123596 } }
{ "_id" : "Jenix", "value" : { "cnt" : 3705, "sum" : 7205291, "cntNormal" : 2376, "sumNormal" : 2719061, "cntCritical" : 1329, "sumCritical" : 4486230, "avg" : 1944.7479082321188, "avgNormal" : 1144.3859427609427, "avgCritical" : 3375.6433408577877 } }
{ "_id" : "Kaymaira", "value" : { "cnt" : 16, "sum" : 91501, "cntNormal" : 13, "sumNormal" : 86764, "cntCritical" : 3, "sumCritical" : 4737, "avg" : 5718.8125, "avgNormal" : 6674.153846153846, "avgCritical" : 1579 } }
{ "_id" : "Mistyayn", "value" : { "cnt" : 2545, "sum" : 10732921, "cntNormal" : 1597, "sumNormal" : 4329100, "cntCritical" : 948, "sumCritical" : 6403821, "avg" : 4217.257760314342, "avgNormal" : 2710.7701941139635, "avgCritical" : 6755.085443037975 } }
{ "_id" : "Rhyea", "value" : { "cnt" : 9568, "sum" : 7576176, "cntNormal" : 7236, "sumNormal" : 5245271, "cntCritical" : 2332, "sumCritical" : 2330905, "avg" : 791.8244147157191, "avgNormal" : 724.8854339414041, "avgCritical" : 999.5304459691253 } }
{ "_id" : "Silverrend", "value" : { "cnt" : 7351, "sum" : 13101369, "cntNormal" : 4238, "sumNormal" : 3451620, "cntCritical" : 3113, "sumCritical" : 9649749, "avg" : 1782.256699768739, "avgNormal" : 814.4454931571496, "avgCritical" : 3099.8230003212334 } }
{ "_id" : "Stabback", "value" : { "cnt" : 7685, "sum" : 12463405, "cntNormal" : 3117, "sumNormal" : 3976589, "cntCritical" : 4568, "sumCritical" : 8486816, "avg" : 1621.783344176968, "avgNormal" : 1275.7744626243182, "avgCritical" : 1857.8844133099824 } }
{ "_id" : "Zuulbash", "value" : { "cnt" : 4501, "sum" : 6388398, "cntNormal" : 3373, "sumNormal" : 3449622, "cntCritical" : 1128, "sumCritical" : 2938776, "avg" : 1419.3285936458565, "avgNormal" : 1022.7162763118886, "avgCritical" : 2605.2978723404253 } }

Awesome. Basically I'm combining a view and a stored procedure, but unlike your standard SQL server, I get to use javascript for the processing language.