The PVP Guide of Statistics

Intro

Thank you to the 260 of you who contributed your data to our collective PVP stat guide. We will be examining it to derive sound, statistically driven conclusions.

I want to personally thank PKB, Caslandr, Leojms, Andy (FB) and Agent Slee (FB) amongst many others who have helped to conceptualize and crystalize this project.

Statistical Primer

Here are some basic stat primers. 

1. We build a model of the world. 
2. We test it with data.
3. We refine the model according to the result.
4. We use these results to predict outcomes.
5. Multi-variable regression is one way to test how closely the model resembles reality.

Nitty Gritty

1. We build a model of the world. Rating is a dependent variable because it depends on other variables. What affects ratings in Marvel world? Independent variables affect rating (we assume they are independent of rating, or that rating does not affect them). One such variable is Health; perhaps more health means higher rating. Other variables could be attack, defence, accuracy, evasion. Number of attacks perhaps influences rating. It could be 800, the intercept, or starting rating. Or perhaps we were wrong and a factor does not influence rating, (this is the null hypothesis).

http://i61.tinypic.com/vctaae.png

1. We test it (empirically) with data. This means we input your 260 observations into Excel, import into Stata, and run various tests on it.
2. We refine the model according to the result. This means looking at the first test results and proposing different tests to measure how accurate the model is, and what it tells us.
3. We use the results to predict outcomes in the world. We want to reach a stage where we can look at any player's statistics and decide: what rating can we expect them to get?  Taking all the data, which stat is most important?The billion dollar questions. 
4. Multi-variable regression is one way to test how closely the model resembles reality. The main way we do this is multivariable regression.
   1. P-value/significance: the lower the number (if below 5%, a good academic cutoff), then we can conclude the variable affects rating. In more technical terms, this is rejecting the single null hypothesis.
   2. Only if significance <5% do we consider magnitude/coefficient: the bigger the coefficient, (these are the beta_1,...,beta_6), the stronger the effect. Negative number means more of it lowers rating.

Real data time

Time for some real data. First, let us look at rating vs level. Isn't it super interesting? Note how level 260-299s have significantly lower ratings than others. This is probably because they are fighting Level 300s all the time and are at a significant statistical disadvantage.

Also note the sample is not a good sample: the average rating is 1164 with standard deviation 327. We expected it to be 800. Rating points, like energy-matter, are neither created nor destroyed. Since 1164>800, not enough low ranked agents are doing the survey. Perhaps agents who don't do the survey are also not reading the Wiki and thus perform poorly in PVP.

http://i62.tinypic.com/121e70w.png

Power curves

Okay look at these beautiful power curves! See how you measure against your fellow agents at your level.

http://i61.tinypic.com/2moeeyq.png

Matrix Scatter Plots

This one looks absolutely wacky but I promise you it's harmless as Luke Cage. It shows how any two statistics are distributed in 2 dimensions. ROW and COLUMN. For example, say you want to look at rating and attacks. Rating on the y-axis (1st ROW), attacks on the x-axis (2nd COLUMN), VOILA. The square shows rating and attacks.

Again! Rating and level. Rating (1st ROW), level (3rd COLUMN). In that little box is the exact same graph as the one above. Easy isn't it?

http://i58.tinypic.com/jjttas.jpg

The Regression Meat

Here's the full model regression. Note I included level in the model, because if not our results will be biased. Take a deep breath. Remember the two simple steps. P-value and magnitude. What can we observe?

Look at number of attacks. P-value is 0.000 (under 5%, good!) so attacks is a good predictor of rating. Now you ask by how much? Magnitude/coefficient: 0.380. Meaning on average, if we attack one more time, we expect the agent's rating to increase around 0.380. WOW. Better get attacking!

http://i60.tinypic.com/o0nfyt.png

Similarly Look at level. P value is 0.000 (good), so its a good predictor. Does it correlate with lower ratings? Yes. Remember the Level 260-299 drop. That's our sample bias. Does it cause lower ratings? No, but that's a technical discussion related to ELO rating systems and level groups.

TotalHP: P-value over 5%, not significant. Health seems to have no effect on rating, is it surprising? Atk: P-value 0.001 (good). Attack is a good predictor of rating. Quiz time 1: deduce the magnitude yourself. Quiz time 2: are defence, accuracy, and evasion significant?

Good model or not?

Look at the adjusted R-squared. (I will avoid being technical). This means that our model explains 42% of the variation in rating. Pretty good eh? 100% is maximum (won't happen because no model is perfect), and 0% means its a horrible model because it can't predict anything. More to come!

I appreciate your comments and suggestions for improvement. This should be a collaborative project

Part 2: Season 15

Take a look at part two : WIP. http://avengersalliance.wikia.com/wiki/User_blog:Meisterjohnny/The_2nd_PVP_Guide_of_Statistics