Wednesday, April 22, 2015

Does coaching work? We spent $1,300 to find out.

In January, we launched the DotaCoach Progress Experiment to measure whether coaching actually makes students better players. We believed it did (strongly enough to start this very business), but we didn’t have proof. So we gave two months of free professional coaching to ten students and measured their progress. In total, we gave away 180 free lessons, at a total cost of $1,300 USD. Now that the results are in, we’d like to share them with you. Over the coming weeks, we'll be releasing a series of posts showing quantitative and qualitative data about our students and their progress.

Today we’re kicking things off with the most commonly-used way to measure skill, MMR. Although MMR is not a perfect measurement of skill, it's reasonably sound mathematically, it's the best single metric that we have, and it's proven to be quite accurate given enough games played. With that out of the way, let's take a look at the MMR graph.

 

Details
This graph shows how the MMR of students and controls changed over three and a half months leading up to the experiment. It then resets everyone to zero (at the “Coaching Begins” vertical line) and shows the same comparison over the course of the experiment. Prior to the experiment, both groups fluctuate but have no significant change in their MMR. Once the experiment starts, the students receiving lessons show clear improvements compared to the controls.

Students and controls 
Of our original 10 students, 3 dropped out within the first week or two due to real-life commitments. This left us with the 7 students you see in the graph above. To provide a basis for comparison, we also gathered data on 6 players who had applied to be a part of the experiment, but were not selected. These players are referred to as controls, or non-students. As you may have noticed, the controls actually dipped below zero slightly during the course of the experiment; we expect this was just a random anomaly, rather than them being heartbroken from not being chosen for the experiment.

MMR calculation 
We originally had students sign up for Dota 2 Toplist to track MMR throughout the experiment, but the site has not worked properly since mid-February. So we had to improvise a bit. We pulled all match data for every student and control for the last six months from the Dota 2 Match History API. We estimated their MMR by looking at ranked matches and assigning +25 for a win and -25 for a loss. This proved to be a sufficiently accurate predictor for players in the experiment’s range, 1,500-4,500 MMR. We also cross-verified our estimates by looking at the resulting MMR at different dates for which we had recorded MMRs; it was accurate within 2% for all students.

Note that the Dota 2 Match History API does not distinguish between solo and party ranked play, so this graph includes both. This is annoying for us and hard to adjust for since some people don’t play party at all and we don’t have data for controls. For our students it looks like about 75% of the change is coming from solo games, which conservatively works out to about a 350 solo MMR improvement on average.  

Lesson counts
One of the graph’s options allows you to see the number of lessons each student took. These range from 10 hours (Student #3) to 37 hours (Student #5) over the course of the two months. 

Analysis
As you can see, coaching clearly improves players. Looking at the MMR percentiles Valve published, students in our experiment leapfrogged about 10-20% of the Dota playerbase in just two months.
You may be thinking, “I’ve had my MMR swing by a few hundred at times, so what?” Great question! What makes these results significant is that the improvement happened consistently across several students. Normally when you see swings, people go up and down but they average around zero over time. But during the experiment, every student saw improvement, and the average improvement was well above zero. This indicates that the students are actually getting better; it is very unlikely that every student in the experiment happened to have a big MMR upswing during the course of the experiment. As you probably noticed, we added standard error bars to the graph to help show this. They give a quick, intuitive sense of the significance of these results, and show that the improvements are not merely coincidence or a temporary swing. (If anyone is interested, we can run the numbers for other statistical tests in future posts, or share anonymized data with interested parties.)

Conclusion
This data suggests that coaching is an effective way for players to improve their skills. An average improvement of 350 MMR in two months is a significant gain. For those who are looking to improve their skills, this experiment provides compelling evidence that coaching can be an effective way to improve. Whether it's from a professional like we offer at DotaCoach, or a friend who has offered their time and advice, we hope you'll give coaching a try.

One student commented thusly on what he took away from the experiment and these results: “get a mentor whether paid or unpaid, thank him, appreciate him, and next time u want to spend $10 on a hat - maybe invest in your gameplay and spend an hour with a coach.”



We’d love to hear what data/analysis you’re interested in for future posts. Let us know by commenting here, on Facebook, tweeting us, or sending a mail to contact@dotacoach.org and we’ll try to answer your question in future posts.



Tuesday, April 14, 2015

Coaching Sponsorship for Collegiate Team UBC

The UBC eSports Association is a non-profit student gaming club centralized around the University of British Columbia here in our home city of Vancouver. They have an amazing history in eSports and have helped incubate several star eSports players including EG's Aui_2000 (Canada's highest paid eSports player). With Aui_2000 leaving UBC to pursue Dota full-time, the UBC Dota 2 Team does not play at a professional level.

UBC competes in the Collegiate StarLeague (CSL), an intercollegiate gaming league open to colleges and universities across North America. Expectations for UBC in the CSL Playoffs were humble - they're currently seeded at 65th. When league representatives predicted the outcome for the UBC team, they first said "UBC may potentially take a game, but it would be quite the upset." When predicting the second round game the league reps confused UBC with the California giant UCB. But despite the nay-sayers, UBC rose to the challenge and now stands among the eight teams remaining in the CSL playoffs.

In early March, DotaCoach.org approached the UBC team and offered to pair their members with some of our professional Dota 2 coaching staff. The thinking was that the UBC team contained strong individual players but lacked experience as a competitive unit. DotaCoach.org recommended some of our most experienced coaches and entered into a sponsorship agreement which paired the UBC team with mentors from Team ChW

The coaches from ChW, kAitorA and assistant coach Rallphy, have over 20 years of competitive Dota experience between them and have competed for years in JDL Division 2. They are top players from Serbia who coach Dota professionally through DotaCoach.org to help support their dream of becoming professional gamers.

The team and the coaches met weekly over the past month. They play live games with the coaches commenting, they analyze recordings of skirmishes, and discuss strategies and ways to improve the team's decisions. Individuals from the team have had 1-on-1 sessions with the coaches, but overall the focus has been on building up the team's play.

UBC has what it takes to win CSL. They knew how to win long before we entered their lives - but the competition is getting fierce. There have already been some exciting games to avoid elimination; including a squeaky-close eighty-minute Game 2 win against the #11 seed, University of Massachusetts. Players on the UBC team agree that the coaching sessions have sharpened their edge and improved their chances. 

For us at DotaCoach.org, facilitating and sponsoring this mentorship feels good. We can help young local players grow through competition while helping team ChW grow as professional coaches. We're looking forward to more upsets from Team UBC!  Tune in to CSL on Twitch to watch with us.

If you have an amateur or collegiate team which would benefit from coaching, tweet us @DotaCoachDotOrg.

Thursday, April 9, 2015

DotaCoach Advertising Campaign

Hey Coaches!

To date, our students have rated their satisfaction with your lessons as 4.7 out of 5 stars. You're doing awesome, so we're doing what we can to inform players about the value that your coaching can bring them.

Today marks the first day of our advertising campaign targeting Dota players. Keep your eyes open for any of our 114 banners appearing on your favourite Dota sites!






Saturday, April 4, 2015

Wrapping up the Experiment

Earlier this year we announced the DotaCoach Progress Experiment. The idea was pretty simple: we felt coaching was a great way to improve your Dota skills, but we didn't have any proof. So we gave free coaching to nine students and measured their progress over eight weeks. Now that the experiment has concluded, we'll be publishing some data analysis on the results over the coming weeks.

We're interested to hear what type of data you would like to see on this. Obviously we'll be showing things like MMR, KDA, GPM, XPM, and similar stats and how they changed over time. We also have a control group that we tracked -- applicants to the experiment who were not selected; we'll be including data from these players as well (anonymized, of course). But we'd like to know if there's anything non-obvious you'd like us to take a look at. We have all the match data that Dotabuff and the Dota Match History API provide for each of the players, so there's potential to do some interesting things. For example, one thing we're looking at is specifically how GPM improved for people when playing carry heroes, since GPM is more correlated with playing well for those heroes than it is for support heroes. Another might be: for the players who improved the most, which stats correlated most highly with their improvement; i.e. can I become better quicker by focusing on objectives, or decision making, or is improvement in last hitting a better precursor to in-game succcess?

We will be in contact with the individual players to get some qualitative feedback to help answer these questions in addition to the standard quantitative analysis, so even some slightly fuzzier questions are fair game. For example, we've already asked them about what type of coaching (live game, replay analysis, skills training in private lobby, etc.) they thought had the biggest impact.

You can leave any thoughts/suggestions you might have in the comments here, or email us: contact@dotacoach.org. Looking forward to hearing from you!
-Jay