Since launching DotaCoach last month, we've conducted over 50 lessons through the site. Reviews from students have been overwhelmingly positive. But while we love to hear that people enjoy using the site, that isn't enough for us. We want to know whether we're actually helping to make people better Dota players.
This is a difficult but important question to answer. We've considered a few different ways of approaching the question directly as part of the service, for example asking students to track their own MMR, or building a bot to track it for them. But they all suffered from being invasive, unreliable, or both.
We were mulling this over during the holidays, when Kenn had a great idea: Why don't we conduct a full-blown experiment to measure progress? We'll give away free coaching to a few interested participants, measure their progress over time, and gather their feedback. They'll get free lessons from some of our top coaches, and we can measure whether (and how) coaching is effective. We both loved this idea, and so the first ever DotaCoach Progress Experiment was born.
Here's how it worksThroughout this week we'll be collecting sign-ups from any student interested in participating. A week from today, on Jan 19th, we'll select the participants and begin the experiment. We'll choose 5-10 students, aiming to select a reasonable distribution of skill, experience, and play style.
Every week, each student will receive coaching and play ranked games to help track their progress. To ensure a reasonable sample size, we'd like each student to get about three lessons per week. While this may not be possible every week, we want to find students with relatively stable schedules to help minimize noise and increase the likelihood that they'll be able to participate for the full duration of the experiment.
For quantitative measurement, we'll look at everything available -- MMR, win rate, GPM, XPM, K/D/A, etc. MMR will obviously be the most important measure; while we know not everyone is a huge fan of it, we're fairly confident that it's the best way, on average, to measure a player's skill -- certainly better than anything else currently available.
Every two weeks, we'll gather a few pieces of qualitative feedback from the students to help supplement and give context to the quantitative measurements.
The coachesOur coaches will be integral to this experiment. We want to use a mix of coaching assignments in the experiment: some people will be assigned to a single coach, while others will use our regular matchmaking queue to find the best coach available at their convenience. This should give insight into the importance of things like coach consistency, long-term lesson plans, etc. We'll pay our coaches the same as we would for regular, paid lessons.
While all of our approved coaches will be eligible to participate, particularly for those students using the matchmaking queue, we wanted to highlight a few of the coaches who will be participating in the experiment:
Our highest rated coach to-date, Bot-Easy has been playing Dota for 9 years, including 2 years as a pro in WC3 Dota.
SevenThings started with WC3 Dota, before moving on to HoN and now Dota 2. He has a passion for coaching, having coached for years before joining DotaCoach. He recently won the inaugural Red Bull LAN Seattle, defeating pro Jimmy "DeMoN" Ho in the process.
Masakary is a 22 year-old student from Romania. His team Balkan Bears Corleone (BBC) is trying to break into the pro scene and has recently defeated pro teams including Moscow 5 Int. In the last few months he has been ranked as high as #12 on the Dota 2 EU Leaderboards. Masakary has been playing Dota in its various forms for 8 years and loves coaching others.
How to sign up
To apply, ensure you meet at least the required criteria below,
and then fill out this form.
- You speak English fluently. English does not need to be your first language, but all of our coaching is done in English, so you should be comfortable enough with spoken English that this will not impair your learning.
- Your MMR is relatively stable. Your ranked winrate should be close to 50%. If it isn't, your MMR might still be adjusting to match your skill level and it would skew the experiment.
- You must be willing to do at least three coaching sessions per week. Occasional scheduling issues aside, we expect you to be able to maintain this for almost every week, otherwise the sample size will be too small and the results will not be nearly as useful. If you want to do more than three, that's great -- we'll pay for up to five sessions per week.
- You must sign up for Dotabuff and Dota 2 Toplist if you haven't already. This helps us to track stats and MMR.
Nice to haves:
- Your MMR is between 1000 and 4500. Most of our students fall into this range, and we want our test sample to be relevant to our users.
- You play at 10+ games of Dota per week. We want people who are already playing consistently, because we want people who will continue to play consistently during the experiment.
- You already track your MMR via Dota 2 Toplist or some other MMR tracker.
That's all for now. We're looking forward to reading your applications! As always, let us know if you have any questions or comments.
-Jay and Kenn