Wednesday, March 16, 2016

Ever coached an amateur team? Volunteer with the Learn Dota 2 League

The newly announced Learn Dota 2 League is a casual Dota 2 League run through the /r/learndota2 subreddit. It is a tournament for players who are new, looking to improve, and interested in participating in organised Dota at their skill level. All teams have an average MMR of under 3k and the maximum individual MMR for a player is 3.5k. The league currently consists of about 450 players with sign-ups closing April 1st.

DotaCoach.org is getting involved in the league to help players get more coaching and to create structured volunteer opportunities for coaches. Our goal is that this league can become a forum for coaches to practice their skills, to gain experience coaching organized teams, to compare the impacts of various coaching methods, and to encourage coaches to help each other improve.

If you're interested in working with a team in the league or just practicing individual coaching with players in the league, please register to coach.

Wednesday, March 9, 2016

Balance Patch for Reputation System

Our Reputation System for Coaches attempts to measure our confidence in the quality of each coach who creates a profile page on DotaCoach.org. This system is the backbone of our leaderboard and our matchmaker. Since launching last year, we have received a lot of feedback about our reputation system from both students and coaches. This post will explain three changes which we recently made based on this feedback.

How is reputation calculated?

In short, coaches earn or lose reputation whenever Dota Coach receives evidence showing how the coach performed. You can view a full list of reputation events and their associated values here.

This is the overview of the reputation for a top coach.


What is current reputation? Current reputation is the metric used to rank coaches on the leaderboard and used by our matchmaker. Until now, this is the formula we used to calculate a coach’s current reputation. In change #3, a new updated formula is given.

Current Reputation = Reputation from MMR + reputation from students coached within 70 days

We aspire to be an open platform where anybody with the skill and drive to coach professionally can become a top coach. To achieve this, we employ automated, evidence-based, transparent metrics for measuring our confidence in each coach. The current reputation score aims to make it so our leaderboard shows only the best and most active coaches on the site; and that newer coaches still have a chance of making it to the top.


Change #1 - “Coaching Experience” reputation events now factors in the cost of the lesson

Coaches previously received 100 reputation per hour for lessons sold. With this change, they now receive 15 reputation per dollar per hour.  The old system gave an advantage to coaches with low prices and in rare cases coaches lowered their price in order to artificially inflate their reputation.

Impact: We have recalculated reputation for all lessons going back to July, 2015 to reflect the change. This went live on March 2nd. Coaches who have charged less than $6/hr might see a slight decline in their reputation score, but all other coaches should see an increase in overall reputation.

Change #2 - “Current Reputation” value cannot decay below 5,000

Each day at midnight, we calculate the coaches' Current Reputation based on the last 70 days. If the coach doesn’t remain active, their current reputation will appear to “decay.” Extreme levels of decay can move top ranked coaches down to equal ranks with a new coach with zero experience (eg. Kumbomaman has 275 hours coached but less have than 2,000 reputation). Surely this is unfair and the system should have higher confidence in the abilities of these highly experienced coaches.

This change makes it so a coach’s current reputation doesn’t decay until a coach gains more than 5,000 reputation and then it can only decay to a floor of 5,000. Currently, that ranks you 17th on the leaderboard. If you don't see your name on the leaderboard, you have to coach one hour in the last 70 days to be listed (no change there).

Impact: No impact to the top 15 coaches, but a few of the old top coaches have re-appeared in the middle of the leaderboard.

Change #3 - “Current Reputation” uses clearer and more understandable decay window

The algorithm we were using to calculate current reputation is a bit mysterious and chaotic. It has resulted in radical swings of reputation with no transparency into the cause of the swings. The resulting confusion undermines the system.

We are moving to a simpler algorithm which is more stable and more transparent.


Current Reputation = Reputation up to 5,000 + additional reputation earned within 70 days

Example: This coach has a life-time reputation of about 50,000. The first 5,000 reputation is not subject to decay. The reputation after 5,000 has decayed and is not counted in current reputation. The reputation delta shaded yellow represents the coach's "additional reputation earned within 70 days." This coach has a current reputation of about 22,000.



Impact:  This has caused a reshuffling in the top coaches list. Top coaches may see a reduction in their overall reputation. 

Join the Discussion

The factors which make a player into a good coach are varied and complex. Expressing this multi-dimensional concept into a number is hard.  Just as Valve’s MMR system allows them to build technologies like a matchmaker and a leaderboard for Dota, so does the reputation system enable our matchmaker and leaderboard for DotaCoach.org.

Just like players feel that MMR often underrepresents their true skill, our reputation system can leave coaches feeling under-represented. Send us any thoughts or suggestions you have on how this can be improved @DotaCoachDotOrg or at contact@dotacoach.org.

Future Changes

A good measure of a coach’s quality is how impactful he is on his student’s MMR. We will soon start tracking the MMR for each of a coach’s students to attempt to capture evidence of the coach’s impact.

Current reputation isn’t the only interesting attribute about coaches. We will be adding a new page to capture the all-time great coaches.