What is the value of open source engagement?
Evaluating the effectiveness of open source engagement, whether you are releasing your own code or contributing to other peoples’ projects, is a fairly ambiguous undertaking.
Traditional methods of evaluation of “engagement success” are tied to easily quantifiable events. In sports, athletes are evaluated by game play rules and physical performance such as how fast, how high, and how many. In sales, you might be measured on customer conversions and monetary sales figures.
Open source community evaluation is less tied to tangible artifacts and outcomes. “Success” in open source communities involves three key areas:
- Individual influence (how much am I doing?)
- Organization influence (how important are we to the community?)
- Popularity (does anyone care about what you are doing?)
This is very similar to research & development. In R&D, one might evaluate individual influence through publications, organizational influence through patents and sales of developed products (indirect sales), and popularity through citations and market share.
This article will specifically focus on organizational influence and proposes using commits as a heuristic to gauge an organization’s engagement level with an open source project. To clarify, while revenue is a driving factor in our economy and for many businesses, the focus of this article is not on revenue. The focus is on values; revenue is an outcome.
How do you measure the value of open source engagement?
The current paradigm for open source community metrics is heavily influenced by the limited context of the individuals involved in those communities. Specifically, people with a narrow background in computer science and IT have little training in social science and assume system metrics are sufficient for measuring humans. In systems, precision is important for identifying performance bottlenecks, uptime service agreements, and capacity planning.
Traditionally the tech industry has applied this context to human metrics (dubbed here the “Human-System Metric Paradigm”), based on the assumption that the value of workers is the volume of the work they produce. Therefore, metrics must measure the volume of work such as lines of code or number of tickets closed with a very high degree of precision. In addition, worker compensation and rank are directly tied to this reported volume of work.
Quantatitive volume-of-work metrics don’t really tell us anything about the value or nature of the work. They can also result in behavior modification when the competitive sales paradigm is applied and the volume measured is tied to tangible rewards.
Have you ever modified your work activity to “pad” a volume-of-work metric?
Humans cannot be measured by human-system metrics because humans are too ambiguous. Metrics should be question-driven, based on your goals and values and how you want to achieve them, rather than purely data-driven. In fact, no data collection should be embarked upon without proper introspection and reflection on how the metrics inform your larger goals (and if they are still relevant).
How do you break out of the human-system metrics paradigm?
- Ask better questions about open source engagement
- Seek meaningful answers that drive strategic discussions (not just dashboards)
Github: A Social network platform for code
User profile example on Github
Many open source communities and organizations host their projects through Github (not to be confused with git, the command line tool that Github provides an interface to). Github is a treasure trove of public activity just waiting to be analyzed, yet Github themselves fail to offer anything insightful in this area! Every year Github releases the State of the Octoverse report, which is a prime example of data-driven metrics. The report is chock-full of pretty plots and counts of things but is otherwise very disconnected and incoherent, failing to tell any kind of story or share anything deep or meaningful.
Github’s The State of the Octoverse report
Every repository’s page contains basic counts of things like commits and contributors. There’s even an insights page for even more (but not particularly insightful) counts of things. One cannot formulate a meaningful sentence using these types of human-system metrics.
A repository page featuring numbers of commits and contributors. Can you make a meaningful sentence out of this?
The insights page for a Github repository. What meaningful questions does (or even can) this answer?
Commits as a measure of influence
Commits are a historical record of attributed code changes. The current popular measure for “ranking” influence is by counting up the number of commits and assigning them to an author or a company based on identifying a set of authors.
Common question: Who has the most commits? Better question: How significant is an organization’s influence?
Identifying organization affiliation
The most common method for company identification is either through the domain name of the email address the author used to commit or by the company field in the author’s Github profile. The company field is free-form text, has no modification history, and is not really verifiable. In addition only a minority of Github users actually use it.
A user committing to a repository using their work email address is the most reliable method of organization affiliation identification. However, accuracy across projects is still a challenge because authors don’t always use the same email address and some never use their work email address.
Tensorflow – distribution of email domain types
Tensorflow – percent of authors per email domain
Tensorflow – unique authors per email domain. What does this tell us about how engaged organizations are?
The human-system metrics approach to this problem is to develop a complex solution for author affiliation identification in a quest for precision, but only small gains can be had for the large increases in complexity and overhead. A question-driven approach built on proven social science and statistical methods suggests that you do not need precision, as long as you’re asking the right questions.
A graph of email addresses used by identified Googlers in the Tensorflow project
Use of a network to disambiguate authors and committers. Is the increased complexity worth the gain?
At least some affiliations can be identified with high certainty. Therefore, the heuristic to consider is that if an organization is actively engaged then they will have at least 1 commit with an associated email domain for a given time interval. This heuristic also prevents “gaming” by having members make a lot of erroneous commits to projects to pad their leaderboard position since only 1 will be counted for a given time interval.
Committers to Tensorflow by email domain. Given that these should be mostly Googlers and the huge proportion of personal email domains, anything else must be significant.
Caveat: Could this heuristic fail to identify a highly engaged organization?
Commit patterns instead of commit counts
Engagement frequency (commit interval)
When you think about relationships with people, the people you probably have the strongest ties to are likely the ones you’ve interacted with over a longer period of time. Using this model, an alternative to counting commits is to instead look for at least one commit, consistently and for how long. This also shows how long an organization has been committing to a project, something that just a count of commits can’t show.
Specifically this metric measures the frequency of time intervals with at least one commit (eg, months) and answers the question: How long has the organization been engaged?
Organizations with more than 6 months of Tensorflow commit months with at least one identified affiliated author
Organizations having the most Tensorflow commit quarters with at least one affiliated author
Engagement lifetime (commit interval sequence)
When plotted as a time series, the commit intervals illustrated above can also show us how consistently the organization has been engaged.
Organizations having the most months with an affiliated commit
Organizations having the most quarters with an affiliated commit
When is counting commits per organization sufficiently accurate (and low-cost)?
In open source there are two basic types of governance: closed and open. Closed governance means that a project leadership is controlled by a single entity and committers will largely be affiliated with that entity. Depending on the goals of the controlling entity, a project may be converted into an open governance project, which means any community member can take on a leadership role. In these cases, you can expect a higher rate of identification among committers belonging to the originating organization.
When the identification is essentially X and ~X, then the proportion of committers and commits attributed to each group should indicate if the majority are associated with the originating organization.
How well is a project doing at open governance? An open governance project, ideally, should not have a majority of committers representing one organization. This metric can also be used to demonstrate concerns on projects that have historically been open governance, especially if qualitatively there is a known majority influence.
The IBM Cognitive Open Technology team recently hosted a Tensorflow panel at the conference Index that featured several key Tensorflow Googlers. The claim was made that Google was working to diversify the Tensorflow community by increasing non-Google committers.
Unique Tensorflow committers: This appears to support Google’s claim
Tensorflow commits by committer affiliation: What additional questions might this raise about Google’s claim of committer diversity?
You’ve seen how counts of commits and authors alone are not particularly insightful. However, when plotted over time, commits can provide insight into the nature of the relationship organizations have with a project. Additionally, when considering what affiliations can be identified with more certainty, rather than chasing down what can’t, you can gauge if one or more organizations are dominating the activity in an open source project.
Final thoughts: The caveats of commits
Commits are not a sufficient single measure of engagement. Other activities must be considered to have an informed strategic discussion, not to mention the actual content of the commits themselves. Additionally the metrics proposed here are still in the exploration phase and have yet to be rigorously proven. Regardless, these metrics should offer a starting place to inspire better questions and better conversations about your own open source engagement goals.
- State of the Octoverse Annual Report
- Identifying Tensorflow Authors + Committers by Email Address
- Applying a Network to Tensorflow Commit Emails
- Tensorflow Email Diversity
Cross-posted from Augustina’s R Happy FUN! Blog