10+ tools to help you visualize your GitHub and Git project data 
Tags: opensource, git, github, visualization
Published: 2019-12-18 16:13:31

Any important decision should be grounded on data. This is also true for any decision that affects your software projects. You shouldn't reach any conclusion regarding the health of your project or the actions to take to improve it without a good look at the data describing the project evolution (in terms of code changes but also regarding the community changes, especially if we are looking at an <a href="https://livablesoftware.com/5-ways-to-thank-open-source-maintainers/" target="_blank" rel="noopener noreferrer">open-source project</a>). Most times,<strong> this implies getting some meaningful data out of the Git repository and the code hosting platform (e.g. <a href="https://github.com/SOM-Research" target="_blank" rel="noopener noreferrer">GitHub</a>) where your project resides.</strong>

And this is where problems start. It's actually quite difficult to easily get data out of Git/GitHub. And if you want to analyze a number of projects (e.g. to know how yours is doing compared to other "similar" ones), it's almost impossible without investing too many hours. And it's not because of a lack of interest in mining this data. Plenty of researchers (<a href="https://som-research.uoc.edu/" target="_blank" rel="noopener noreferrer">ourselves</a> included) focus on developing <a href="https://livablesoftware.com/learned-software-development-mining-github-plus-concerns/" target="_blank" rel="noopener noreferrer">new theories on software engineering stemming from mined data</a>. There is even <a href="https://2020.msrconf.org/" target="_blank" rel="noopener noreferrer">a conference devoted to this field</a> of research.

But, so far, I haven't been able to find a tool that
<ul>
 	<li>Covers a good number of data sources around the project (for sure Git and the issue/bug tracker but also other importers, e.g. from communication channels the community may be using)</li>
 	<li>Supports extracting data from a number of projects "on-demand". Even better, if the tool would help to find me the projects I want to learn more from (using some kind of search tool based on the project language, size, popularity,...) and</li>
 	<li>Let me decide how to analyze the data (instead of limiting it to a number of predefined visualizations). Even better if it has a temporal dimension</li>
 	<li>Without spending weeks preparing the scripts to run the process (and more weeks waiting for them to finish)</li>
</ul>
While I haven't found the perfect tool (for me), at least we do have a number of good tools that will help you prepare this kind of <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load" target="_blank" rel="noopener noreferrer">ETL </a>process for software data. Depending on your scenario one of them may be enough. Or you can combine them. Let's see the Git and/or GitHub analysis tools I know (and let me know the ones I may be missing). As usual, this post does not pretend to be an exhaustive and perfect analysis of the tools but just a way to sort out a little bit the myriad of notes and thoughts I had written down in several places.

<h2>GitHub API</h2>
The most obvious choice. GitHub itself offers <a href="https://developer.github.com/v3/" target="_blank" rel="noopener noreferrer">a public API</a> to query any project. Unfortunately, there is a <a href="https://developer.github.com/v3/#rate-limiting" target="_blank" rel="noopener noreferrer">limit to the hourly number of requests </a>so using the API is not a good solution if you're looking to analyze large projects (or do some global analysis on a number of them). But if what you want to build some kind of <a href="https://livablesoftware.com/tools-visualize-github-profile/" target="_blank" rel="noopener noreferrer">dashboard focused on a single project or contributor,</a> this is more than enough. One nice aspect is that you can also <a href="https://developer.github.com/webhooks/" target="_blank" rel="noopener noreferrer">subscribe to get notified after certain events</a> occur in a project. This is the strategy we use in our <a href="https://xatkit.com/github-stargazer-alerts-stars-project/" target="_blank" rel="noopener noreferrer">stargazer bot</a>.

Keep in mind that, via this API, you can access basically all the info you see when browsing the GH repo of the project but you have a limited perspective on the internals of the "Git side of the project" (e.g. if you want to know what lines of code were modified during the last day).
<h2>Microsoft GHCrawler</h2>
The <a href="https://github.com/Microsoft/ghcrawler" target="_blank" rel="noopener noreferrer">GHCrawler</a> is a robust GitHub API crawler that walks a queue of GitHub entities transitively retrieving and storing their contents. GHCrawler is specially useful if you want to keep track of a set of orgs and repositories. Note that the previous rate limits still apply but GHCrawler employs token pooling and rotation to optimize use of your API tokens (if you're able to collect several ones from "friends and family").
<h2>GitHub Archive</h2>
<a href="https://www.gharchive.org/" target="_blank" rel="noopener noreferrer">GitHub Archive</a> is a project to record the public GitHub timeline, archive it, and make it easily accessible for further analysis. GH Archive stores all <a href="https://developer.github.com/v3/activity/events/types/" target="_blank" rel="noopener noreferrer">GitHub events</a> in a set of JSON files that you can later download and process offline as you wish.

Alternatively, GH Archive is also available as a public dataset on <a href="https://developers.google.com/bigquery/">Google BigQuery</a>: the dataset is automatically updated every hour and enables you to run <a href="https://developers.google.com/bigquery/docs/query-reference">arbitrary SQL-like queries</a> over the entire dataset in seconds. See the following query as an example

<script src="https://gist.github.com/igrigorik/90fab69946d27842882c.js"></script>
<h2>GHTorrent</h2>
GHTorrent monitors the <a href="https://api.github.com/events">Github public event time line</a>. For each event, it retrieves its contents and their dependencies, exhaustively. It then stores the JSON responses to a <a href="http://ghtorrent.org/raw.html">MongoDB database</a>, while also extracting their structure in a <a href="http://ghtorrent.org/relational.html">MySQL database</a>.

As you can see, its goal is similar to GH Archive. GH Archive aims at providing a more exhaustive collection of events while GH Torrent makes a stronger effort in giving you the events data in a slightly more structured way to make it easier for you to get all the information surrounding the event. See <a href="https://homepages.dcc.ufmg.br/~mtov/pub/2018-vem-thais.pdf" target="_blank" rel="noopener noreferrer">this paper for a better comparison</a>.
<h2>Gitana</h2>
<a href="https://livablesoftware.com/gitana-git-software-project-inspector-database/" target="_blank" rel="noopener noreferrer">Gitana</a> was our own attempt at solving the problem of building a user-friendly and generic solution for software data analysis. With Gitana, you could import a Git repo and the associated GitHub data in a relational database and then just use SQL queries to get out the data you needed. We had built importers from instant messaging and forum as well to try to get a complete picture of the project status.

<a href="http://livablesoftware.com/wp-content/uploads/2017/12/gitanaosp-fig1.png"><img class="wp-image-118509 aligncenter" src="https://livablesoftware.com/wp-content/uploads/2017/12/gitanaosp-fig1.png" alt="Gitana data model" width="676" height="396" /></a>

As you probably realized, I use the past tense to talk the tool. This is because it's part of the <em>archived</em> category in <a href="https://som-research.uoc.edu/research-tools/" target="_blank" rel="noopener noreferrer">our list of tools.</a>
<h2>Kibble</h2>
<a href="https://kibble.apache.org/" target="_blank" rel="noopener noreferrer">Apache Kibble</a> is a suite of tools for collecting, aggregating and visualizing activity in software projects. In follows a similar architecture to Gitana (and in fact to other several tools in the list), with a central Kibble server and a set of scanner applications specialized in working with a specific type of resource (a git repo, a mailing list, a JIRA instance etc) and push compiled data objects to the Kibble Server.

Based on this data, you can customize a dashboard with a number of widgets displaying project data (language breakdown, top contributors, code evolution,...). In this sense, Kibble is more of a tool that helps you to create a website displaying data of your project.

<a href="http://livablesoftware.com/wp-content/uploads/2019/12/kibble.png"><img class="alignnone size-large wp-image-118746" src="https://livablesoftware.com/wp-content/uploads/2019/12/kibble-1024x493.png" alt="" width="1024" height="493" /></a>
<h2>CHAOSS</h2>
<a href="https://chaoss.community/" target="_blank" rel="noopener noreferrer">CHAOSS</a> is a Linux Foundation project focused on creating analytics and metrics to help define community health. As part of this initiative, we can find several tools aimed at mining the data needed to compute such health metrics:
<ul>
 	<li><a href="https://github.com/chaoss/augur" target="_blank" rel="noopener noreferrer">Augur</a> is a <a href="http://augur.osshealth.io/" rel="nofollow">Flask web application</a>, <a href="https://oss-augur.readthedocs.io/en/dev/library-documentation/python.html" rel="nofollow">Python library</a> and <a href="http://augur.osshealth.io/static/api_docs/" rel="nofollow">REST server</a> that presents metrics on open source software development project health and sustainability. The goal is to facilitate the quick prototyping of new metrics of interest for the CHAOSS community.</li>
 	<li><a href="https://github.com/cregit/cregit" target="_blank" rel="noopener noreferrer">Cregit</a> focuses on generating views to visualize the provenance of the code changes</li>
 	<li><a href="https://chaoss.github.io/grimoirelab/" target="_blank" rel="noopener noreferrer">GrimoireLab</a> by <a href="https://bitergia.com/" target="_blank" rel="noopener noreferrer">Bitergia</a> is by far the more mature and ambitious tool in the pack. The aim of GrimoireLab is to provide an open source platform for:
<ol>
 	<li>Automatic and incremental data gathering from almost any tool (<em>data source</em>) related with contributing to Open Source development (source code management, issue tracking systems, forums, etc.)</li>
 	<li>Automatic data <em>enrichment </em>to clean and expand the data collected above (merging duplicated identities, adding additional information about contributors affiliation, calculation delays, geographical data, etc.)</li>
 	<li>Data visualization, allowing filtering by time range, project, repository, contributor, etc. GrimoireLab uses <a href="https://www.elastic.co/products/kibana" target="_blank" rel="noopener noreferrer">Kibana</a> to power all these cool visualizations on top of the collected data (see figure below)</li>
</ol>
</li>
 	<li><a href="https://github.com/chaoss/prospector" target="_blank" rel="noopener noreferrer">Prospector</a> had similar intentions but is now abandoned.</li>
</ul>
<h2><a href="http://livablesoftware.com/wp-content/uploads/2019/12/bitergia.png"><img class="wp-image-118747 aligncenter" src="https://livablesoftware.com/wp-content/uploads/2019/12/bitergia-1024x605.png" alt="" width="786" height="464" /></a></h2>
<h2>Sourced</h2>
<a href="https://sourced.tech/" target="_blank" rel="noopener noreferrer">Sourced</a> calls itself the "Data Platform for your Development Life Cycle". In contrast with the previous tools, it focuses more on the code of the project than on the collaboration of the community around it. Thanks to their universal AST, you can query very specific details of a codebase in a language-agnostic way.

<img class="aligncenter" src="https://sourced.tech/img/all.png" width="567" height="525" />

In the <a href="https://github.com/src-d" target="_blank" rel="noopener noreferrer">Sourced GitHub organization</a>, you can find several interesting data analysis projects. My favourite ones are <a href="https://github.com/src-d/go-git" target="_blank" rel="noopener noreferrer">go-git</a> ( highly extensible git implementation library written in pure Go), <a href="https://github.com/src-d/hercules" target="_blank" rel="noopener noreferrer">Hercules</a> (to perform analysis tasks over the full commit history of a repo) and <a href="https://github.com/src-d/gitbase" target="_blank" rel="noopener noreferrer">gitbase</a> ( a SQL database interface to Git repositories).
<h2>Hubble</h2>
<a href="https://github.com/Autodesk/hubble" target="_blank" rel="noopener noreferrer">Hubble</a> visualizes <a href="https://enterprise.github.com/">GitHub Enterprise</a> collaboration, usage, and health data. As such, its focus on helping large companies understand how their inner organizations, projects and contributors distribute and collaborate together.

Hubble Enterprise consists of two components. The <a href="https://github.com/Autodesk/hubble/blob/master/updater"><em>updater component</em></a> is a Python script that queries relevant data from a GitHub Enterprise appliance and stores the results in a Git repository once a day. The <a href="https://github.com/Autodesk/hubble/blob/master/docs"><em>docs component</em></a> is a web application that visualizes the collected data and is hosted with <a href="https://pages.github.com/">GitHub Pages</a>.

<a href="http://livablesoftware.com/wp-content/uploads/2019/12/hubble.png"><img class="wp-image-118748 aligncenter" src="https://livablesoftware.com/wp-content/uploads/2019/12/hubble-1024x570.png" alt="" width="810" height="451" /></a>
<h2>And there is more</h2>
There are also a couple of closed-source commercial tools you may want to look at: <a href="https://snoot.io/" target="_blank" rel="noopener noreferrer">Snoot</a> and <a href="https://waydev.co/pricing/" target="_blank" rel="noopener noreferrer">Waydev</a>. Curious also to see that even Amazon (<a href="https://github.com/amzn/oss-dashboard" target="_blank" rel="noopener noreferrer">OSS Dashboard</a>), Netflix (<a href="https://github.com/Netflix/osstracker" target="_blank" rel="noopener noreferrer">OSSTracker</a>) and PayPal (<a href="https://github.com/paypal/Gander" target="_blank" rel="noopener noreferrer">Gander</a>) had, at some point, their own software data analysis platform. They are now all abandoned. Same as <a href="https://github.com/zalando-incubator/catwatch" target="_blank" rel="noopener noreferrer">CatWatch.</a>

You may also want to take a look at <a href="https://www.stackalytics.com/" target="_blank" rel="noopener noreferrer">StackAnalytics</a> (an interesting initiative around the <a href="https://www.openstack.org/" target="_blank" rel="noopener noreferrer">OpenStack</a> community data), <a href="https://github.com/gotec/git2net" target="_blank" rel="noopener noreferrer">Git2Net </a>(Python package that facilitates the extraction of co-editing networks from git repositories) and <a href="https://www.aboutcode.org/" target="_blank" rel="noopener noreferrer">AboutCode</a> (a set of tools aiming to uncover different aspects of a codebase, including license information)