A comprehensive literature review of quantitative OSS health metrics

In preparation for launching our OSS health platform, we recently conducted a systematic literature review to establish a solid methodological foundation for selecting the metrics we want to use in the first iteration of our platform. Together with Jacqueline Schmatz, a master's student at St. Pölten UAS, we surveyed a total of 958 papers from several scientific databases through a literature review process following the PRISMA methodology. After applying several exclusion criteria (e.g. removing outdated studies, those comparing OSS and proprietary software, or those which did not contain any specific criteria or metrics for OSS health) and adding further relevant papers through citation-searching, we arrived at a selection of 26 papers, which we read and excerpted thoroughly. 

Our results were twofold: In a first step, we conducted a conceptual analysis to determine what the main aspects of OSS health were according to existing research literature. In a second step, we created a curated set of representative metrics from existing work corresponding to the aspects identified in the first step to achieve a usable and application-focussed operationalisation of these aspects. 

The four fundamental aspects of OSS health we established in the first step were: 

1. Community and Contributors;
2. Development and Activity;
3. Criticality and Maturity; and
4. Compliance. 

For each of these four aspects, we then chose metrics to operationalise them. Our selection criteria for the metrics included measurability, relevance and replicability. By applying these selection criteria, we arrived at the following list of metrics:

1. Community and Contributors:

• a. Bus Factor
• b. Pareto Principle
• c. Average Contributors per File
• d. Number of Support Contributors
• e. Size of Community
• f. Elephant Factor 

2. Development and Activity:

• a. Support Rate
• b. Technical Fork
• c. Change/Pull Request
• d. Issues
• e. Branch Lifecycle
• f. Churn

3. Criticality and Maturity:

• a. Criticality Score
• b. Maturity Level
• c. Code Dependencies
• d. Security Advisories 

4. Compliance:

• a. OSI-Approved License
• b. GitHub Community Metric 

The methodological details of our review will be available shortly; we have written up our findings in the form of a scientific article which is currently under submission at an international conference.