Spaces:
Running
Running
File size: 2,425 Bytes
91eaff6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
# Appendix: ML OSS Evaluation Rubric
The following checklist provides an evaluation rubric for open source code related to machine learning research.
For any given code repository, tally a score based on these questions:
- **Q1:** Does the repository use a business-friendly license?
- **Q2:** Does the code install correctly with either `pip` or `conda` package managers?
- **Q3:** Are the library dependencies reasonably current, not using pinned versions for popular libraries?
- **Q4:** Has the project provided sample code which runs without exceptions?
- **Q5:** Can the sample code reproduce the published results of the research?
- **Q6:** Does the library provide affordances for data integration, i.e., it's not optimized for a particular benchmark?
- **Q7:** Can the code be called programmatically as a library, i.e., not run primarily through a command line interface (CLI), and not requiring container/microservice orchestration?
- **Q8:** Will the library and its dependencies pass a reasonable level of security audit without structural changes?
- **Q9:** Does the code support concurrency and parallelization?
- **Q10:** Has the repo been maintained within the past six months?
## Dependency Evaluations
Based on this checklist, the dependencies integrated within this project scores as follows:
rubric | `OpenNRE` | `pulp` | `qwikidata` | `REBEL` | `spaCy` | `Spotlight` | `SpanMarker` | `transformers`
--- | --- | --- | --- | --- | --- | --- | ---
Q1 | x | x | x | x | x | x | x | x
Q2 | x | x | x | x | x | x | x | x
Q3 | x | x | x | x | x | x | x | x
Q4 | x | x | x | x | x | x | x | x
Q5 | x | x | x | x | x | x | x | x
Q6 | x | x | x | x | x | x | x | x
Q7 | x | x | x | x | x | x | x | x
Q8 | x | x | x | x | x | x | x | x
Q9 | x | x | x | x | x | x | x | x
Q10 | x | x | x | x | x | x | x | x
[`OpenNRE`](https://github.com/thunlp/OpenNRE/)
[`pulp`](https://github.com/coin-or/pulp)
[`qwikidata`](https://github.com/kensho-technologies/qwikidata)
[`REBEL`](https://github.com/Babelscape/rebel)
[`spaCy`](https://spacy.io/)
[`spaCy-DBpedia-Spotlight`](https://github.com/MartinoMensio/spacy-dbpedia-spotlight)
[`SpanMarker`](https://github.com/tomaarsen/SpanMarkerNER/)
[`transformers`](https://github.com/huggingface/transformers/)
There were many other open source code projects which were evaluated
but scored < 8 and were therefore considered unusable for our work.
|