Class Competition
Rubric
To receive a grade, you must create a pull request in the course blog repository with a final (updated) version of your class competition summary. Note that you should be updating the same file used for your draft. The final version of your class competition summary will be assessed across 8 categories. Each category (described below) will be graded on either a Superior/Pass/Fail or Pass/Fail basis. Assignment letter grades are calculated in the following manner:
| Grade | Criteria |
A | No Fail and at least 2 Superior |
B | All Pass |
C | 1 Failure |
D | 2 Failure |
F | 3+ Failure |
Task summary
- Superior: Pass criteria + one or more of the following:
- a)discussion of related tasks
- b) challenges of task (with examples)
- c) discussion of SotA approaches
- Pass: All of the following:
- a) description of task (label space, dataset size, design, etc.)
- Fail: Task Description is inadequate or absent
Exploratory data analysis
- Superior: Pass criteria + additional analysis (ex. measures of diversity)
- Pass: All of the following:
- a) observations supported with example data points
- b) dataset size (per class)
- Fail: EDA is inadequate or absent
Approach
- Superior: Pass criteria + approach highlights some novelty/creativity
- Pass: Description of Approach and motivation for use.
- Fail: Description of Approach is inadequate or absent
Results
- Superior: Pass criteria + measure of robustness of approach (ex. stratified k-fold cross validation, etc.) and/or statistical analysis
- Pass: Quantitative results on leaderboard noting delta with baseline
- Fail: Reporting of Results is inadequate or absent
Error analysis
- Superior: Detailed error analysis on held-out data (characterization of error) with examples
- Pass: Some error analysis on held-out data
- Fail: Error analysis is inadequate or absent
Reproducibility
- Superior: Containerized build with clear step-by-step instructions for reproducing leaderboard numbers
- Pass: clear step-by-step instructions for installing dependencies and reproducing leaderboard numbers
- Fail: Unable to reproduce results (ex. lack of instructions, missing components, etc.)
Leaderboard result
- Superior: Pass criteria + leaderboard submission exceeds the baseline
- Pass: All of the following:
- a) post specifies name of submission on leaderboard
- b) post includes score of leaderboard submission
- Fail: Any of the following:
- a) post does not specify name of submission on leaderboard
- b) post does not include score of leaderboard submission
Link to code repo
- Pass: All of the following:
- a) post includes a link to the code repository
- b) the code repository is not empty
- c) the repository uses the required assignment template (i.e., the repository is owned by the appropriate GitHub organization)
- Fail: Any of the following:
- a) post does not include a link to the code repository
- b) the code repository is empty
- c) the repository does not use the required assignment template
