Class Competition
Rubric
To receive a grade, you must create a pull request in the course blog repository with a final (updated) version of your class competition summary. The final version of your class competition summary will be assessed across 8 categories. Each category (described below) will be graded on either a Superior/Pass/Fail or Pass/Fail basis. Assignment letter grades are calculated in the following manner:
| Grade | Criteria |
A | No Fail and at least 2 Superior |
B | All Pass |
C | 1 Failure |
D | 2 Failure |
F | 3+ Failure |
Task summary
- Superior: Pass criteria + one or more of the following:
- a)discussion of related tasks
- b) challenges of task (with examples)
- c) discussion of SotA approaches
- Pass: All of the following:
- a) description of task (label space, dataset size, design, etc.)
- Fail: Task Description is inadequate or absent
Exploratory data analysis
- Superior: Pass criteria + additional analysis (ex. measures of diversity)
- Pass: All of the following:
- a) observations supported with example data points
- b) dataset size (per class)
- Fail: EDA is inadequate or absent
Approach
Your planned approach must incorporate techniques (algorithms, features, etc.) covered in LING 539.
- Superior: Pass criteria + approach highlights some novelty/creativity
- Pass: Description of Approach and motivation for use.
- Fail: Description of Approach is inadequate or absent
Results
- Superior: Pass criteria + measure of robustness of approach (ex. stratified k-fold cross validation, etc.) and/or statistical analysis
- Pass: Quantitative results on leaderboard noting delta with baseline
- Fail: Reporting of Results is inadequate or absent
Error analysis
- Superior: Detailed error analysis on held-out data (characterization of error) with examples
- Pass: Some error analysis on held-out data
- Fail: Error analysis is inadequate or absent
Reproducibility
- Superior: Containerized build with clear step-by-step instructions for reproducing leaderboard numbers
- Pass: clear step-by-step instructions for installing dependencies and reproducing leaderboard numbers
- Fail: Unable to reproduce results (ex. lack of instructions, missing components, etc.)
Leaderboard result
- Superior: Pass criteria + leaderboard submission exceeds the baseline
- Pass: All of the following:
- a) post specifies name of submission on leaderboard
- b) post includes score of leaderboard submission
- Fail: Any of the following:
- a) post does not specify name of submission on leaderboard
- b) post does not include score of leaderboard submission
Link to code repo
- Pass: All of the following:
- a) post includes a link to the code repository
- b) the code repository is not empty
- c) the repository uses the required assignment template (i.e., the repository is owned by the appropriate GitHub organization)
- Fail: Any of the following:
- a) post does not include a link to the code repository
- b) the code repository is empty
- c) the repository does not use the required assignment template
