1 How You Will Be Assessed

Every assessment in EEB 603 has an evaluation form setting out the criteria I use and how the points are allocated. All of them are published here, in advance.

Please write with the relevant form open. These forms are not something applied to your work after the fact — they are a description of what good work looks like in this course. If anything in them is unclear, or if you think your project does not fit the criteria as written, come and talk to me before the deadline rather than afterwards.

Deadlines for each assessment are on the course schedule; the rules on late work and firm deadlines are in the syllabus.

1.1 How the Assessments Fit Together

The six assessments in this course form two arcs of three, each worth 300 points — half the course each. Each arc builds on itself, and the second arc repeats the shape of the first on work of your own.

The two arcs have exactly the same shape — a small piece of work, a large one, and a piece of communication (50, 150 and 100 points respectively). The first arc rehearses on shared material; the second applies the same sequence to your own research.

Table 1.1: All assessments in EEB 603, grouped into the two arcs. Each arc is worth 300 points, and the course totals 600. Learning outcome numbers refer to the Course Learning Outcomes in the syllabus.
Assessment Points Learning outcomes Evaluation form
ARC 1 — Bioinformatics for Reproducible Science (PARTS 1–2) — 300 points
Participation and engagement 50 1, 2, 3, 6 See the syllabus
Bioinformatics tutorial (written) 150 4, 5, 6 Rubric (pdf)
Teaching the tutorial 100 6, 7 Rubric (pdf)
ARC 2 — Individual Project (PART 3) — 300 points
Project proposal 50 1, 2, 3, 4 Rubric (pdf)
Final report 150 1, 2, 3, 4, 5 Rubric (pdf)
Oral presentation 100 3, 4, 7 Rubric (pdf)

2 Arc 1 — Bioinformatics for Reproducible Science

300 points, across PARTS 1 and 2.

This arc is about acquiring the theory and the computational tools of reproducible science, and then proving you have them in the most demanding way there is: by teaching them to somebody else. It runs in three steps — you learn the material in class, you build a tutorial from it, and you teach that tutorial to your peers.

The three components are deliberately sequential. What you learn in PART 1 is what your tutorial is built from; the tutorial you write is what you then teach. Falling behind in the first makes the next two harder.

2.1 Participation and Engagement — 50 points

Learning outcomes 1, 2, 3 and 6.

Attendance and engagement in PARTS 1 and 2. PART 1 is where the reproducibility crisis, open science, the CARE principles and data management are studied through discussion of the assigned readings — which is where you first meet outcomes 1–3. You will meet them again in the individual project, where you are asked to apply that understanding to research of your own. Class time is also where I teach you how to build a tutorial in the first place — how to scope it, structure the exercises, and pitch it so your peers can finish it in two sessions — which is why outcome 6 begins here rather than at the point you start writing. PART 2 is then where you and your peers teach one another, and a laboratory cannot be taught to an empty room.

This component has no separate rubric; the criteria, including what counts as being present, are set out in full in the syllabus.

2.2 Bioinformatics Tutorial — 150 points

Learning outcomes 4, 5 and 6.

You design a tutorial on a chapter from PART 2 of the course, written in knitr/rmarkdown, built around exercises that give your peers a defined set of computational skills. This is normally a group assignment — see Group Work and How It Is Graded.

The rubric covers the introduction and theoretical background, learning outcomes, package requirements, dataset description, content quality, code quality, structure, references, and submission protocol.

Firm deadline: your tutorial must reach me one week before you teach it — your peers need that week to install packages and prepare. See the schedule for your chapter’s dates.

2.2.1 Where Each Tutorial Lives

Every tutorial has its own folder on the course Google Drive. The group teaching a chapter uploads its material there one week before the class; everybody else downloads it and works through it before that class, which is the point of the one-week deadline.

Chapter Folder Taught in class
Chapter 9 Tutorial material 10 and 12 November
Chapter 10 Tutorial material 17 and 19 November
Chapter 12 Tutorial material 1 and 3 December
Chapter 11 Tutorial material 8 and 10 December

The order is 9, 10, 12, 11, not numerical. Dates are confirmed in the schedule, which is the authority if these ever disagree.

Bioinformatics Tutorial rubric (pdf)

Write it as a protocol. The course chapters are deliberately built the same way: an explanation of why the skill matters, then a numbered step-by-step procedure, then a checkpoint so the reader can confirm it worked. You are asked to use that format. See How These Tutorials Are Written for what it looks like from the inside.

2.3 Teaching the Bioinformatics Tutorial — 100 points

Learning outcomes 6 and 7.

You teach your tutorial to the class over two laboratory sessions, guiding your peers through the exercises and troubleshooting as they work. This is the communication step of the first arc: explaining a technical workflow clearly enough that somebody else can run it is a different skill from being able to run it yourself.

The rubric covers presentation quality, explanation of key concepts, guidance through the tutorial, the exercises you developed, engagement and facilitation during the session, and professionalism.

Teaching rubric (pdf)

3 Arc 2 — The Individual Project

300 points, across three stages.

This arc applies the same sequence to your own research: you plan a piece of reproducible work, you build it, and you communicate it. Where the first arc rehearses on shared course material, this one is yours — your thesis project, or a published study if your topic is not yet defined.

All three stages are assessed on their own form, and each form records the combined total across all three, so you can always see where you stand in the arc as a whole.

3.1 Project Proposal — 50 points

Learning outcomes 1, 2, 3 and 4.

A short proposal, due about a third of the way through the semester, setting out what you intend to do. I read every proposal and return written feedback, so that you can act on it while there is still most of a semester left.

Two things are asked of this proposal beyond a conventional project plan:

  • It must itself be a reproducible document — an R Markdown source that knits, with at least one working code chunk and a bibliography. Your proposal then becomes the skeleton of your report rather than something written once and discarded.
  • It must say where your project sits within reproducible science, and how you will know whether your approach works. You are not expected to solve reproducibility in general; pick a part of the process, and tell me what will be reproducible at the end that is not reproducible now, and how you will test that. The chapters give you the vocabulary for all three:
    • Where it sits. Start from the definitions in Chapter 1: reproducible science, reproducibility and replicability are not synonyms, and saying which one you mean already narrows the project. Then locate the work itself. The chapters give you three maps for this, and you need only whichever fits your project best:
      • the steps of the scientific process, if your project is about how a study is designed, run or reported;
      • the data life cycle, if it is about how data are created, documented, stored, shared or reused;
      • the workflow in Figure 3.4, if it is about the analysis itself, since that figure shows which step of the process each of your code chunks implements.
    • What is broken now. The glossary of threats and the solutions that mitigate them in Chapter 3 are where to look. A proposal that names the specific failure it addresses is far stronger than one gesturing at reproducibility in general.
    • How you will test it. Use the standard your group applied to its reproducibility charter: a commitment only counts if somebody else could check whether you kept it. The seven TOP research practices, with their three levels (Disclosed, Shared and Cited, Certified), are a ready-made way of saying how far you intend to go.

3.1.1 Length, and How to Check It

Keep the proposal under 1000 words, counting your prose only: code, output, figure captions, the bibliography and the appendices do not count. It is a limit rather than a target, and a concrete 800 words is worth far more than a padded 1500.

That limit used to be stated as two pages. It no longer is, because you are submitting an R Markdown document that knits to HTML, and an HTML page has no page breaks to count. So instead, let the document count its own words.

3.1.1.1 Step-by-Step Procedure

  1. Open your proposal in RStudio and scroll to the end of your prose, just above the # References header.
  2. Insert an R code chunk there with the Insert button.
  3. Set the chunk options line to exactly {r wordcount, echo = FALSE}, where:
    • r comes first, and is not optional. It tells knitr that the chunk contains R code. Leave it out, writing {wordcount, echo = FALSE}, and knitting fails with Unknown language engine 'wordcount': knitr reads the first word inside the braces as the name of a programming language, and there is no language called wordcount.
    • wordcount is the label, so the chunk has a name you can find again.
    • echo = FALSE keeps the counting code out of the knitted document. The count is a measurement of your proposal, not a part of it, and nobody needs to read the code that produced it.
    • Do not add eval = FALSE. This chunk has to run, and eval is TRUE by default, so leaving it out is exactly what you want.
  4. Copy the code below into the chunk. Its last line assigns the number to n_words instead of printing it, so the chunk itself produces no visible output.
###
# Count the prose words in this document
# Excludes the YAML header, code chunks, inline code, HTML and citation keys
###
# knitr::current_input() returns the name of the file being knitted, so the
# next line reads THIS document's own source. That is why the snippet needs no
# filename: it works in anybody's proposal, whatever they called it.
#
# NOTE: this only works while knitting. Outside a knit there is no "current
# input", so current_input() returns NULL and you get the error
# 'con' is not a connection. Do not test this chunk with the Run button:
# knit the document instead.
src <- readLines(knitr::current_input(), warn = FALSE)

# Drop the YAML header
y <- grep("^---\\s*$", src)
if (length(y) >= 2) src <- src[-(y[1]:y[2])]

# Drop fenced code chunks, fences included
fence <- grepl("^\\s*```", src)
src <- src[!(cumsum(fence) %% 2 == 1 | fence)]

# Drop inline code, HTML tags, citation keys and Markdown punctuation
txt <- paste(src, collapse = " ")
txt <- gsub("`r [^`]*`", " ", txt)
txt <- gsub("<[^>]*>", " ", txt)
txt <- gsub("\\[@[^]]*\\]", " ", txt)
txt <- gsub("[#*_>|`]", " ", txt)

# Store the count, do not print it
words  <- strsplit(trimws(txt), "\\s+")[[1]]
n_words <- length(words[nzchar(words)])
  1. Below the chunk, write one line of ordinary text that reports the number using inline code, the syntax you met in Inline Code in PART A: 
    This proposal contains `r n_words` words.
  2. Knit the document. The sentence appears with the number filled in, for example This proposal contains 842 words.

💡 Why inline code rather than letting the chunk print

A chunk that prints its result gives you ## [1] 842, which looks like debugging output in the middle of a proposal. Inline code puts the number inside a sentence you wrote, which is what you want in a document somebody is going to read.

It is also the honest version of the thing this course keeps arguing for: the number is recomputed from your text every time you knit, so it cannot drift out of date the way a figure typed by hand would.

3.1.2 How to Submit It

Where it goes. Upload your submission to our course Google Drive, in Individual_reports > Individual_projects > Project_proposal.

Hand in a folder, not a single file. Inside Project_proposal, create a folder named with your own first and last name, for example Jane_Doe, and put inside it everything needed to knit the proposal on another machine:

  • the R Markdown source (.Rmd);
  • the bibliography file (.bib) and the citation style file (.csl);
  • any data file your code chunks read;
  • any image you inserted with include_graphics(), in the subfolder your paths point at.

The test is simple: could I knit it without asking you for anything? That is the same test as the rubric’s reproducible document criteria, and the same one your bioinformatics tutorial has to pass. A source that knits on your laptop but not in your submitted folder usually has one cause: a file that lives somewhere outside the folder, which you can find by moving the folder somewhere else and knitting again.

Submit the source, not an export. A PDF or a Word file cannot be knitted, so it cannot earn the marks reserved for the proposal being a working reproducible document.

3.1.3 Revise and Resubmit

📝 Open to everyone, and entirely optional

Every proposal has been graded and every evaluation returned, and you now have the opportunity to revise and resubmit this assignment to earn up to the full 50 points.

To do so, address the comments I provided and implement the suggested changes. You should also complete the rubric, indicating how you addressed each section and providing a brief self-evaluation, including how many points you would award yourself for each section.

If you choose to take advantage of it, submit your revised proposal on our course Google Drive, along with your completed self-evaluation rubric, by Thursday, 22 October at 5 PM MT.

I encourage you to take advantage of this opportunity to learn from the feedback and improve your work.

What to hand in. Two things, in the same First_Last folder you used before, in Individual_reports > Individual_projects > Project_proposal:

  • the revised proposal, as a folder that still passes the test in How to Submit It: could I knit it without asking you for anything?
  • the completed rubric, downloaded from the button below. For each of the seven sections, write how you addressed it and the score you would award yourself.

How the revised mark works. The score from your revised proposal replaces your original score, so you can only gain. The rubric and its criteria are unchanged, which is why filling it in yourself is part of the exercise: by the end of it you will have read the standard your report will be judged against from the inside.

The 1000-word limit still applies. If the feedback asks you to add a section, you may need to trim elsewhere to make room. That is deliberate: concision is one of the criteria, and deciding what to cut is part of the work.

Proposal rubric (pdf)

3.2 Final Report — 150 points

Learning outcomes 1, 2, 3, 4 and 5.

The full reproducible workflow, tailored to your thesis project or a selected publication. The rubric accommodates both full projects and pre-studies for students who do not yet have data.

The rubric covers framing of the scientific problem, use of data sources, the data management workflow, reproducible code and computational workflow, reproducibility and transparency, citations, and writing quality.

Deadline: accepted up to one week late, and no later — see the Late Work Policy.

Report rubric (pdf)

3.3 Oral Presentation — 100 points

Learning outcomes 3, 4 and 7.

The third and final stage. An 8-minute presentation of your project during the final week, to an audience of scientists who are not specialists in your area. A central goal is to show how you applied the tools and practices of reproducible science.

The rubric covers topic introduction, scientific content, reproducible science practices, slide design, oral delivery, and accessibility to a broad audience. A fifth of the marks go to Reproducible Science Practices — the tools and workflow you used, and how transparent and replicable they are — so this is not only a test of how well you speak.

Presentation rubric (pdf)

4 How the Assessments Map onto the Learning Outcomes

Every Course Learning Outcome is assessed somewhere, and Table 4.1 shows where. If you want to know why a particular assignment asks what it asks, this is the answer.

Table 4.1: Course Learning Outcomes and the assessments in which each is evaluated.
# Learning outcome Where it is assessed
1 Explain the evidence for, and causes of, the reproducibility crisis; distinguish reproducibility from replicability. Participation (PART 1 discussion); project proposal; final report
2 Evaluate the factors that break reproducibility in Ecology, Evolution & Behavior, and identify strategies that mitigate them. Participation (PART 1 discussion); project proposal; final report
3 Apply open science, data management and the CARE principles across the full data life cycle. Participation; project proposal; final report; oral presentation
4 Build reproducible workflows using R, RStudio, R Markdown and knitr. Bioinformatics tutorial; project proposal; final report; oral presentation
5 Use version control with Git and GitHub to track, document and share code. Bioinformatics tutorial; final report
6 Design and teach a bioinformatics tutorial that gives peers a defined set of computational skills. Participation (tutorial design taught in class); bioinformatics tutorial; teaching the tutorial
7 Communicate research rationale, methods and results to scientific and non-specialist audiences. Teaching the tutorial; oral presentation

Six of the seven outcomes are assessed in both arcs. You meet them first on shared course material, and then again on your own research. Only outcome 6 — designing and teaching a tutorial — belongs to the first arc alone.

That is particularly true of outcomes 1 and 2, on understanding the reproducibility crisis. You first meet them in the PART 1 readings and discussion, but they are not left behind there: the proposal asks you to say which part of the reproducibility problem your project addresses and what will be reproducible at the end that is not now, and the report asks you to assess the reproducibility of existing work and to distinguish what is reproducible from what is merely replicable. Understanding why reproducibility fails is not background reading for this course — it is what your individual project is built on.

5 A Note on These Forms

These rubrics are themselves written in R Markdown and rendered to pdf from source held in the course GitHub repository, alongside everything else in this course. If you spot an error in one, please tell me — and note that you now know how to fix it yourself.