Assessment and Impact

At The Carpentries we are “Always Learning”. We embody the growth mindset we teach using data and community feedback to measure our impact and foster continuous improvement.

Measuring the Impact of Our Workshops and Trainings

We conduct pre and post workshop surveys for each Carpentries Workshop We Run

Find Copies of our Current Surveys

The Carpentries Pre and Post-Workshop Surveys

• Pre-Workshop Survey
• Post-Workshop Survey

Instructor Training Pre and Post Surveys

• Pre-Training Survey
• Post-Training Survey

Long-Term Impact Survey

Additionally, we began collecting data on the long-term impact our workshops are having on Carpentries learners. The Carpentries Long-Term Impact Survey was launched in March 2017, and data is collected every six months.

Reports

We release regular reports on the work and impact of The Carpentries. These reports are published through Zenodo and can be accessed through the links below.

Instructor Training Report

• 2021 Instructor Training Report

Analysis of Software and Data Carpentry’s Pre- and Post-Workshop Surveys

• Analysis of Software and Data Carpentry’s Pre= and Post Workshop Surveys (2018)

Post-Workshop Reports

• Analysis of Software Carpentry’s Post-Workshop Surveys (2017) . The data used in this report is available in this repository.

• Post-Workshop Survey Report (2016)

Long-Term Survey Reports

• Analysis of The Carpentries Long-Term Surveys (2020) These results are also available on GitHub.

• Analysis of The Carpentries Long-Term Impact Survey (2018)

• Carpentries’ Long-Term Feedback Survey Report (2017)

Data

The Carpentries provides a series of data feeds that you may be interested in. More information about these feeds are available in the Data Feeds GitHub repository.

Data sets (provided as csv files) and assessment reports are in a GitHub repository, along with the version of the surveys that were in use at time of analysis.

• Data for Post-Workshop Survey Report (2018-January)

• Data for Long-Term Survey Report (2018-January)

Measuring The Carpentries Reach

The following reports use data from our AMY database to describe the reach of The Carpentries. This includes number of Instructors certified, number of workshops held, number of learners taught, and number of countries that held Carpentries workshops.

• Programmatic Assessment Report (2018 Q1)

• Programmatic Assessment Report (2018 Q2)

• Programmatic Assessment Report (2018 Q3)

Other Assessment-Related Publications

When Do Workshops Work? A Response to the ‘Null Effects’ paper from Feldon et al.(2017) Author: Karen R. Word. Contributors: Kari Jordan, Erin Becker, Jason Williams, Pamela Reynolds, Amy Hodge, Maxim Belkin, Ben Marwick, and Tracy Teal.

This was a collaborative response to the paper: Feldon, David F. et al. Null effects of boot camps and short-format training for PhD students in life sciences Proc Natl Acad Sci U S A. 2017 Sep 12; 114(37): 9854–9858. doi: 10.1073/pnas.1705783114. Our data suggest that we are having a positive impact, and we expect that other short-format programs can be similarly effective. Read our full response.

Webinar with Rochelle Tractenberg: Debrief (2018)

The Assessment Network held a webinar with Rochelle Tractenberg. Dr. Tractenberg directs the Collaborative for Research on Outcomes and Metrics at Georgetown University, where she is a tenured professor in the Department of Neurology. Our starting point was the controversy about short-format training which arose last year, following the publication of a 2017 PNAS paper titled “Null effects of boot camps and short-format training for PhD students in life sciences”. The Carpentries design and deliver short-format training for working with software and data; trainees are researchers from various fields. The Carpentries’ initial response to the paper discussed many ways in which we have been successful with respect to our goals for Software Carpentry and Data Carpentry workshops. However, given that short-format training is a known challenge for generating sustainable content learning, we hoped that Dr. Tractenberg’s expertise might shed some light on areas with room for improvement. See the full webinar.