Institutional Support for STACK in Edinburgh

University of Edinburgh

Interview with Chris Sangwin

Abstract

At the University of Edinburgh, online assessment has been consolidated with in-house support, mostly with STACK, for the majority of year one and two mathematics modules and for many other mathematics and general science courses. A dedicated learning-technologist post was created to help course organisers implement online assessment. Replacing the current online assessment with human marking is currently estimated to save the School over 6100 hours of work marking students' work each year.

Motivation

The School of Mathematics at the University of Edinburgh had, over the years, grown to use a number of separate online assessment systems. However, there was growing concern that using separate systems created a number of problems, for example students had to learn the syntax of many different systems. It was also problematic that students needed to purchase access to the online systems provided by publishers. At the same time, student numbers in the UK continued to grow [1], and there were growing expectations from students for more frequent formative assessment in their courses. This motivated the school to bring all assessments in-house with STACK.

Execution

Give an example of a matrix with eigenvalues {-7,6}. A wrong answer is given, with feedback tailored to the student's answer.
Figure: A typical Introduction to Linear Algebra question.

STACK was first used in Edinburgh at a small scale for the Lothian Equal Access Programme for Schools (LEAPS) summer school in 2016. Following the success with LEAPS, STACK was used in larger courses such as Introduction to Linear Algebra, a year one module with over 600 students, in the 2016-17 academic year. The University's primary learning environment is Blackboard "LEARN", so students gain access to a dedicated STACK service via the LTI protocol through their normal LEARN pages.

Having identified the need for additional support, the School of Mathematics created a dedicated "Learning Technologist" post. The primary goal of this post was to transform existing (largely paper-based) problem sets into online assessments and to develop alternatives to existing external online assessments. The Technologist used the following process to transform these assessments.

  1. Before the year started, current forms of assessment were reviewed, and suitable STACK questions were written together with course organisers.

  2. During the academic year, quizzes were made available, and their usage monitored.

  3. When the year was over, this monitoring data was analysed and questions were updated where appropriate.

Creating online assessments is significant additional work, but once the quizzes have been created they require minimal work to maintain and can last for the lifetime of the course. Because of this cost-benefit relationship, employing a Learning Technologist has been an effective strategy in ensuring change actually took place.

Results

In 2019, many courses have online assessments using STACK. A course will typically have weekly or fortnightly assessment, many using a combination of flipped classroom "reading quizzes" (RQ), formative "practice quizzes" (PQ) and summative "assessed quizzes" (AQ). Some courses also make an optional mock online exam available to students. The following table provides an overview from the 18/19 year, including the number of students, how often quizzes were given out (per week W or semester S), the average number of questions per quiz (#Q/quiz) and the total number of questions (#Qs).

Year Course Students Quizzes #Q/quiz #Qs
1 Introduction to Linear Algebra 604 2 RQ/W 2-3
1 AQ/W 6-8 110
1 Calculus and its Applications 594 1 RQ/W 4
1 AQ/W 10 150
1 Proofs and Problem Solving 344 1 RQ/W 2 20
1 Engineering Mathematics 1a 393 3 PQ/W 5
Mathematics for Natural Sciences 1a 138 1 AQ/W 7-10 240
1 Engineering Mathematics 1b 401 3 PQ/W 3-5
Mathematics for Natural Sciences 1b 136 1 AQ/W 7-10 200
1 Mathematics for Physics 1 193 5 AQ/S 5 25
1 Mathematics for Physics 2 202 5 AQ/S 6 30
1 Fundamentals of Algebra and Calculus 113 5 PQ/W 10-20
3 AQ/W 6-12 950
2 Probability 312 1 AQ/W 3-5 36
2 Several Variable Calculus and DEs 287 1 PQ/W 5-12
1 AQ/W 5-10 180
3 Honours Algebra 202 9 AQ/S 1-8 46
3 Combinatorics and Graph Theory 60 2 RQ/W 2
4 AQ/S 1-6 45
3 Symmetry and Geometry 36 1 AQ/W 4-7 45
4 Galois Theory 27 1 RQ/W 2 22
PG Fundamentals of Optimization 190 3 AQ/S 4-8 16
PG Introduction to Probability and Statistics 23 1 PQ/W 6-8
5 AQ/S 4 62
Table: Courses using STACK assessments at The University of Edinburgh 2018-19.

Cost

Over the course of this project, the Learning Technologist estimates that developing a fully functioning quiz of 8 questions took about two person-days, or 16 hours of work, to create. As an example, consider the large first-year course "Introduction to Linear Algebra", with around 600 students in 65 tutorial groups. Replacing half of the weekly hand-ins with online assessments has saved each tutor over one hour of marking per week. Hence, over 65 hours of work is saved each week as a result. In addition, the students now complete more than double the number of practice problems, providing them with enhanced formative feedback which would be impossible to resource otherwise. Since online quizzes can be reused each year, this will be a consistent saving for as long as the course remains.

Overall, it is estimated that STACK saves the University over 6100 hours of work each year [2].

The James Clerk Maxwell Building, with students walking.
Figure: Many School of Mathematics courses are taken at the James Clerk Maxwell Building.

Barriers

The goal was to have the first 4-6 weeks of quizzes ready before the beginning of a semester. However, as the term progressed, it was not unusual for the last few quizzes to be ready only "at the last moment". This did not give the question authors a lot of time to review their questions, but does reflect the realities of teaching.

It was also important that the course organiser were fully involved in the authoring process. They had to give clear and explicit guidance on the learning objectives and help review the mathematical content.

Finally, when working with large question banks, organising became difficult. Large Moodle question banks are tricky to browse, and maintenance becomes tedious when questions are duplicated between similar courses.

Enablers

The most significant factor in the success of this project was the dedicated Learning Technologist post. This was essential, as it assured course organisers had practical support, and that online quizzes could have a consistently high level of quality.

It was helpful to be able to work closely with STACK developers when designing quizzes, in a way that is difficult with non-open-source programs. For example, the "numerical" input type which helps students enter answers at a pre-specified level of numerical accuracy, was designed to help cope with the difficulties of assessing vague student answers in statistics questions [3].

What's Next?

The School of Mathematics continues to expand the use STACK at the University of Edinburgh. Future challenges involve modifying STACK to suit more conceptual courses, such as group theory and real analysis, as well as expanding into statistics and computer programming using the CodeRunner system [4]. There are also plans for wider use of STACK in summative assessments, with the possibility of using it in online examinations.

References

[1] Patterns and trends in UK higher education 2018. Universities UK, September 2018. ISBN: 978-1-84036-409-5.

[2] C. J. Sangwin and K. Zerva. Developing online learning materials to support undergraduate education at the University of Edinburgh. Mathematics Today, 2019.

[3] K. Zerva. Developing STACK assessments in Edinburgh, 2017-2019. In Contributions to the 1st International STACK conference 2018 in Furth, Germany. Zenodo, 2019.

[4] R. Lobb and J. Harlow. Coderunner: a tool for assessing computer programming skills. ACM Inroads, 7(1):47{51, March 2016.