Institutional Support for STACK in Edinburgh
University of Edinburgh
Interview with Chris Sangwin
Abstract
At the University of Edinburgh, online assessment has been consolidated with inhouse support, mostly with STACK, for the majority of year one and two mathematics modules and for many other mathematics and general science courses. A dedicated learningtechnologist post was created to help course organisers implement online assessment. Replacing the current online assessment with human marking is currently estimated to save the School over 6100 hours of work marking students' work each year.
Motivation
The School of Mathematics at the University of Edinburgh had, over the years, grown to use a number of separate online assessment systems. However, there was growing concern that using separate systems created a number of problems, for example students had to learn the syntax of many different systems. It was also problematic that students needed to purchase access to the online systems provided by publishers. At the same time, student numbers in the UK continued to grow [1], and there were growing expectations from students for more frequent formative assessment in their courses. This motivated the school to bring all assessments inhouse with STACK.
Execution
STACK was first used in Edinburgh at a small scale for the Lothian Equal Access Programme for Schools (LEAPS) summer school in 2016. Following the success with LEAPS, STACK was used in larger courses such as Introduction to Linear Algebra, a year one module with over 600 students, in the 201617 academic year. The University's primary learning environment is Blackboard "LEARN", so students gain access to a dedicated STACK service via the LTI protocol through their normal LEARN pages.
Having identified the need for additional support, the School of Mathematics created a dedicated "Learning Technologist" post. The primary goal of this post was to transform existing (largely paperbased) problem sets into online assessments and to develop alternatives to existing external online assessments. The Technologist used the following process to transform these assessments.

Before the year started, current forms of assessment were reviewed, and suitable STACK questions were written together with course organisers.

During the academic year, quizzes were made available, and their usage monitored.

When the year was over, this monitoring data was analysed and questions were updated where appropriate.
Creating online assessments is significant additional work, but once the quizzes have been created they require minimal work to maintain and can last for the lifetime of the course. Because of this costbenefit relationship, employing a Learning Technologist has been an effective strategy in ensuring change actually took place.
Results
In 2019, many courses have online assessments using STACK. A course will typically have weekly or fortnightly assessment, many using a combination of flipped classroom "reading quizzes" (RQ), formative "practice quizzes" (PQ) and summative "assessed quizzes" (AQ). Some courses also make an optional mock online exam available to students. The following table provides an overview from the 18/19 year, including the number of students, how often quizzes were given out (per week W or semester S), the average number of questions per quiz (#Q/quiz) and the total number of questions (#Qs).
Year  Course  Students  Quizzes  #Q/quiz  #Qs 

1  Introduction to Linear Algebra  604  2 RQ/W  23  
1 AQ/W  68  110  
1  Calculus and its Applications  594  1 RQ/W  4  
1 AQ/W  10  150  
1  Proofs and Problem Solving  344  1 RQ/W  2  20 
1  Engineering Mathematics 1a  393  3 PQ/W  5  
Mathematics for Natural Sciences 1a  138  1 AQ/W  710  240  
1  Engineering Mathematics 1b  401  3 PQ/W  35  
Mathematics for Natural Sciences 1b  136  1 AQ/W  710  200  
1  Mathematics for Physics 1  193  5 AQ/S  5  25 
1  Mathematics for Physics 2  202  5 AQ/S  6  30 
1  Fundamentals of Algebra and Calculus  113  5 PQ/W  1020  
3 AQ/W  612  950  
2  Probability  312  1 AQ/W  35  36 
2  Several Variable Calculus and DEs  287  1 PQ/W  512  
1 AQ/W  510  180  
3  Honours Algebra  202  9 AQ/S  18  46 
3  Combinatorics and Graph Theory  60  2 RQ/W  2  
4 AQ/S  16  45  
3  Symmetry and Geometry  36  1 AQ/W  47  45 
4  Galois Theory  27  1 RQ/W  2  22 
PG  Fundamentals of Optimization  190  3 AQ/S  48  16 
PG  Introduction to Probability and Statistics  23  1 PQ/W  68  
5 AQ/S  4  62 
Cost
Over the course of this project, the Learning Technologist estimates that developing a fully functioning quiz of 8 questions took about two persondays, or 16 hours of work, to create. As an example, consider the large firstyear course "Introduction to Linear Algebra", with around 600 students in 65 tutorial groups. Replacing half of the weekly handins with online assessments has saved each tutor over one hour of marking per week. Hence, over 65 hours of work is saved each week as a result. In addition, the students now complete more than double the number of practice problems, providing them with enhanced formative feedback which would be impossible to resource otherwise. Since online quizzes can be reused each year, this will be a consistent saving for as long as the course remains.
Overall, it is estimated that STACK saves the University over 6100 hours of work each year [2].
Barriers
The goal was to have the first 46 weeks of quizzes ready before the beginning of a semester. However, as the term progressed, it was not unusual for the last few quizzes to be ready only "at the last moment". This did not give the question authors a lot of time to review their questions, but does reflect the realities of teaching.
It was also important that the course organiser were fully involved in the authoring process. They had to give clear and explicit guidance on the learning objectives and help review the mathematical content.
Finally, when working with large question banks, organising became difficult. Large Moodle question banks are tricky to browse, and maintenance becomes tedious when questions are duplicated between similar courses.
Enablers
The most significant factor in the success of this project was the dedicated Learning Technologist post. This was essential, as it assured course organisers had practical support, and that online quizzes could have a consistently high level of quality.
It was helpful to be able to work closely with STACK developers when designing quizzes, in a way that is difficult with nonopensource programs. For example, the "numerical" input type which helps students enter answers at a prespecified level of numerical accuracy, was designed to help cope with the difficulties of assessing vague student answers in statistics questions [3].
What's Next?
The School of Mathematics continues to expand the use STACK at the University of Edinburgh. Future challenges involve modifying STACK to suit more conceptual courses, such as group theory and real analysis, as well as expanding into statistics and computer programming using the CodeRunner system [4]. There are also plans for wider use of STACK in summative assessments, with the possibility of using it in online examinations.
References
[1] Patterns and trends in UK higher education 2018. Universities UK, September 2018. ISBN: 9781840364095.
[2] C. J. Sangwin and K. Zerva. Developing online learning materials to support undergraduate education at the University of Edinburgh. Mathematics Today, 2019.
[3] K. Zerva. Developing STACK assessments in Edinburgh, 20172019. In Contributions to the 1st International STACK conference 2018 in Furth, Germany. Zenodo, 2019.
[4] R. Lobb and J. Harlow. Coderunner: a tool for assessing computer programming skills. ACM Inroads, 7(1):47{51, March 2016.