STACKInnovating Education in Maseno, Kenya
Maseno University, Kenya
Santiago Borio, IDEMS
Michael Obiero Oyengo, Maseno University
Abstract
IDEMS International is working with Maseno University to implement online assessment for their mathematics courses. Developers from IDEMS and Maseno University created online quizzes for two courses during the pilot of the project, running on servers at the University of Edinburgh. The main challenges of the project were related to poor student access to WiFi and time pressure for the question authors. The feedback from the students was mostly positive, and the pilot paves the way for more similar work in the future. This includes online assessment for many more courses at Maseno University, as well as collaborations with other African universities and secondary schools.
Introduction
IDEMS International (Innovations on Development, Education and the Mathematical Sciences) is a Community Interest Company working to support different development causes, mostly in Africa, in fields of education and mathematical sciences. The non-profit organisation generates open education resources and offers services to Universities and other institutions. One of their projects was to help implement Computer Aided Assessment (CAA) in African Universities, of which Maseno University was chosen as a pilot.
Maseno University, Kenya, has a large number of students in its undergraduate mathematics courses, many courses having 800 enrolled students and some more than 1000, but there is little support for lecturers in terms of marking. This puts pressure on their ability to reliably do weekly assessments for students. By changing the continuous assessment from a paper format to an electronic system, the hope was that lecturers would have more opportunities to concentrate on their teaching and supporting students in other ways.
Why Use STACK?
There were a number of reasons for why STACK was chosen for this project. Firstly, there was already some familiarity with the STACK system within IDEMS, as one of the contributors had previously worked with STACK question authoring. Secondly, the main STACK developer offered assistance, including the opportunity to run the service on servers at the University of Edinburgh. Finally, having compared STACK with other systems, the developers found that the flexibility in answer tests, recognition of mistakes through potential response trees and the ability to have questions that require numerical or algebraic input for answers made STACK more powerful than the alternatives.
Execution
The developers from IDEMS and Maseno University created online STACK quizzes for the courses "Calculus I" and "Introduction to Linear Algebra". In the case of "Calculus I", the main lecturer provided a written quiz structure and list of questions, so the main work was just typing these into STACK and implementing randomisation. In contrast, the quizzes of "Introduction to Linear Algebra" were written mostly from scratch, only using a few questions from the STACK sample materials.
Initially, students are given a syntax quiz to learn the STACK syntax, and then in each of the following 10 weeks they are given a formative "mastery quiz" and a summative "test quiz". The mastery quiz can be taken as many times as the students wants, but they must get a score of 80% in the mastery quiz to unlock the respective test quiz, where students are only given one attempt. In "Introduction to Linear Algebra", this restriction was later reduced to 70%, as lecturers felt students were having a hard time keeping up with materials. The mastery and test quizzes together count for 30% of the students course grade. Towards the end of the course there is also a review week with STACK questions, and in "Introduction to Linear Algebra" students can take an online practice exam.
Feedback
After the first year, the students of the two courses were asked to fill out a survey about their experience with STACK. The first few questions regarded accessing the course. Maseno University does not have reliable WiFi, and it is not common for students' accommodation to have WiFi either. As a result, most students had to use either their own or a friends' phone, and purchase a data package to access the course. Indeed, when asked to rate the difficulty of different aspects, many students ranked "accessing the internet" in the range of "slightly challenging" to "very challenging".
When asked to rate how helpful they found different aspects of the assessments, many students ranked getting feedback, being able to have multiple attempts at mastery quizzes, having weekly quizzes and having the mastery quizzes open all semester as useful. Most students did not seem to have too much difficulty with the STACK syntax, ranking it in a range of "not a challenge" to "moderately challenging". This was surely helped by the attempts to ensure students were learning the syntax, for example through the introductory syntax quiz. However, it also seems there is room for improvement here.
The students were also given an opportunity to add any general comments or suggestions. The majority of these answers were from students who were happy with STACK and said it was useful for getting a deeper understanding of the material.
With regards to quizzes, many students requested more than one attempt at test quizzes, with extended deadlines. They requested more mastery quizzes, suggesting they found them helpful, and some students suggested access to harder, exam-style questions.
There were also some comments about the feedback for questions. Due to time constraints, the calculus course only had feedback in the form of worked solutions, and for linear algebra, students were only given outlines to solutions. Students said that they wanted better feedback, and so one of the goals for the next iteration will be to add tailored feedback to all questions.
Finally, there was a call to improve accessibility for the visually impaired.
Challenges
As is evident from the survey responses, there was a challenge regarding the students' access to internet. However, this did not end up being as big a problem as anticipated, as students seemed able and willing to pay for data packages to access the course.
Furthermore, due to the short term allocation of courses and lecturers, there was a significant time pressure for writing the quizzes. As a result, there was no time to implement tailored feedback, and a number of quizzes were delivered late. However, this did not seem to discourage students or create major issues in the completion of quizzes.
Enablers
A major enabler for the project was the ability of IDEMS to assist the University in question authoring. The questions were written by Michael Obiero Oyengo and David Ambogo from Maseno University and Santiago Borio, Danny Parsons and David Stern from IDEMS. Additionally, the main STACK developer C. Sangwin helped give crash courses in using STACK, which sped up the learning process. The course was also run on a server at the University of Edinburgh, which meant the developers did not have to worry about setting up a server and could focus entirely on question authoring. Finally, the STACK sample questions helped speed up the question authoring process, as the developers could use them as templates when writing similar questions.
What's Next?
Following the success of this pilot course, IDEMS is looking to expand into similar projects. The first step will be improving the two initial courses. Once the analysis of the data from the pilot course is finished, it will be possible to identify the main areas that need work. These two courses have been transferred to an IDEMS server, so they can be easily used for online assessment at other universities. The server will be built to cope with universities with many students, as IDEMS expects to use it for some courses with more than 1000 students.
A number of Universities have expressed interest in this project, including universities from Kenya, Ethiopia, Rwanda, Tanzania and Uganda. In particular, the mathematics department at Bahir Dar University, Ethiopia has agreed to investigate implementing a number of IDEMS' courses. In August 2019, IDEMS is holding a workshop, where representatives from these universities will discuss STACK and how to best implement e-assessment at their institutions. The 5-day workshop will include talks on the advantages of using online assessment, as well as practical work to prepare lecturers who want to use IDEMS' online course material. The workshop is funded by a grant from the Commission for Developing Countries (CDC) of the International Mathematical Union.
IDEMS is also helping Maseno University create online materials for several of its other courses. In the 19/20 term, Maseno University will run the improved "Calculus I", as well as five other courses with online assessment: "Descriptive Statistics", "Introduction to Probability Theory", "Basic Mathematics", "Calculus II" and "Vector Analysis". For these courses, IDEMS will be training some lecturers to author their own questions, which will increase their control of the online quizzes. As with the two pilot courses, these courses are planned to be piloted at Maseno University first, then refined based on feedback and made available as an OER for other universities.
Finally, IDEMS is looking at using STACK in secondary schools. There are plans to develop an e-book for African schools with accompanying STACK questions, including an offline system for schools with poor WiFi. IDEMS is also evaluating opportunities to develop an open assessment system for English schools.