Integrating STACK at the University of Trieste, Italy.

Juma Zevick Otieno, Danilo Lewanski, Daniel Doz, Wastalas D'Asise Dogbalou

Background

The University of Trieste (UniTS) is a prominent player in Italy’s academic landscape, recognized for its efforts to align with contemporary educational trends in education. This is despite the largely traditional nature of Italy’s education system (for more details click here). In Italy, university assessments are traditionally summative, relying heavily on high-stakes, end-of-semester exams to measure student understanding. This model places emphasis on theoretical knowledge assessed at the end of the semester, often lacking interim feedback that could aid in bridging knowledge gaps as they arise. Although effective in certain respects, this system has significant limitations, particularly in its lack of continuous feedback mechanisms that could help students identify and address learning gaps throughout the semester. By regulation, lecturers are not obligated to administer continuous assessments during the semester. In response to this, during the 2022/23 academic year, UniTS launched a pilot project to integrate STACK, an open-source assessment tool designed for STEM subjects, starting with the Department of Mathematics, Informatics, and Geological Sciences (MIGe). Developed through a partnership with IDEMS International, UK-based community enterprise focused on supporting education and sustainable development projects worldwide. This started with a Probability and Statistics course, later scaling up with the addition of a Linear Algebra course in the academic year 2023/24. STACK was selected due to its automated, real-time feedback, allowing students to learn from their mistakes and progress through iterative practice.

The Italian Government, through its National Recovery and Resilience Plan PNRR - Piano Nazionale di Ripresa e Resilienza, which calls for innovation and digitalization in education, provided financial support accross various learning institutions in Italy, which allowed UniTS to sponsor two PhD research positions focused on STACK integration and expand collaborative opportunities with institutions abroad, mostly located in Africa.

This case study outlines the progress and insights gained from implementing STACK at UniTS to guide similar educational advancements in Italy and beyond. The project’s primary goal is to modernize assessment methods in STEM, providing students with more frequent, individualized feedback and aligning assessment practices with UniTS’s commitment to sustainable educational improvement.

Overview of Assessment Practices in Italian Universities

In Italy, university assessments typically includes three key phases:

1. Multiple-Choice Questions (MCQs) – In the first phase, students complete MCQs designed to test their foundational understanding of the course material. This part helps ensure students have grasped the basic concepts before advancing to more complex assessments.
2. Problem-Solving Exam – Students who pass the MCQs proceed to a problem-solving exam. This sitting exam challenges them to apply their knowledge in practical scenarios. Only students who score at least 18 out of 30 points on this section can move forward to the final stage.
3. Oral Exam – In the final phase, students undertake an oral exam, where they demonstrate their understanding through verbal explanations, engaging with instructors to discuss and defend their ideas. This phase evaluates students' depth of knowledge and ability to communicate concepts effectively.

The grading system in Italy ranges from 18/30, the minimum passing score, to 30/30, with “30 e lode” (cum laude, occasionally noted as 31/30) for exceptional performance. Continuous assessment, while increasingly valued, remains optional and is typically up to individual instructors.

In the 2023/2024 academic year, STACK was integrated into two courses at the University of Trieste. The first course, Probability and Statistics, had previously used STACK in the prior semester. The second course, Linear Algebra, was divided into two parallel sections, Course A and Course B, which followed identical curricula but were taught by different instructors. STACK was used to to support lecturers with continous assessment and give feedback to students. STACK was used to support lecturers with giving students practice questions in the course for continuous assessment and to provide students with timely feedback. Only mastery quizzes were implemented—these are specialized quizzes designed to help students achieve a deep understanding of key concepts. The quizzes remained open throughout the semester, allowing students ample time to practice and master the material at their own pace.

Implementation Phases

1. Training: The initial introduction of STACK at UniTS was led by IDEMS International, which focused on training instructors who were new to digital assessment tools. This training helped instructors become familiar with STACK’s features and effective practices for digital assessments.

2. Customizing Questions: The next step was to tailor the question sets to fit the specific learning objectives of each course. Instructors, with IDEMS support, reviewed resources from the IDEMS open question bank and created new questions where necessary. IDEMS also provided access to open-source resources like the HELM Project and the African STACK Open Question Bank to enrich the question pool.

3. Engaging Students: To help students get comfortable with STACK, instructors provided informal onboarding sessions within each course. These sessions included tutorials on navigating quizzes, using feedback effectively, and retaking quizzes to reinforce learning. Initially, students were uncertain about the option for multiple quiz attempts, but they quickly adapted and came to value the immediate, constructive feedback provided.

Challenges and Solutions during Integration:

The integration of STACK at UniTS came with a few key challenges:

1. Technical Issues: There were initial technical difficulties with STACK’s server integration, but IDEMS provided support to resolve these issues. Additionally, the lead instructor addressed technical issues as they arose throughout the course, ensuring smooth ongoing functionality.

2. Question Development: Crafting effective questions that met STACK’s technical requirements was initially time-consuming. However, by using pre-existing materials from the African Question Bank and the HELM Project, instructors were able to expedite the question development process, tailoring questions to fit course objectives more efficiently.

Outcome

To evaluate STACK’s impact on student performance, we analyzed engagement data from three courses at UniTS: two Linear Algebra courses (Course A and Course B) and one Probability and Statistics course. This analysis aimed to understand how varying levels of STACK engagement influenced student outcomes. In Course A of Linear Algebra, students had access to all STACK quizzes and completed each one throughout the semester, showing consistent engagement. In contrast, Course B students attempted only 6 out of the 10 available quizzes, despite following an identical curriculum and sitting for the same final exams as Course A. This variance in engagement levels between the two otherwise identical courses provided a unique opportunity to explore how consistent quiz participation might affect exam performance and mastery of course content.

In the Probability and Statistics course, STACK had been used previously, and this time the new student cohort also engaged with all available quizzes, completing at least one attempt per quiz. Therefore, we did not analyze pre- and post-STACK implementation for this course as we did for the two Linear Algebra courses. Since this was the second cohort to use STACK, there was no baseline data without STACK for comparison. Instead, our focus here was on evaluating overall performance trends with STACK already in place.

In comparing student performance in the two Linear Algebra courses (A and B), there were notable differences post-STACK integration. For Course A, the analysis showed a significant improvement in student performance after integrating STACK. The Mann-Whitney U test indicated a significant difference in grade distributions between the pre- and post-STACK groups, suggesting that STACK’s continuous assessment and feedback likely contributed to improved outcomes. For Course B, however, the results did not show a statistically significant change in performance post-STACK integration, as the Mann-Whitney U test returned a non-significant result.

While the results suggest a positive impact of STACK, we recognize that correlation does not equal causation. The improvement in performance could partly stem from factors like individual motivation or external influences. We discuss these considerations and potential limitations in detail in a separate research paper. That paper also presents insights from a K-means classification analysis. This analysis showed that students with steady engagement tend to achieve moderate performance and are more often female. In contrast, students with irregular study habits span all performance levels and are more commonly male.

What Students Say About STACK

To gather insights into student experiences with STACK, we conducted a qualitative feedback analysis based on a survey administered to students in the Probability and Statistics course. Since this was the second time STACK was used in this course, students were able to provide well-informed feedback on its advantages and challenges. Due to time constraints, similar qualitative feedback was not collected from the two Linear Algebra courses. For more in-depth findings, please refer to our published paper click here.

Here are some key themes that emerged from the feedback:

Instant Feedback and Clarity
Many students highlighted the benefit of immediate feedback and clear instructions provided by STACK, which allowed them to quickly assess their understanding.

“...Comodo e si ha subito il risultato… mi sono trovato molto bene, soprattutto per studiare. L'esame era chiaro e non necessitava di dover dimostrare i vari passaggi...”
Translation: "...Convenient and you get the result quickly... I found it very good, especially for studying. The exam was clear and didn’t require demonstrating each step."

Concerns about Evaluating the Thought Process
Some students expressed concerns that STACK emphasized final answers rather than the problem-solving process. They felt this approach might limit their ability to demonstrate comprehensive understanding during exams.

“Rimango dell’opinione che gli esami online siano molto più scomodi rispetto a quelli cartacei. Credo che con carta e penna sia più facile valutare il processo risolutivo, mentre su Stack viene preso in considerazione solo il risultato finale.”
Translation: "I remain of the opinion that online exams are less convenient than paper-based ones. With pen and paper, it's easier to assess the solution process, whereas STACK only considers the final answer."

Reduction of Exam Anxiety
For some students, practicing with STACK before exams helped reduce anxiety, making them more comfortable with digital assessments.

“Fare l'esame su STACK non crea ansia e pressione. Avendo fatto molta pratica sulla piattaforma, mi sembra normale fare gli esercizi attraverso lo schermo.”
Translation: "Doing exams on STACK doesn’t create anxiety or pressure. Having practiced a lot on the platform, it feels normal to do exercises on the screen."

Concerns over Technical Issues
Despite improvements, some students continued to report technical difficulties, including issues with devices and internet access, which occasionally disrupted their assessment experience.

These insights provide meaningful perspectives on students' experiences with STACK in the Probability and Statistics course, and we believe they likely reflect broader patterns that could apply to students in the two Linear Algebra courses (A & B) as well.

Conclusion and Next Steps

The integration of STACK in our Linear Algebra and Statistics courses has yielded promising outcomes, particularly when students fully engage with the quizzes. Moving forward, our efforts will concentrate on refining and expanding our STACK content. We have already developed new STACK questions, including Parsons problems, which will be implemented in the 2024/2025 Linear Algebra course. This will enable us to further assess how we can leverage STACK teaching and learning approaches.

1. Refinement and Expansion: We will continue to refine existing STACK questions, particularly those used in Linear Algebra. The newly authored STACK questions will be rolled out in the 2024/2025 semester, allowing us to monitor their impact on student learning outcomes.

2. Collaboration and Research: We are constantly looking out to establish collaborations through STACK development and research ideas, since we are still new at this, both within Italy and beyond. This will play a vital role in expanding our understanding and ability to carry out research experiments that explore the broader application of STACK in mathematics education.

3. Integration in More Courses: We plan to extend STACK to more first-year courses, particularly in disciplines like Physics, where it can significantly aid in students’ comprehension of complex concepts. Expanding STACK's application will not only provide students with a more interactive learning experience but also generate additional data to evaluate its overall effectiveness across different fields.