The F-ACT dashboard
The F-ACT dashboard has been specifically developed to address individual differences in academic writing competencies and support needs among students. The dashboard collects all the received assessments and feedback conveniently in one place and shows which competencies and related skills a student scores high and low on. This way the students know what they are doing well and where they need to pay more attention. In addition, the F-ACT dashboard provides a number of suggestions for resources the students can consult for each skill.
Background and hypotheses.
There are several examples of competency dashboards for students available in literature that demonstrate the feasibility and usefulness of such systems (Villamane et al, 2018; Nussbaumer et al, 2015;Grann & Bushway, 2014). For example, there are indications that students who use a competency dashboard demonstrate competencies at higher levels and persist in their program at greater rates (Grann & Bushway, 2014).
The aim of the F-ACT dashboard is to support students’ development of academic writing competencies and student ownership of feedback practices and study strategies. The F-ACT dashboard is available over multiple courses so that competency development becomes visible over a longer time. The F-ACT dashboard aggregates the competency and skill score based on teacher and peer feedback, as well as self-assessments (but also enables splitting these scores) and provides resources for further competency and skill development.
We hypothesize that the use of the F-ACT dashboard may increase students’ ownership of their own study behavior, resulting in more targeted requests for feedback from teacher and peers, and more targeted division of study time over the competencies and skills that require further development (Winstone et al, 2017; Little et al, 2024).
The project will run in the pre-master and bachelor programs of pedagogical sciences.
Realization of the F-ACT dashboard
The F-ACT dashboard is developed in powerBI. The data that are visualized in the dashboard are extracted from the educational application FeedbackFruits and consist of textual feedback and Rubric scores. A list of useful resources is provided in the dashboard as well (extracted from an Excel file provided by the teachers), such as links to videos, pdfs or websites.
Evaluation
In the evaluation of the F-ACT dashboard we will focus on three topics:
- Usage level
- Usability
- Usefulness
Participation in the evaluation will be based on consent.
Usage level.
We consider a good usage level of the F-ACT dashboard 80% of the students that participate in the academic writing skills courses in the premaster and bachelor program pedagogical sciences, which is roughly 280 students. Based on how the dashboard will be introduced in the program (see description under timeline), we aim to achieve this number in the academic year 2025 – 2026.
Usability.
We are aiming for an average score of at least 5 on the technology acceptance model 4 (TAM4; Hornbæk & Hertzum, 2017) at the end of the academic year 2025 – 2026. By including students and teachers in the development process, we believe it will be feasible to reach this score.
Usefulness.
The dashboard will be considered useful if students who use the dashboard show better competencies, have better grades on their academic writing assignments, are able to request more specific feedback, are more capable in handling and giving feedback, and managing their study time (e.g. focus on competencies that need further development). We will assess the usefulness by combining data about usage level of the dashboard with outcomes of the feedback literacy behavior scale (Dawson et al, 2023) and the feedback orientation scale (Linderbaum & Levy, 2010).
Results 2023 - 2024
In the academic year 2023 - 2024, we piloted the dashboard in an undergraduate (second year bachelor + premaster) course. The dashboard was introduced to 350 students. Most students did not use the dashboard on a regular basis. For example, in the last three months only 12 students logged in (+/- 3.5% of the students). On average, these 12 students accessed the dashboard 6 – 7 times in these three months. In combination with the low response rate (1 - 6% of the students) on the usability and usefulness questionnaires we could not properly evaluate the F-ACT dashboard.
Based on the limited information we received, we identified the following reasons for the limited uptake:
- The dashboard relied on data that were extracted from FeedbackFruits. When we introduced the dashboard at the beginning of the academic year, there was no FeedbackFruits data available from previous years. It only started to contain information after the first half of the course. This may have demotivated students to consult it;
- FeedbackFruits had been newly introduced and not fully adopted by all teachers. Therefore, there was no academic writing skill information available in the dashboard for some of the students;
- Using the dashboard was voluntarily and not incorporated into the curriculum, which led to low awareness of the existence of the dashboard despite the introduction session.
In the upcoming academic years, we will focus on further integrating FeedbackFruits in the academic writing skill courses. Next, we will gradually embed the dashboard more in the courses, so students must use it to complete an assignment. Lastly, we will promote the dashboard by creating a promotional video and sharing that video with the students.
Results 2024 - 2025
This academic year we focused on implementing the underlying data source Feedback fruits (FbF) of the F-ACT dashboard in 80% of the courses that teach academic writing skills in the premaster and bachelor program pedagogical sciences.
We also prepared for next year’s goal: improve the usability of the F-ACT dashboard and reach a score of five on the technology acceptance model (TAM)4. To reach this goal, we consulted the user experience team of Utrecht University (UU), who performed a usability assessment. We additionally asked students who used the F-ACT dashboard to fill in the TAM4 questionnaire this year.
Currently, FbF is implemented in 86% of the courses and the F-ACT dashboard was used in one course. In the next academic years, the F-ACT dashboard will be introduced in the other academic writing skill courses.
The outcomes of the UX-assessment and the TAM4 questionnaire showed that students found the F-ACT dashboard easy to use, but felt that its usefulness needs to be enhanced. The students’ recommendation was to further integrate the dashboard into the participating courses in the upcoming academic years, which is what we planned.
The implementation of FbF and the F-ACT dashboard is on track. The TAM4 and the UX-report show that the usability of the F-ACT dashboard is good. We will also assess the usefulness of the dashboard and analyze feedback receptivity and feedback literacy in relation to dashboard use in the next academic years.
Results 2025 - 2026
This academic year we focused on further consolidating the underlying data source Feedback fruits (FbF) of the F-ACT dashboard in 80% of the courses that teach academic writing skills in the premaster and bachelor program pedagogical sciences. Additionally, we aimed for 80% of the students for whom the dashboard is available to open the dashboard at least once. Lastly, we wanted to have usability and usefulness scores of at least 5 (scale 1 – 7) on technology acceptance model (TAM)4 questionnaire.
The dashboard was introduced in four courses: Developmental Psychology (DP) for cohort 24 (period 1) and the Bachelor Thesis for cohort 23 (periods 3 & 4). Additionally, the dashboard was made available to master’s thesis students, containing historical data from previous bachelor cohorts.
FbF is now used in 86% of the courses teaching academic writing skills. The usage of the dashboard changes over time, in an expected manner throughout the courses. Once the dashboard is introduced in a course, in that week usage peaks; 80 -85% of the students visited the dashboard at least once. During the course, usage level fluctuates between 15 – 25% of the students. Two weeks after the last assignments is added to the dashboard, usage level drops to 2-3%.
Usability and usefulness both scored four on the TAM4 questionnaire, which is lower than our aim of five. Students reported technical issues during the dashboard’s introduction, which might have caused the lower than aimed for scores. They further suggested improving the explanations of the dashboard’s functionality and goals during group or tutor meetings.
In academic year 2026–2027, the focus will be on the full integration of F-ACT into pre-master and bachelor programs, supported by experts from Onderwijsadvies & Training. Key priorities include creating promotional and instructional videos for not only students, but also teachers and course coordinators that can support further integration of the F-ACT dashboard in the current teaching processes.
Timeline
Academic year 2026 - 2027.
The dashboard is expected to be fully embedded in the academic writing skill courses of pedagogical sciences program.
Possibility for internship
Yes. You can contact team LA or the project leaders about internship possibilities.