The D-PAC project covers several research foci situated on different research domains: measurement research (research focus A), feedback to individuals (research focus B), feedback to organisations (research focus C), feedback to assessors (research focus D) and design science research (research focus E).
Measurement Research – Research Focus A
The issues that research focus A attempts to answer are closely related to the method of CJ compared to more commonly used assessment practices. In the current literature on CJ there are several shortcomings concerning the underpinning of the reliability, validity and efficiency of CJ compared to scoring using descriptive scoring schemes (rubrics). In this project, research will thus be conducted to find out what type of CJ is most reliable, what is the validity of CJ’s interpretative arguments on scoring and generalisation and on the efficiency.
A second set of issues concerns whether a competence could be measured more validly using single or multiple representation sources, for instance audio, video or textual data (portfolios). At the start of the project, no studies have investigated the impact of portfolios on the validity of the judgements and the use of video, audio, longer texts, …
Marije will focus on the validity of CJ. Especially, how do assessors apply CJ and how do their decisions (choosing one of the representations) lead to a valid ranking? To investigate whether scores are valid, it is important that assessors apply the method appropriately. Therefore, the critical assumptions on how assessors come to their decision need to be investigated. One main assumption is related to construct validity and states that assessors should base their decisions on essential aspects of the competence. Secondly, in order to generate a reliable ranking, shared consensus on the conceptualization of the competence is necessary. Therefore, it should be investigated to what extent assessors agree on how they conceptualize the assessed competence. Additionally, Marije will examine how the characteristics of assessors, for instance expertise, decision style or self-efficacy, influence the way assessors apply CJ.
Different types of adaptive procedures for comparative judgement have been proposed and used in literature. This “list” is not by far exhaustive. But more importantly, no extensive research is to be found on the efficiency and effectiveness of adaptive procedures. In other words: Regarding adaptive procedures there are few studies looking into the trade off between number of comparisons and accuracy. And, there is no thorough research on the possible bias in resulting estimates after using adaptive procedures.
In his research, San will thus focus on the following questions:
- What are the current adaptive procedures used in the domain of CJ? And what are possible other promising procedures outside the domain of CJ?
- How efficient are these procedures compared to the non-adaptive selection procedure in CJ?
- What is the accuracy/effectiveness of these procedures?
Dr. Liesje Coertjens & Maarten Goossens
Next to the day-to-day coördination of the project, their research focuses on the comparison between CJ and the more traditional method of rating. This is investigated with regard to three elements. First, the validity: can evidence be found on which method produces more valid results? Second, which of the two methods generates the highest reliability? Third, what time investment is needed to arrive at these scores.
Next to this, Liesje will conduct simulation studies to investigate the performance and efficiency of different types of adaptive comparative judgement and she collaborates on the development of the analytics module of the D-PAC tool.
Feedback to Individuals – Research Focus B
Although quite some research on the impact of feedback on individuals has already been conducted, the CJ methodology will create data that opens up new possibilities to design feedback reports.
Research has pointed out that the effective use of feedback is also influenced by the way feedback is provided. Until now, no research has been done on the possible formats and content of the individual feedback reports based on CJ in general and more specifically when implemented in an online tool like D-PAC. Therefore research on feedback to individuals will focus on (a) exploring the feedback report characteristics concerning rankings and scores based on CJ, (b) how assessees perceive them and (c) what the effect is on assessees.
Feedback to the Organisation – Research Focus C
Roos Van Gasse
In the research focus on feedback at organisational level, the core objectives for this project are to understand the organisational processes and effects of the use of CJ in general and D-PAC specific feedback and to explain differences in both these processes and effects. This study explores the effects of feedback use at organisational level, as rigorous empirical studies into effects of feedback use at organisational level are scarce, and many often cited studies are short (small scale) studies or opinion pieces.
The current knowledge base fails to describe how workers in organisations (teachers/healthcare workers) interact with feedback and which mechanisms are necessary to result in desired outcomes of feedback processes at organisational level. Therefore this study will open up the black box in between receiving feedback and the effects of feedback use at organisational level.
The presented framework(s) enable(s) the description of organisational processes and effects of usage of these new possible feedback types. Based on existing research hypotheses, potential explaining factors were organised in three groups: (a) individual learner characteristics, (b) organisation/context characteristics, and (c) feedback system characteristics.
Feedback to Assessors – Research Focus D
Feedback to assessors is not a common thing. Literature on this topic is scarce. Therefore we will initiate exploratory studies, looking for answers on the following questions: What kind of feedback would assessor wants? For which purpose do they want feedback and what do they intend to do with it? Mixed-method research technics will be used to inform us regarding this novel research field.
Design Science Research – Research Focus E
Dr. Tanguy Coenen
Design Science Research aims to provide prescriptive knowledge on how to build certain artifacts. Applied to Information Systems, it is rather new, but applied to other disciplines like architecture or engineering, it has been around for a long time. Providing such prescriptive knowledge requires a complete view of the artifact, together with the knowledge base that is used to guide its design. By evaluating the artifact in its environment of intended use, the prescriptive knowledge can be grounded. Rigor, a necessary characteristic of all scientific disciplines, needs to be applied both to the design process and to the evaluation process.
In the case of D-PAC, the artifact under scrutiny is the information system itself. By applying a research and design process based on Action Research, Tanguy provides both guidance on the design of the system while contributing Design Science Research prescriptive knowledge that is relevant to the literature on online assessment.