D-PAC successfully handles video-material on large scale

A first pairwise comparison experiment with video material in D-PAC is successfully completed. The goal of this experiment was twofolded: (1) test the tool on the scalability using videos; (2) and test the inter-rater reliability.

A group of 134 students in Education Sciences had to judge 9 clips on the quality of the simulated scientific semi-structured interview demonstrated. The pairwise comparisons were all scheduled synchronously. So, in total 134 assessors were simultaneously interacting with the D-PAC system which was sending out video clips to these assessors. During the experiment no technological issues arose, leading to a very positive conclusion on the scalability of the D-PAC tool.

In order to test the inter-rater reliability the group of assessors was split in three random groups consisting out of 46, 44 and 44 assessors. All of the groups assessed the video’s in a comparative manner. The only difference between the groups was in terms of providing feedback when every comparison was completed. Group 1 was not specifically instructed to give any argumentation or feedback during the process. The second group was asked to give a short overall argumentation for their choice after each comparison. Group 3 was asked to write down some positive and negative features of each interview after each comparison. The amount of comparisons each group made was 520, 354 and 351 comparisons, respectively.

Based on the pairwise comparison data we calculated the Scale Separation Reliability for each of the three groups of assessors separately. The results are given in Table 1. From this table it can be seen that the reliabilities are high (.91 – .93).

Table 1. Scale separation reliability and average number of comparisons per video

Scale Separation Reliability Average number comparisons per video
Group 1 .93 104
Group 2 .93 79
Group 3 .91 78


To provide an answer on the question of inter-rater reliability we calculated the correlations between the estimated abilities (based on the Bradley-Terry –Luce Model) of each of the three assessments (see Table 2).  The Spearman rank correlations between the two assessments in which assessors had to provide an argumentation (Group 2) and where assessors had to provide feedback (Group 3) is the highest (.87). The Spearman rank correlations between the scores resulting from the assessment without any argumentation (Group 1) and the two other conditions are somewhat smaller (.82 and .84). Overall these correlations are high.


Table 2. Spearman Rank Correlations between scores coming from the 3 groups of assessors

Group 1 Group 2 Group 3
Group 1 1
Group 2 .82 1
Group 3 .84 .87 1


Given that each of the 36 possible pairs were assessed by multiple assessors within and between the three groups, we were able to calculate the agreement between assessors for each possible pair. In Figure 1 the agreement is plotted per pair, split up for the three groups of assessors. As shown, the average agreement in each group overall is around 77%. For some pairs the agreement is only 50%, for other pairs the agreement is 100%. These differences can, of course be partially attributed to the fact that in some of the pairs are more difficult to judge than some other pairs. Comparing the results of the three groups showed no significant differences.

fig 1 blog 3

To conclude, this pairwise comparison experiment first of all demonstrates the robustness of the tool to deal with large numbers of assessors assessing video clips simultaneously. From the resulting scales and pairwise comparison data learned us that the inter-rater reliability seems to be rather high as well.

D-Pac field trials

We at D-Pac are very happy to see that more and more organisations are interested in testing the Open source D-Pac platform (https://github.com/d-pac). As you can see, in the table below we have run a variety of assessments in diverse applications in the education and HR sector.

In each of these assessments, the organisation that we work with gets the opportunity to find out what assessments through Comparative Judgement can mean for them in a hands-on way. The deal is that we provide the hosted software and advice on how to set it up and run the system for free, while the organisation that hosts the assessment provides us with the research data we need to contribute to the advancement of the knowledge on Comparative Judgement and the tools that support it.

Competence Domain Assessees Assessors Reliability
1 Argumentative writing Education 135 High-school students 68 Teachers and teachers in training 0.81 average
2 Writing formal letters Education 12 High-school students 11 Teachers in training 0.68
3 Mathematical problem solving Education 58 High-school students 10 mathematics teachers + 4 mathematics teachers in training 0.80 average
4 Capability of visual representation in the arts domain Education 11 High-school students (147 representations) 13 teachers 0.85
5 Interpreting statistical output using peer-evaluation Education 44 Master students 33 Master students 0.80
6 CV-screening Human resources 42 candidates 7 HR professionals 0.88
7 Self-reflections Education 22 Master students 9 teachers 0.75
8 Project proposals 6 projects 5 judges 0.71

Besides the domains of education and human resources, we are also very curious to find out what other applications could exist for Comparative Judgement. For example, would it be useful in A/B testing in the marketing or IT development domain and would it be interesting as an alternative voting mechanism in for example TV shows?

To conclude, there could be a range of applications for the D-PAC tool that we have not yet thought of, so we are looking for ideas on how we could expand the impact of our tool.
Feel free to contact us if you are interested in testing D-PAC in your own context and together we can find out how we can help each other: d-pac@uantwerpen.be


The D-PAC team

Paper gepubliceerd

Performance assessments worden vaak gebruikt om competenties te evalueren. Maar hoe scoor je nu het best zulke performance assessments? Via criterialijsten? Holistisch?

Het artikel  “Competenties kwaliteitsvol beoordelen: brengt een comparatieve aanpak soelaas?”, gepubliceerd in  ‘Tijdschrift voor Hoger Onderwijs’, stelt de comparatieve beoordelingsmethode als methode voor om performance assessments te beoordelen in een hoger onderwijscontext.