Comparative judgement is nowadays predominantly used in the educational domain. The D-PAC team aims to explore CJ’s strengths beyond this realm, for example in the recruitment and selection domain. Therefore, we conducted a try-out investigating whether or not D-PAC was successful when applied to CV-screening. Consequently we partnered with Hudson (http://be.hudson.com – a human resources consultancy company) using a received job opening from a client. Forty-two CV’s were received and D-PAC was used with 7 assessors to compare the CV’s. Assessors also provided pairwise feedback to justify each choice. The main questions were related to reliability and validity: (1) how reliable is the D-PAC assessment on CV screening with expert assessors (if the assessment would be performed again, how strongly will the ranking resemble the current one)? And (2) do all assessors look at the same and relevant criteria of the CV’s in relation to the job ad (validity)?
A group of 91 students second bachelor of the University of Hasselt in the track physiotherapy had the following task at the end of this year:
- - They had to formulate a clinical research question based on their experience as a physiotherapist;
- - Then they searched for a relevant scientific paper and formulated an answer to the research question based on the article.
- - At last they had to evaluate the article and point out the strengths and weaknesses of their study.
Normally all these papers are evaluated by one or two tutors. These tutors judge the paper by giving a ‘passed’ or ‘failed’ and provided feedback. You can imagine, this results in a substantial workload, especially when more than one task per student needs to be marked.
The next film is a testimony of an architecture professor who used D-AC for a peer assessment of mood boards. Because the movie is in Dutch, you can read a short summary of the main findings.
In the last months, several D-PAC try-outs have run. In these try-outs, assessments are set up in diverse organizations. For the organizations, the aim is to experiment with D-PAC. For us as a team, the try-outs are valuable to gain information on different aspects of D-PAC: the user-friendliness of the tool, how the tool can be embedded in real life situations and on how information out of D-PAC is used.
A first pairwise comparison experiment with video material in D-PAC is successfully completed. The goal of this experiment was twofolded: (1) test the tool on the scalability using videos; (2) and test the inter-rater reliability.
We at D-Pac are very happy to see that more and more organisations are interested in testing the Open source D-Pac platform (https://github.com/d-pac). As you can see, in the table below we have run a variety of assessments in diverse applications in the education and HR sector.