[Terms marked with * are further explained in the glossary below the main post]
The Cardiff IMGAME team went to Helsinki between 30 September and 1 November 2013 to conduct a new Imitation Game on religious beliefs at Helsinki University. During the two day visit, both Step1 and Step2 of the Imitation Game were completed. 19 different sets of questions and corresponding Non-Pretender answers were generated during Step1 (S1) by Helsinki University students who self-identified as ‘active, practising Christians’. A total of 188 Non-Christian students acted as ‘Pretenders’ in Step2 (see here for an explanation of the different IMGAME steps). Step1 and Step2 thus helped to generate 188 different transcripts, which were then allocated to 40 different Judge Files*. In the weeks after the visit, our Finish colleague Maria-Elisa Salonsaari recruited 40 Christian students who acted as Step4 (S4) Judges.Each S4 Judge received a unique Judge file and their task was to read through the different transcripts and try to identify the Non-Pretenders. A total of 373 judgements were made and of those 203 were correct identifications of Non-Pretenders, while 69 were incorrect identifications and 101 judgements were counted as ‘cannot identify’. This resulted in a Pass Rate – the rate at which Pretenders can successfully avoid being identified as such by Judges – of 64%.
The figure of 64% represents an average figure that is based on the individual pass rates of each of the 19 Dialogue Sets*. These 19 question-set related Pass Rates are shown in the chart below.
The chart shows the range of Pass Rates of the various Dialogue Sets (DS). DS12 and DS13 have Pass Rates of around 30%, which means only 3 out of 10 Pretenders has been able to avoid successful identification through Judges. In contrast, DS14 has a Pass Rate of just above 100%, which means that Judges were unable to successfully distinguish between Non-Pretenders and Pretenders based on the provided answers to the sets of questions.
As this particular IMGAME was a repeat of an Imitation Game conducted in October 2012, we can are able to compare the results. As mentioned above, the overall Pass Rate of the 2013 experiment is 64%. The overall Pass Rate of the 2012 experiment is 65%. On this basis, we might argue that we have successfully repeated the 2012 experiment. This would, however, be a premature conclusion. Looking closer at the available data the Pass Rates of the different Dialogue Sets of the 2012 show a somewhat different profile compared with 2013.
Further analysis is necessary to fully understand the results and the meaning of the Pass Rates in each experiment.
Transcript: a transcript or ‘dialogue transcript’ is essentially the output of an Imitation Game. It always consists up of three components: a set of questions, a corresponding set of Non-Pretender answers and a corresponding set of Pretender answers.
Dialogue Set: a Dialogue Set refers to a limited number of transcripts. Each transcript in the Dialogue Set shares the same set of questions generated by a judge as well as the same corresponding set of Non-Pretender answers provided by a non-pretender. These two shared components are generated during Step1. Each transcript in the Dialogue Set contains, however, a different set of corresponding Pretender answers. The sets of Pretender answers are generated during Step2.
Judge Files: Judge Files contain a set number, usually between 7 and 10 transcripts. A computer programme takes care of the allocation of transcripts to particular Judge Files. The software ensures that each Judge File does not contain more than one transcript from the same Dialogue Set.