A detail of Analysis done#
This document summarises all the different analysis shown on the website. It also includes definitions of some terms used for better clarity.
About the experiment#
We have with us datasets from two different surveys.
The first survey, conducted at AI lab, was under controlled circumstances. By this, I mean that participants did not disturb each other during the experiment. They did all the 10 sets without any interruptions or breaks. Typical completion time was 20-25 minutes.
The second survey was a website, hosted on a public domain. Participants got the same set of instructions as the ones who participated in the first survey. The typical time of completion was around 2 days, but it varies between the participants.
Another difference between both the surveys, and probably an important one at that is the rating scale presented. In the first survey, we used an aggressive rating scale or the way I like to call it–the Alpha version. While in the second survey, we used a passive one–the Beta version.
α Scale
- Very Dissimilar
- Dissimilar
- Neutral
- Similar
- Very Similar
β Scale
- Dissmilar
- Slightly Dissmilar
- Neutral
- Slightly Similar
- Similar
Methods#
Participant profiles#
Simple things first; I made a profile of every participant who took part in either version of the experiment. Profiles highlight necessary information about the participant that gives an overall understanding of his/her stance about the similarity/dissimilarity in our set of chosen Telugu characters.
Each profile constitutes of three sections: Basic information, Summary of Data, and Ratings for all pairs.
The Basic information section gives details about known languages and Average reaction time of the experiment trail.
The next section, Summary of details, is a table that has the count, number of occurrences, the mean, and the standard deviation of each score.
The last section, Ratings of all pairs, as the name suggests, has a table showing just that.
Min-Maj Analysis#
Before we get into the details, I would like to define a few terms to facilitate better communication.
Agreement percentage: It denotes the percentage of participants who rated a pair, a particular score.
Majority: A pair has a majority if, the highest agreement percentage of that pair, is more than 50%.
Dominance: The score associated with the highest agreement percentage is the dominant score, while the percentage is called the dominant percentage.
The Min-Maj analysis aims to understand the distribution of the dominant percentages. We split the dominant percentage of all the pairs into 10 classes. Consequently, each class has a class interval of 10. All the pairs with dominant percentages between 0%-50% are listed in the Minority report while the rest are in the Majority report.
Overall Analysis#
This section has the Dominant score, the median score, the mean and standard deviation of the reaction time for each pair.
Novert analysis#
To see how domain experts(Telugu folks) par against novices, I combined the datasets from two versions and then divided them into two sets: Experts and Novices, and repeated the above analysis.