Plot reliability app

Plot reliability of a task based on mean and sample variance of participants' scores in tasks with binary outcomes

Mean of participants' scores
(must be between 0 and 1):

Sample variance of participants' scores
(must be greater than 0, default in pandas, R, Matlab, for numpy use with ddof=1):

Number of subjects in your pilot, N
(must be greater than 1, recommended 50):

Number of trials per subject in your pilot, L
(must be greater than 1, recommended >=30):

Time to collect the trials
(optional, time in minutes, use with "Plot time" toggle):

Plot error

Plot time

x-axis limits 80

Desired reliability 0.8

Error of the estimate is high due to low N or L. Estimation of the number of needed trials might be inaccurate. We recommend adding more subjects and/or trials per subject to your pilot.

Given this data, we are unable to compute the coefficient. Add please more subjects and trials to your experiment (see the publication for an explanation on the model's limitations).

For this to function you have to provide time in minutes it took to collect the number of trials L that you entered.

Computed coefficient C is very high in your task potentially due to low variance between participants. We recommend reconsidering this task for estimating individual differences.

Reliability plot

Plots will appear here after you enter and Plot your values.

Disclaimer

The purpose of this tool is to aid in determining the number of trials that will be sufficient for a reliable estimate of individual differences, so that you can make an informed decision while designing your experiment. There are limitations to this tool (please see original paper cited below for a complete explanation of these limitations and recommendations). The authors cannot guarantee that the output from the tool will be accurate for all possible datasets and experiments. Use with caution and common sense!

If you use this tool, please cite the original paper:

A measure of reliability convergence to select and optimize cognitive tasks for individual differences research
Jan Kadlec, Catherine Walsh, Uri Sadé, Ariel Amir, Jesse Rissman, Michal Ramot
Commun Psychol 2, 64 (2024); doi: https://doi.org/10.1038/s44271-024-00114-4