On April 29, PLOS ONE, a leading peer-reviewed open access scientific journal, published our jointly authored paper on Conducting large, repeated, multi-game economic experiments using mobile platforms. This study was comprised of two of the largest synchronous economics experiments with paid human subjects that have ever been conducted and enabled the exploration of new findings pertaining to large vs. small group size effects. The experiments also validated the break-through usage of mobile devices and online payment systems to overcome challenges of conducting large scale experiments in-person as well as remotely online during the COVID-19 pandemic. We proudly present this study as an example of how the future of human subject experiments can scale well beyond laboratory walls.
This experiment was made possible through a close collaboration with Professor Zhi Li from Xiamen University (XMU) in China. Our author team was comprised of Professor Zhi Li (XMU), Po-Hsuan Lin (Caltech and MobLab), Si-Yuan Kong (MobLab), Dongwu Wang (MobLab), and Professor John Duffy (UC Irvine).
Background
In July of 2019, we were presented with the opportunity to conduct in-person incentivized experiments with a group of over 600 students from around the country that were gathered to attend an economics summer camp at XMU. The timing was tight, and we had to quickly devise an experimental setup that could work with such a large in-person group of participants in a single auditorium. Thus, we took full advantage of MobLab's pre-programmed mobile experiments to design and conduct a multi-game study using students' own devices and leveraging 4G network coverage to avoid overwhelming the university's WiFi network. The experiment session was ultimately a success and allowed us to gather observations for 633 subjects across eight classic laboratory games.
Left: Walter Yuan and Professor Li inside the 2019 experiment venue. Right: Students participate in the experiment using a variety of mobile devices.
As we analyzed the data and began to plan a repetition of the experiment for summer 2020, the COVID-19 pandemic swept the world into chaos and curtailed the possibility of any in-person events at XMU. However, a silver lining appeared as the 2020 summer camp would go virtual via livestreaming, presenting the opportunity to repeat our experiment in a fully online environment with live participants at home. Once again, the combination of MobLab and mobile payments enabled us to run our experiment with a nearly identical setup to preserve comparability with the previous year's results. We gathered observations from 585 subjects that largely replicated our findings from 2019, building confidence in the robustness of our results as well as indicating that differences between the in-person and remote settings were not particularly impactful.
Key Results
Left: MobLab's (Keynesian) Beauty Contest game. Right: Player distributions by level-k reasoning classification.
We were able to leverage our large-scale, multi-game approach to obtain interesting new findings on group size effects and how individual characteristics may play a role in behavior across games that would be difficult to obtain in traditional, limited capacity laboratory settings. For example, we find that as group size increases, the dynamics of player choices and distribution of behavioral types can vary significantly as observed in the Beauty Contest game (faster convergence to equilibrium) and Linear Public Goods game (more altruists, fewer free riders) respectively.
Overall, we find that our game-by-game results largely match those observed in past canonical laboratory studies where group sizes are similar. Furthermore, the results of our in-person experiment in 2019 were not substantially different from those of our at-home remote experiment in 2020. These encouraging findings support the prospect of conducting efficacious and incentivized human subject experiments using online mobile experiment platforms such as MobLab - a boon for researchers who seek to conduct large scale synchronous experiments and/or reach subjects outside of a physical laboratory.
Looking Ahead
2020 was an extremely challenging year for both educators and researchers in nearly every discipline. The COVID-19 pandemic upended normal student life and also made conducting traditional laboratory human subject experiments a very difficult task for economists and other social scientists. Thus, we are thankful for the opportunity to demonstrate an innovative and feasible path forward for researchers to conduct incentivized experiments in a variety of flexible in-person and remote settings. We hope that others will continue to build upon our approach.
Acknowledgements
This study was done in close collaboration with Xiamen University, and we thank the staff and organizing committee of the XMU summer camp. In addition, we recognize the extensive testing, logistical coordination, and on-site support that was provided by Professor Li's students and thank them for their efforts. We are also grateful for the helpful feedback from our colleagues and friends Colin Camerer, Joseph Tao-yi Wang, Thomas Palfrey, and others who attended our e-seminars. Finally, this study would not have been possible without the close support and diligent efforts of our MobLab colleagues, including Walter Yuan, Celine Lin, Yiming Liu, and Raymond Jia.