On Thursday 7th of November our Singapore ReproducibiliTea journal club met for the second session, this time, to discuss analytic flexibility and how this represents a threat to reproducible and reliable science. The target paper to discuss in this session was False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant by Simmons, Nelson and Simonsohn (2011). Just before the session, we also discovered a follow up written by the same authors in 2018 and discussing the impact of their 2011 publication.
As we are quite a mixed group of people from very diverse backgrounds and career stages, it can be quite challenging to lead the discussion in a way that everyone can follow and finds it relevant to their work. Our amazing discussion leader Dr. Suzy Styles from the School of Social Sciences at Nanyang Technological University (NTU) did a great job in achieving these goals. By asking us to clarify key terms used in the by Simmons et al. (2011) and varying small and whole group discussions, she made sure that everyone had the opportunity to tailor the discussion to their own needs and interests.
A central topic that we focused on was the concept of p-hacking understood as the manipulation of decisions made throughout the data analysis process to produce a desired p-value. So-called researcher degrees of freedom, such as re-running statistical analyses with a subset of the originally collected data or including or excluding co-variates in the analyses are common examples of analytic flexibility applied by researchers to obtain desired results. Many of these decisions increase the likelihood of finding false positive effects and therefore represent an important threat to reliable and reproducible science.
To illustrate this problem, we first discussed the two experiments described by Simmons et al. (2011) to demonstrate how the chances of finding a significant p-value depend on the analytic decisions made by researchers. In one of the small group discussions one of our participants shared this interactive tool mentioned by Christie Aschwanden (2015) in her blogpost Science Isn’t Broken – It’s just a hell of a lot harder than we giver it credit for thanks to which you can experience p-hacking for yourself:
Finally, Dr. Suzy Styles showed us preliminary results from a meta-analysis she has been working on to identify all the decisions researchers apply when analysing electroencephalogram (EEG) data. Her findings reveal that authors rarely describe all of the possible analytic decisions they have to make to process this type of data in their scientific reports. This lack of transparency makes it difficult for the reader to know if the results of the study might be a consequence of p-hacking. Furthermore, it creates obstacles for future research to investigate the robustness of these findings by conducting exact replications. To reach a better understanding of the impact of different analytic decisions on the results of studies with EEG data, Dr. Suzy Styles and her team will now compare the results of 900 different analytic options resulting from seven key EEG data processing decisions: the so-called garden of forking paths shown below.
So far, we are very happy with some first indicators of the impact of our Singapore ReproducibiliTea journal club. Besides the enrichening discussions at our weekly sessions every Thursday, our Twitter account is also getting a lot of attention on a regional and international basis. Thanks to this medium of communication, we now joined a Slack group of other researchers from South East Asia interested in Open Science practices. A very concrete idea that came up within this context is to try to create multilingual versions of these two Open Science posters created by the Institute for Globally Distributed Open Research and Education (IGDORE).
In this way, we want to tackle potential language barriers and make sure that raising awareness on questionable research and Open Science practices occurs at a worldwide level. If you want to help us translate these posters into different languages, please get in touch with us. We also plan to set up a Zoom link for our weekly meetings, so that researchers located at NTU’s second campus in Novena and at other research institutions can virtually join our Open Science chats every Thursday from 1 to 2pm. Follow us on Twitter at @SingReproTea for updates on this matter.
In our next session on Thursday 14th of November, we will focus on the topic of Questionable Research Practices: are they really that common? And why are they problematic? based on the publications Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling by John, Loewenstein and Prelec (2012) and The Nine Circles of Scientific Hell by Neuroskeptic. Researchers in the field of education might be interested in complementing these views coming from psychology with a recent preprint entitled Questionable and Open Research Practices in Education Research by Makel, Hodges, Cook and Plucker (2019). We are very happy to welcome Dr. Gianluca Espósito and Atiqah Azhari from the Social and Affective Neuroscience Lab (San Lab – @SANLabNTU) at NTU as our discussion leaders for next week. So read the paper and come along for Teh Peng, snacks and Open Science chats! We will be waiting for you at:
The Arc – Learning Hub North, TR+18, LHN-01-06
Thursday 14th of November, 1-2pm
