Week 3 of our ReproducibiliTea journal club season 2, held on 15th April 2021 was facilitated by Shermain Puah. The focus of this session was construct validation, and we discussed the article by Flake and colleagues “Construct Validation in Social and Personality Research: Current Practice and Recommendations” published in 2017.
Before the group engaged in an extremely fruitful and productive discussion, I shared my main takeaways from this article. From the title of the article, it was clear that this paper could be broken down into (i) problems with how construct validation is being carried out and; (ii) the best practices of construct validation.
Current Practice/Problems
As Flake et al. (2017) pointed out, a bulk of the current social and personality research literature include latent variable measurement, however, sufficient validity evidence of the measurement instrument is often lacking. Or (debatably) worse, inaccurately used.
A big problem that most would agree on is the gross misuse of, and overreliance on, Cronbach’s α in the literature. Additionally, many broad constructs were observed to have been studied with very short scales, resulting in poor construct representativeness. To this end, however, it appears the root of the problem mainly lies in the lack of sufficient validation. For example, shortened scales may still reflect accuracy of the purported construct of interest if there is sufficient construct validity evidence (see Beymer et al. [2021] for validation of a shortened scale).
The consequence of the lack of sufficient construct validity is perfectly described in Flake et al.’s article: “Many constructs studied … lack appropriate validation, which will contribute to questionable conclusions and difficulty of subsequent research to replicate” (p. 5). Therefore, it is imperative that we establish valid measurement prior to making any interpretation of our results.
Recommendations/ Best Practices
Based on key findings of their review and previous seminal works on measurement, Flake and colleagues put together a list of recommendations for improving current practice.
A fundamental step for supporting researchers in the field highlighted by Flake and colleagues is ensuring adequate formal/graduate training on measurement. Many graduate programs offer no courses on measurement topics (e.g. test construction or classical test theory), leaving budding researchers unaware of the various methodologies associated with construct validation.
Discussion
Although formal training is useful, many agreed that more needs be done to ensure future studies utilise more vigorous methodology. In addition to mandating graduate curriculum to include important measurement topics, we hope to see stricter review guidelines from journals when reviewing papers that include latent variable measurement. Furthermore, encouraging authors to submit preregistrations of their planned methodology and analyses was also mentioned as a viable solution against the current lack of validity evidence reported by authors.
All in all, if authors received proper training in measurement topics, thought extensively about construct validation (which would be reflected in their preregistration), and coupled with strict guidelines from journals, we may see improvements in future research in the field of social and personality psychology.
More recently, I came across an interesting article which proposed using qualitative interview methods and machine learning to generate “gold standards” measure of their construct of interest (Jayachandran et al., 2021). Machine learning is gaining prominence in psychology, and it appears rather promising for use in latent variable measurement.
Several interesting discussion pointers were raised pertaining to the practical issues of construct validation – e.g. sample size, time, and money. The cost-benefit trade-off is a struggle that many researchers in Singapore may resonate with. Balancing between the practical constraints in a project and performing ongoing construct validation for one’s measures, has proven to easier said than done. However, as researchers, we should keep in mind the consequences of poor measurement and the lack of construct validation; ultimately, any claims made about the phenomenon of interest based on our measure has implications for practice/theory and, perhaps even for policy making.
Click here to view the discussion slides.
Thank you to all who were able to make time for our discussion on construct validation! I conclude with this quote by Andrew Gelman in one of his blogposts: “The replication crisis in social psychology (and science more generally) will not be solved by better statistics or by preregistered replications. It can only be solved by better measurement.”
In our next session on 22nd April, Michelle will be facilitating the discussion on teaching reproducible science. Do read the paper “How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science” by Chopik and colleagues, and join us virtually for yet another interesting open science discussion/chat. Bring your own Tea and snacks 🙂 We will be meeting virtually at:
https://ntu-sg.zoom.us/j/99534005049
Meeting ID: 995 3400 5049. Passcode: 032787
Click here for the schedule of the Singapore ReproducibiliTea Journal Club sessions.
