Open research is a movement that advocates transparency, collaboration and accessibility in research. In 2022, NTU Library introduced the Open Research Checklist and one of the Checklist items is Open Methods. Making methods open, transparent, and accessible to the public is known as Open Methods. Open methods promote trust, reproducibility, and reusability (1).
As part of NTU Library’s initiatives to advocate for broader adoption of open research practices, we invited Dr. Crystal Steltenpohl from the Centre for Open Science (COS), to share with the NTU research community on open methods. COS is a non-profit technology organization dedicated to increasing the openness, integrity, and reproducibility of scientific research. Its Open Science Framework (OSF) is open-source software designed to facilitate open methods and collaboration in scientific research. Crystal, based in the United States, conducted the two hybrid workshops on 27 and 29 February 2024 via Zoom.
Workshop 1: Finding and Writing Open Methods
In the first workshop titled: ‘Finding and Writing Open Methods’, Crystal provided an overview of open science practices and their contributions in making research more available, reproducible, and impactful by sharing data, methods, and results. This workshop also geared towards orientating NTU, NIE staff and students to the OSF project management tool to support researchers in the entire project lifecycle.
Participants were given the opportunity to explore the platform and many found it useful as it integrates project files, data, and protocols, and enhances collaboration through connections to third-party services like Dropbox and Google Drive. OSF centralizes locations for researchers, facilitating collaboration among those using different types of software. It offers 5GB of storage for private projects and up to 50GB per component for public projects. Crystal recommended adding more components to increase storage, noting there are no limits to the number of components per project.
During the hands-on activity, participants were encouraged to search for reproducible protocols in their research field, to reuse. Notable platforms they searched in were protocol exchange, protocols.io and PLOS ONE. One of the participants pointed out an observation – that whilst some repositories may have detailed protocols information, others were less comprehensive, as what he encountered. Crystal encouraged participants to consider what information they would need to replicate research and what might be missing from existing protocols. She also introduced tools like SciScore and Penelope.ai to check for protocols and manuscripts for transparency.
Key Questions and Discussions
Chemical Engineering Protocols:
“Can chemical engineering methods and procedures be found in protocols.io?”
Crystal noted that protocols.io might not be commonly used by chemical engineering researchers and offered to help find suitable repositories.
Editing Public Registrations:
“For the registration process, when the documents are uploaded into the OSF, this is usually done at the data analysis stage. However, if we do upload it and make it public, and during the process of the analysis, we discovered that this method that we chose initially may not be suitable. Is there a way to edit the process in OSF, once this information has been made public?”
When asked about updating methods during data analysis, Crystal explained that updates should be included in Registrations with justifications.
Linking ELN to Projects:
“What are the best practices surrounding linking ELN to the project? Is it via a link to the ELN?”
Crystal suggested using OSF as an Electronic Lab Notebook (ELN) or exporting ELN files to include them as components in OSF.
Crystal ended the session with the hands-on activity, where participants were tasked to create a test protocol, and share it with their collaborators. She emphasized that registrations can be assigned to existing projects or new ones, regardless of the project’s status.
Workshop 2: Preregistration and Registered Reports
The second workshop led by Crystal was a deep dive into one of the key principles of Open Science – Preregistration and Registered Reports. The importance of this topic was evident as it is key in encouraging authenticity and integrity in a researcher’s work. By publicly registering a study design before conducting the research, researchers are less likely to change their analysis after the data collection.
Preregistration and registered reports have been used by researchers doing experiments, randomised clinical trials, descriptive studies, qualitative studies, systematic reviews and others. Even if a researcher is doing exploratory research, they can still preregister, and capture the research questions and the variables they will be studying.
But more importantly, the trainer pointed out that Preregistration is particularly important in addressing publication bias. Studies with positive results are more likely to be published than those with negative ones. The other advantage of registering a study is that information is publicly available. There was a question that shared concern on their ideas getting scooped, to which the trainer assured that the preregistration can be embargoed, until it is ready to be published and made public. The length of time that the preregistration can be embargoed on OSF is up to four years. She added that on the contrary, preregistration allows the researcher to put a stake on their ideas.
One of the questions posed by the participants is if there are any limitations or risks of doing preregistrations? Crystal discussed the risks and limitations of preregistration, such as potential misunderstandings by reviewers unfamiliar with the process. She noted that fields like psychology are becoming accustomed to preregistrations, but qualitative researchers and those in education may find it challenging.
The invited speaker, Dr Pierina Cheung, shared a personal insight, where she recently had some experience trying to submit a paper with a preregistration to an early childhood education journal, and the editor and one of the reviewers asked what a preregistration is in the reviews. She added that, in response to the question raised by the participant, she agrees that general education is moving towards the preregistration direction but there re subfields that are still pretty new to it, and she notes that in early childhood a lot of the studies are mixed methods and qualitative.
For the rest of the session, Crystal ran through the steps to writing a preregistration and shared templates that can be found on OSF, and websites such as AsPredicted.org. The OSF currently has templates for qualitative research, psychological replications, registered reports, secondary data analysis, social psychological research, and systematic reviews. For the next part, participants learned how to register and update a study on the OSF, via a hands-on activity, while Crystal did a demo, using the various templates on OSF.
The second part of the workshop was a sharing by Dr Pierina Cheung, a lecturer (Research Scientist) in the Office of Education Research at the National Institute of Education (NIE). She shared with the audience the practical aspects of using open science practices in research. She started using preregistrations in 2017/2018, as a post doctorate at a university in Canada.
Pierina shared that one difficulty that she personally experienced when it comes to designing the research projects is the study part that you have an idea but detailing it and turning it into something measurable. Or, as someone mentioned earlier, specifying your study design, and fleshing that out, making sure your study design matches you research questions.
The goal of adopting open science practices, as reiterated by Pierina, is to enhance transparency in scientific research.
For more information on Open Methods, do check out our Open Methods LibGuide.
Reference:
(1) Public Library of Science (PLOS). Open Methods.
