Labvanced and the Spirit of Open Science
Since Labvanced is a subscription-based service, this, to many people, may sound like it is not aligned with open science. But this is far from the case. Labvanced, by the very nature of how it is set up and how it allows researchers to create and publish their experiments, not only embraces the spirit of open science, but also helps drive the best practices of conducting online experimental research.
Public Experiment Library and Open Science
Labvanced has several avenues for researchers who are interested in making their online studies as freely accessible and shareable as possible, one such being the Public Experiment Library. Via the Public Experiment Library, researchers can publish their experiment so anyone who wishes to participate can do so. Upon publishing the experiment, the researcher can specify to what extent the study can be used as a template (ie. can be imported by other subscribed users), but also how ‘open access’ looks like for non-Labvanced users - like if only the design / experiment setup can be viewed or if the collected data is also accessible openly.
The Public Experiment Library is one of the largest open science / open experiment libraries in existence and is constantly growing with over 600 studies listed from external institutions around the globe. This enables reviewers to ‘inspect’ the implementation (even without an account) and see the data that was collected (if you have this option selected to have it publicly available / open). We have seen a lot of researchers providing the Labvanced link in their publications in high-impact journals for this purpose.
Also note, that researchers can publish a study in the Public Experiment Library without any cost; the study will be there forever (or as long as the researcher allows it to be). This means that experiments can be created freely and that no other researcher has to pay for inspecting / reviewing your work. We believe that such a higher degree of openness and transparency is yet to be seen anywhere else.
Reproducibility / Reusability
Sharing or exporting the experiment in some kind of containerized form is important for researchers who may want to demonstrate or communicate their experiment design with other researchers without relying on the Public Experiment Library.
All experiments conducted inside Labvanced have the highest degree of reproducibility because the exact same code is used in every single one of them. This is because all study editing / study implementation that researchers do while using Labvanced, will create a JSON data file (ie. experiment specification) that is then “played / run” by the Labvanced experiment player engine, so that the functions executed are the same for all studies across the platform.
Typically, what programming-based experiments rely on is some kind of “custom React/Python/self-hosting solutions” where each complex experiment is kind of its own codebase. And while surely researchers make use and have some code that can be reusable across their studies (ie. components), there is usually a much higher variance in terms of code / logic between these kinds of studies compared to between Labvanced-based studies, which may ultimately diminish reproducibility.
Ultimately, having the same basis of code across studies (like what Labvanced does) is an approach that ultimately helps with scaling, time management, and also creating some kind of framework, as opposed to writing projects again and again in a purely custom way. Of course, being experimental psychology researchers ourselves, Labvanced gives its users the opportunity to code and in those cases there is an accessible JavaScript editor and also for CSS within the app.
The actual tricky part regarding reusability is that there is no universally accepted experiment protocol / experiment format for online studies, especially not when it comes to capabilities such as webcam-based eye tracking, multiplayer support and so on.
The only existing global web standard is in the domain of data storage and data exchange for which JSON is used… and this is exactly what we are using here. Importantly, Labvanced users can export the entire JSON experiment specification at any time and for free, with one click, meaning you can download, save and share your experiment structure (independent of Labvanced) in a standardized JSON format.
As mentioned previously, the JSON format essentially captures the experimental structure. So, for example, if you have a command in Labvanced to ‘record gaze on frame start’ that would be reflected in the exportable/sharable JSON code by something like this (note: this is a simplified example):
event: { trigger: onNewGaze actions [ {setRecVar:{ varID: "j234o234h234234" }} ] }
Thus, on the topic of reusability and reproducibility, experiments built with Labvanced can be communicated beyond the Public Experiment Library by exporting a JSON file where the experiment structure can be shared with others.
Open Source
A lot of the source code behind Labvanced is actually open source and can be seen on Github (https://github.com/Labvanced/) for transparency reasons. Certain parts, however, like Labvanced’s proprietary webcam-based eye tracking technology and neural networks, are not included there. This is because we have to protect our years’ worth of investments in creating the most accurate webcam-based eye tracking technology. It would just be stolen otherwise, and we literally have seen attempts in this direction from both smaller and bigger companies.
Conclusion
It is very common for labs or students to want to rely on some kind of combination of PsychoJS+Pavlovia for simpler experiments, or even go down the road of creating fully custom React/Python/self-hosting solutions for more complex experiments and then proceed share their code as a part of their Open Science protocol.
While programming a study can be a great way to go (we are programmers ourselves), it does come with challenges. Of course it can make sense to program a study completely by yourself, especially if you have IT professionals who are doing that kind of work for complex, large-scale research.
In our opinion it is often more efficient and economical in the long run to use a platform like Labvanced for because altogether we already invested 20+ developer years (ie. 3-4 developers over the course of 7+ years) in writing code that creates powerful (web) experiments in one streamlined process, all while ensuring precise stimuli presentation and giving the researchers to monitor online-specific variables for experimental control.
All-in-all, Labvanced supports open science and transparency and offers several options for how researchers can share and export their studies with other professionals in their field.