A Maturity Model for Operations in Neuroscience Research

Abstract

Scientists are adopting new approaches to scale up their activities and goals. Progress in neurotechnologies, artificial intelligence, automation, and tools for collaboration promises new bursts of discoveries. However, compared to other disciplines and the industry, neuroscience laboratories have been slow to adopt key technologies to support collaboration, reproducibility, and automation. Drawing on progress in other fields, we define a roadmap for implementing automated research workflows for diverse research teams. We propose establishing a five-level capability maturity model for operations in neuroscience research. Achieving higher levels of operational maturity requires new technology-enabled methodologies, which we describe as “SciOps”. The maturity model provides guidelines for evaluating and upgrading operations in multidisciplinary neuroscience teams.

Keywords DevOps  $\cdot$ DataOps  $\cdot$ MLOps  $\cdot$ SciOps  $\cdot$ Capability Maturity Model  $\cdot$ neuroscience  $\cdot$ operations research  $\cdot$ open science  $\cdot$ closed-loop experiments  $\cdot$ digital twin  $\cdot$ FAIR  $\cdot$ reproducible research  $\cdot$ automated workflows  $\cdot$ artificial intelligence

1 Scaling operations in neuroscience research

Neuroscience, driven by technological advancement, stands on the cusp of groundbreaking discoveries. Current techniques enable the capture and modulation of intact neural circuit activity during complex behaviors, aligning these recordings with detailed molecular and anatomical maps. Advanced statistical tools unlock computational patterns in this vast data. Neuroscientists are increasingly coming together in large teams to develop and implement these new techniques at scales beyond the individual laboratory. However, larger projects often undergo chaotic phases as diverse teams strive to integrate disparate technical and organizational methods. The central question looms: How can advancements in neurotechnologies be translated into substantial scientific breakthroughs?

To keep pace with neurotechnologies, research teams require a revolution in their scientific operations. As the complexity of coordinating data and computation grows, it is crucial to boost collaboration, streamline processes, reduce errors, promote data integration and prioritize reproducibility. A new catalyst for discovery is the integration of artificial intelligence into research activities—a prospect that demands a formalized approach to scientific workflows^{1, 2}. Through enhanced operations, teams can integrate and navigate vast data, blending human ingenuity with emerging artificial intelligence. As the challenges increase, a clear roadmap becomes vital, steering research teams from their current practices towards greater capacities for ambitious projects.

2 Capability Maturity Model for Scientific Operations

To address the question of research teams’ readiness to tackle complex challenges with robust methods, we introduce the Capability Maturity Model for Scientific Operations (Fig. 1). Capability Maturity Models (CMMs) have a well-established history in engineering and related fields, providing structured frameworks for assessing and enhancing processes within organizations³. These frameworks are commonly employed by corporations and governments to evaluate contractors and strategically plan for the future. Enhancing operational maturity equips teams to embark on ambitious initiatives aimed at solving complex problems.

Our inspiration for this model derives from the CMMI (Capability Maturity Model Integration) family of models, initially developed by the Software Engineering Institute at Carnegie Mellon University for software engineering and subsequently adapted to various industries⁴. Drawing on our extensive experience in coordinating large-scale research collaborations in neuroscience, we have tailored key concepts from CMMI to the unique challenges and opportunities presented in current neuroscience projects. While our primary focus remains on experimental neuroscience, the core principles we define here can apply broadly in other disciplines.

Our model places research teams into five maturity levels based on their approach to planning and executing essential activities. This categorization includes multiple criteria: team structure, formal processes, training, software development, data management, and computational infrastructure and procedures. The model can act as a step-by-step guide, assisting teams in identifying essential steps to expand their capabilities.

Figure 1: The Capability Maturity Model for Scientific Operations in neuroscience research (Neuro SciOps CMM v1.0). We define SciOps as a set methodologies for implementing advanced capabilities.

Today, most neuroscience teams hover between Levels 1 and 2, though some specialization occurs among subfields. For instance, scientific operations in human neuroimaging tend to be more mature than in experimental neurophysiology⁵. Funding policies and publisher mandates for open data and reproducible results are nudging teams towards Level 3, which emphasizes community standards. By Level 4, teams prioritize research automation, scalable computing, and efficient team workflows. This approach is most commonly achieved in larger, centralized institutions, but can be adapted to smaller teams through better tools and platforms. However, the peak of Level 5, promising a significant leap in discoveries by closing the discovery loop with the help of artificial intelligence, remains a goal yet to be achieved by any team. We delve into each level in the sections below.

2.1 Level 1: Initial

At the outset of scientific endeavors, such as establishing a new laboratory, embarking on a Ph.D. project, or initiating a novel experiment, teams typically find themselves at Level 1 of the maturity model. This stage is characterized by a high degree of flexibility and customization in experimental and analytical methods. Data volumes at Level 1 are typically limited, not requiring high throughput. Each project adopts tailored approaches, with custom software and manual data management on dedicated lab infrastructure. Standardized methods are largely absent at this stage.

Experimentation without rigid methods, often described as “tinkering,” has always been foundational to research⁶. Remarkable achievements, including those recognized with Nobel Prizes, have commonly emerged from Level 1 practices. Nevertheless, in the context of today’s technology-driven and data-intensive projects, there is a growing need for more organized, open, and reproducible methods. Level 1 practices make it difficult for individual labs to adapt their processes for larger collaborations. Operational maturity should align with the complexity and scope of the research being conducted, much like how biosafety levels are established to match research demands.

While maintaining space for intuition-driven and flexible activities is crucial at all maturity levels, successful methods and practices can be standardized and scaled up as research demands increase. Moreover, a team’s maturity level is not necessarily a reflection of its size or its funding level. The ultimate aim of this model is to provide research teams with the necessary tools and guidance to elevate their operational maturity to meet the evolving demands of their research.

2.2 Level 2: Managed

To expand their scope and tackle larger, more complex projects, research teams must progress to Level 2, where the emphasis shifts to develo** lab-wide standards. These standards enhance consistency and predictability in internal operations, thereby facilitating more effective teamwork and leading to more credible findings.

Key characteristics of Level 2 teams include:

1. Repeatable Processes: Laboratories establish uniform methods applicable across various projects. This standardization extends to data management with shared storage, standardized formats, and structured naming conventions, ensuring data integrity. Software practices also evolve, incorporating version control, documentation, testing, and code review processes ^{7, 8, 5}. A notable practice at this level is the development and maintenance of a stable data pipeline, optimized for efficient, long-term use.

2. Role Specialization: Level 2 teams clearly define team roles and responsibilities. This specialization allows for an efficient distribution of tasks and maximizes the use of individual expertise. A common practice is rotating trainees through different roles to provide a comprehensive understanding of lab operations.

3. Quality Control: Level 2 introduces rigorous procedures for continuously monitoring and validating the accuracy and reliability of experimental results. These include instrument calibration, software testing, and signal quality assessment. Quality control criteria are established and periodically updated to ensure the highest standards.

4. Training Programs: Level 2 teams provide a structured onboarding process for new members as well as ongoing training initiatives. These programs aim to keep the team proficient in lab operations and abreast of the latest developments. Some labs extend their training efforts beyond their teams, hosting seminars and workshops for external groups to disseminate their standardized practices.

The progression to Level 2 marks a significant step towards operational maturity, characterized by a systematic approach to research, a focus on quality and reliability, and an emphasis on continuous learning and improvement. However, uniting multiple Level 2 labs within a consortium for collaborative projects poses significant challenges owing to mismatching processes employed in each lab.

2.3 Level 3: Defined

Level 3 marks a stage where research teams embrace practices conducive to robust collaborations spanning various laboratories and disciplines. Level 3 labs excel in joining forces within multilab consortia, streamlining interaction, and harmonizing data processes. Key features of this level include the adoption of common data standards and interoperable computational frameworks, essential for efficient use and sharing of data and software among teams.

Key characteristics of Level 3 teams are:

1. Open-Source Ecosystems: Level 3 teams are deeply engaged with resilient open-source software. These ecosystems are not just about providing tools that are “free to use” but also involve responsive community governance setting standards for quality, reliability, and reproducibility. They form the foundation for state-of-the-art projects and offer community support and educational resources. Level 3 teams align their work with these community-driven open-source endeavors, promoting consistency, integration, reliability, reproducibility, and sustainability.

   These teams adopt disciplined practices for software management, including principled code management, peer review, validation, and testing. This approach enables the creation of evolving data pipelines and computational workflows with minimal downtime, ensuring continuous research progression.

2. FAIR Data: Level 3 teams develop, adopt, and promote harmonized initiatives for data standards, fostering interoperability of tools and processes across research groups. This includes adherence to the FAIR principles (Findable, Accessible, Interoperable, and Reusable) for scientific data⁹. Such standards facilitate the exchange and reuse of complex data, supported by robust systems including data sharing platforms that enable reproducibility and re-analysis. Examples in neuroscience include data standards around the BIDS¹⁰ and NWB projects¹¹.

   Data exchange and reuse requires infrastructure. Robust data sharing platforms and collaborative research environments are set up to facilitate joint research endeavors. Neuroscience data archives such as DANDI¹², BrainLife¹³, BossDB¹⁴, and OpenNeuro¹⁵ not only store the data but also facilitate reproducibility and new analysis. Distributed data management systems such as DataLad, which relies on git-annex, facilitate not only versioning of data but also unified data access and exchange across multiple work sites and archives^{16, 17}.

   FAIR principles foster efficient work for both humans and machines. Seamless automation and effective machine learning rely on machine readability enabled by FAIR data standards¹⁸. On a global scale, the collective output of Level 3+ research contributes to a semantic web of datasets and methods, enabling further aggregation of knowledge.

3. FAIR Workflows: Level 3 teams elevate the application of FAIR principles beyond data management to encompass the entirety of computational workflows, tracking all data transformations from the raw acquired data through processing and analysis to the figures in a paper^{19, 20}. These practices manage the associated code versions, dependencies, environment configurations, and parameters. Data outputs contain their provenance information describing their lineage from the original inputs through all computational transformations. This comprehensive approach ensures that computational analyses are repeatable and shareable, all while minimizing the potential for human error.

   Formal workflows incorporate best practices for testing code logic, including unit, regression, and integration testing, often utilizing benchmark datasets. This rigorous testing framework promotes the reliability and trustworthiness of research results.

   The assimilation of formal workflow specifications has been disparate across scientific domains. Geosciences and bioinformatics have been at the forefront, rapidly adapting and benefiting from these advancements. Neuroscience, especially outside human neuroimaging, has been more reticent, perhaps because of its vast array of data modalities and the inherent challenges in achieving standardization²¹.

Level 3 teams, exemplified by the International Brain Lab and participants in the NIH U19 program, demonstrate how open science practices significantly enhance operational capacity²². They view open science not as a burden but as an opportunity to become more efficient, credible, and accessible in their research endeavors. This level represents a significant step towards a more integrated and collaborative scientific community, where shared knowledge and resources propel research to new heights.

2.4 Level 4: Scalable — Introducing SciOps

Levels 4 and 5 transform research operations by adopting technologies for automation, scalability, and efficient collaboration. We collectively refer to these methodologies as SciOps, aligning with the principles of other successful “Ops” disciplines in the technology industry: DevOps, DataOps, and MLOps.

In the software industry, DevOps cultivates a seamless, semi-automated pipeline for software development and operations^{23, 24, 25}. DevOps relies on a set of enabling technologies: containerization, version control, infrastructure as code (IaC), continuous integration / continuous deployment (CI/CD), logging and monitoring. Repetitive tasks are automated, transitions are minimized, computational resources are meticulously orchestrated, and processes are continuously monitored and rendered observable. The core driver of productivity in DevOps is the integration of collaborating teams’ activities into a unified automated workflow. Rather than delivering their outputs separately, these teams contribute to a joint automated pipeline, guided by well-defined processes. Elimination of hand-offs between teams reduces errors and shortens development cycles.

Similarly, the field of data analytics has scaled up operations by extending DevOps into the methodologies of DataOps^{26, 27, 28} and MLOps²⁹. These strategies streamline teamwork in data gathering, processing, and analysis and accelerate the development and deployment of machine learning models. They encourage teamwork among data specialists through clear communication, integration, and automation, resulting in more accurate and timely insights.

The emergence of SciOps was surveyed in a 2022 National Academies consensus study on Automated Research Workflows², spanning various data-intensive research domains with one notable exception: neuroscience. These workflows represent the convergence of computation, lab automation, and artificial intelligence, spanning across the entire research cycle: experiment design, observations, simulations, data collection and analysis, and learning from results to inform further experiments and simulations.

Level 4 leverages technology to dramatically scale and streamline coordinated activities of multidisciplinary teams collaborating on complex projects. Its key capabilities include:

1. Experiment Automation: Level 4 teams automate experimental processes, including data collection and analysis, including experiment execution, where instruments for experiment control, stimuli, sensors, and data acquisition are integrated into the data pipeline and workflow management.

2. DataOps: Level 4 teams implement standard practices for optimizing data management, processing, and analysis while integrating advanced computational infrastructure. The primary focus is on achieving efficiency, scalability, maintaining quality control, and enabling seamless collaboration.

   At this level, each project is supported by a meticulously defined shared data pipeline. This data pipeline serves as a formalized process for aggregating primary data while ensuring consistency and integrity, acting as the authoritative source for all downstream activities. As teams progress to higher maturity levels, the data pipeline extends to encompass all critical phases of the data lifecycle.

   These automated data pipelines exhibit adaptability and transparency, allowing for the continuous evolution of data analysis methods through collaborative endeavors. Workflow logic undergoes ongoing validation through automated testing to minimize the risk of human errors. Software and system design are structured to efficiently handle large data volumes, guaranteeing fast data access and movement. Achieving elastic scalability often involves leveraging cloud and high-performance computing infrastructures, in conjunction with established best practices for software development and operation.

3. Collaboration Environments: Level 4 teams establish streamlined collaboration environments that enable diverse, distributed teams to access data exploration capabilities. This includes the use of web-based environments for exploratory analysis and knowledge exchange, user-friendly interfaces for data import/export, and the use of community data archives and software repositories³⁰.

4. Teamflow: Level 4 teams establish efficient project management and communication strategies to support large-scale, multidisciplinary teams. Project management frameworks are adopted to lower the barriers to participation. They prioritize project continuity and fair credit tracking for individual contributions. These teams foster open, transparent, and efficient communication while ensuring data consistency and integrity. Leadership and mentorship activities promote flexibility and adaptability. Integration of rapid software development practices, efficient data management techniques, and collaborative tools enable teams to work effectively and cohesively within a complex and scalable research environment.

Only a select few neuroscience research teams approach Level 4 maturity as the requisite innovations are currently in development. Among those that have made progress in this direction are well-funded institutes and multi-institution consortia, including the Allen Institute for Neural Dynamics, e11 Bio, major BRAIN Initiative initiatives like BICCN ³¹ and MICrONS ³², as well as the International Brain Laboratory ³³. The Virtual Brain Project, another example, enables Level 4 operations through open-source containerized cloud services for multiscale brain simulation and magnetic resonance image processing to enable scalable and reproducible research³⁴. Open cloud platforms that support automated data processing and management, such as brainlife.io¹³, can provide open and secure environments for researchers to collaborate and implement reproducible data analysis. The platforms Virtual Research Environment³⁵ and EBRAINS³⁶ with its federated Health Data Cloud³⁷ peer-to-peer networks³⁸ also serve as collaborative research environments while protecting data privacy in building human digital twins for medical research. These entities are at the forefront of neuroscience data scale and collaboration towards shared research goals and Level 4 operations. A key challenge for the field is to create an ecosystem of tools, standards, and platforms to enable diverse research teams of all sizes and funding levels to adopt these scalable approaches.

2.5 Level 5: Optimizing

The ultimate aim of SciOps is the seamless integration of artificial intelligence (AI) with human cognition, creating a dynamic partnership that continually refines experiment design and enhances knowledge synthesis through concurrent computational modeling. Level 5 is distinguished by the strategic application of advanced technologies and AI to establish a “closed discovery loop.” This entails the formalization and automation of the process of using insights derived from data analysis to actively guide subsequent decisions, ultimately sha** the direction of scientific studies.

Key components establishing AI-accelerated closed-loop studies include:

1. Adaptable experiments: Experiment designs are highly configurable, allowing rapid adaptation to new scientific questions. Parameterized controls include sensory and optogenetic stimulation, recording protocols, and cognitive tasks.

2. Machine learning in the loop: Closed-loop experiments create a continuous automated feedback loop where machine learning algorithms optimize experiment controls to maximize knowledge gain about brain structure and function. This may involve a “digital twin” paradigm, where a simulation of the brain is continuously refined to match recorded data, facilitating concurrent in silico experiments. These closed loops can span multiple temporal scales: from real-time control of experiments to long-term adaptations exploring the hypothesis space, effectively guiding the scientific study.

   Machine learning models powering closed-loop discovery will undergo rapid development, requiring adopting industry-proven MLOps practices whereby new models are tested and deployed continuously.

   Integrated AI systems observe trends in data, identify new phenomena, propose hypotheses and new experiments. Future generative AI systems will consider various information sources, including current literature, existing open datasets, and ongoing discourse between researchers, embedding this knowledge into the data pipeline and providing decision support for further inquiry¹.

3. Human in the loop: While closed-loop experiments automate many processes, human input and insight remain integral. Level 5 workflows feature a symbiotic relationship between AI tools and human ingenuity. This hybrid approach enables rapid hypothesis generation, adaptation of experimental procedures, effective quality control, and the rapid analysis of large-scale data collections. To support human participation, the data pipeline and experiment controls are made observable and explainable.

While no research team in neuroscience (or perhaps in any other discipline) has fully realized Level 5 operations, existing closed-loop experiments showcase elements of this vision through the application of machine learning to optimize experimental conditions based on functional recordings³⁹. These projects involve the analysis of extensive neurophysiology data in large-scale collaborations to train deep neural networks for stimulus generation in a closed-loop manner.

Level 5 of the Neuroscience Operations Maturity Model represents the forefront of neuroscience research, where AI takes center stage, driving innovation and enabling researchers to tackle complex questions with unprecedented efficiency and precision. It will signify a transformative leap in the field’s capabilities and potential for groundbreaking discoveries.

3 What We Need to Advance Modern Scientific Operations in Neuroscience

The evolution of scientific practices, coupled with rapid technological advancements, demands a strategic shift in how we approach research, especially in the context of experimental neuroscience. The transformational power of SciOps methodologies has the potential to reshape the way we think about scientific endeavors. However, to tap into this potential, we must focus on specific areas of development and innovation:

3.1 Action 1: Adopt this Capability Maturity Model

Community Governance: We invite the community to embrace, enhance, and collaboratively govern this Capability Maturity Model. This version, designated as Neuro SciOps CMM v1.0, serves as a starting point, and we are committed to establishing a roadmap and guidelines for contributions, working in coordination with the International Neuroinformatics Coordinating Facility (INCF)⁴⁰.

A centralized community resource will be established through the INCF and other platforms to provide access to the maturity model and related resources. This resource will serve as a valuable reference point for individuals and organizations interested in advancing scientific operations. To facilitate community engagement, we envision the creation of a dedicated SciOps resource and a working group to administer and support the framework and its collaborative development.

Assessment and Roadmaps: The model will support assessment and certification processes, enabling organizations to prepare their processes for various projects and programs. It will also serve as a valuable tool for charting organizational improvements and technological roadmaps.

By embracing this Capability Maturity Model and actively participating in its development and application, the scientific community can collectively advance the field of scientific operations, foster innovation, and drive transformative progress in research methodologies and practices.

Socialize expectations across the neuroscience community: For this model to be adopted, the expectations around how neuroscience needs to be performed to meet 21st century goals will have to be socialized across the neuroscience community, through integration with training programs, townhall meetings and workshops at major scientific gatherings.

3.2 Action 2: Establish SciOps Methodologies

Current standards and policies in neuroscience data are focused on standardization and public data sharing, marking a progression towards Level 3 maturity within our model. Achieving Levels 4 and 5 will require new tools, “SciOps methodologies”, which adapt DevOps principles to the context of neuroscience experiments. Many practices can be transferred from collaborative scientific workflow management systems that have gained prominence in bioinformatics^{21, 41, 42, 43} and from industry DevOps and DataOps open-source frameworks and commercial platforms⁴⁴.

Experiment automation with DevOps, DataOps, and MLOps: Workflow management technologies should seamlessly integrate with neuroinformatics tools and methods, setting the stage for more scalable operations as defined in Levels 4 and 5 of our model. The evolution of neuroinformatics tools should prioritize continuous integration and deployment of research software—whether commercial or community-driven— encompassing experiment control, data acquisition, and analysis. Data should be made available in formats and on infrastructure that allow for scalable storage, processing, and sharing. With the constant flux and scaling of data, formats and infrastructure evolve rapidly, making it critical to establish lasting organizational principles to ensure project continuity. Furthermore, experimental workflows should integrate formal frameworks for embedding artificial intelligence into the discovery loop.

The creation and widespread adoption of innovative tools tailored specifically for SciOps have the potential to simplify intricate processes, streamline tasks, and significantly enhance research efficiency. By embracing this approach, we can usher in a transformative era of scientific operations characterized by heightened methodological prowess and overall efficacy.

3.3 Action 3: Focus on cyberinfrastructure and digital platforms

Digital platforms have the potential to elevate research teams to Level 3+ maturity without the need for extensive investments in engineering expertise and custom solutions by individual projects⁴⁵. However, in the academic landscape, digital platforms often suffer from limited usability, creating the perception that shared infrastructure can impede daily activities, presenting a barrier for sustained adoption. Consequently, some teams opt for building their own solutions rather than embracing centralized platforms endorsed by their communities. Unfortunately, these custom-built infrastructures frequently fall short in terms of service quality, performance, customer support, and long-term sustainability.

A new generation of academic neuroinformatics platforms, including BrainLife ¹³, DABI ⁴⁶, OpenNeuro ¹⁵, SPARC ⁴⁷, and DANDI ¹¹, are positioned to replicate the success of well-established platforms in the fields of bioinformatics and biomedical research such as the Galaxy platform, for example⁴⁸. To do so, they must evolve beyond data sharing (Level 3) to the automation and scalability of workflows (Level 4+).

On the other hand, commercial platforms, driven by competitive pressures, are better positioned to deliver usability, robust customer support, and service continuity. In the field of neuroscience, emerging commercial platforms such as DataJoint Works, Inscopix IDEAS, and CodeOcean provide support for integrated research workflows.

To ensure the success of these platforms, it is essential that they integrate seamlessly with FAIR data and FAIR workflows, emphasizing transparency, accessibility, and interoperability of data and computational services. The widespread adoption of this Capability Maturity Model can assist commercial technology providers in aligning their business models with the overarching objective of expanding research capabilities and enhancing the user experience. Given the diverse nature of the field, it is unlikely that a single platform or toolset will dominate. Consequently, diverse groups with common goals and roadmaps can form alliances around shared standards and open-source frameworks, promoting interoperability, transparency, and reproducibility across their respective platforms.

To realize this multifaceted vision, it is essential to establish a comprehensive strategy that encompasses a blend of academic projects, commercial technology initiatives, and consortial activities, all in alignment with funding policies. This multifaceted approach will guide the formulation of research projects, the marketing efforts of commercial technology providers, and the policies of funding agencies, ultimately fostering the creation of a unified and sustainable ecosystem.