-
Individual context-free online community health indicators fail to identify open source software sustainability
Authors:
Yo Yehudi,
Carole Goble,
Caroline Jay
Abstract:
The global value of open source software is estimated to be in the billions or trillions worldwide1, but despite this, it is often under-resourced and subject to high-impact security vulnerabilities and stability failures2,3. In order to investigate factors contributing to open source community longevity, we monitored thirty-eight open source projects over the period of a year, focusing primarily,…
▽ More
The global value of open source software is estimated to be in the billions or trillions worldwide1, but despite this, it is often under-resourced and subject to high-impact security vulnerabilities and stability failures2,3. In order to investigate factors contributing to open source community longevity, we monitored thirty-eight open source projects over the period of a year, focusing primarily, but not exclusively, on open science-related online code-oriented communities. We measured performance indicators, using both subjective and qualitative measures (participant surveys), as well as using computational scripts to retrieve and analyse indicators associated with these projects' online source control codebases. None of the projects were abandoned during this period, and only one project entered a planned shutdown.
Project ages spanned from under one year to over forty years old at the start of the study, and results were highly heterogeneous, showing little commonality across documentation, mean response times for issues and code contributions, and available funding/staffing resources. Whilst source code-based indicators were able to offer some insights into project activity, we observed that similar indicators across different projects often had very different meanings when context was taken into account.
We conclude that the individual context-free metrics we studied were not sufficient or essential for project longevity and sustainability, and might even become detrimental if used to support high-stakes decision making. When attempting to understand an online open community's longer-term sustainability, we recommend that researchers avoid cross-project quantitative comparisons, and advise instead that they use single-project-level assessments which combine quantitative measures with contextualising qualitative data.
△ Less
Submitted 9 May, 2024; v1 submitted 21 September, 2023;
originally announced September 2023.
-
Tracking the Structure and Sentiment of Vaccination Discussions on Mumsnet
Authors:
Miguel E. P. Silva,
Rigina Skeva,
Thomas House,
Caroline Jay
Abstract:
Vaccination is one of the most impactful healthcare interventions in terms of lives saved at a given cost, leading the anti-vaccination movement to be identified as one of the top 10 threats to global health in 2019 by the World Health Organization. This issue increased in importance during the COVID-19 pandemic where, despite good overall adherence to vaccination, specific communities still showe…
▽ More
Vaccination is one of the most impactful healthcare interventions in terms of lives saved at a given cost, leading the anti-vaccination movement to be identified as one of the top 10 threats to global health in 2019 by the World Health Organization. This issue increased in importance during the COVID-19 pandemic where, despite good overall adherence to vaccination, specific communities still showed high rates of refusal. Online social media has been identified as a breeding ground for anti-vaccination discussions. In this work, we study how vaccination discussions are conducted in the discussion forum of Mumsnet, a United Kingdom based website aimed at parents. By representing vaccination discussions as networks of social interactions, we can apply techniques from network analysis to characterize these discussions, namely network comparison, a task aimed at quantifying similarities and differences between networks. Using network comparison based on graphlets -- small connected network subgraphs -- we show how the topological structure vaccination discussions on Mumsnet differs over time, in particular before and after COVID-19. We also perform sentiment analysis on the content of the discussions and show how the sentiment towards vaccinations changes over time. Our results highlight an association between differences in network structure and changes to sentiment, demonstrating how network comparison can be used as a tool to guide and enhance the conclusions from sentiment analysis.
△ Less
Submitted 24 August, 2023;
originally announced August 2023.
-
Novice programmers strategies for online resource use and their impact on source code
Authors:
Omar Alghamdi,
Sarah Clinch,
Mohammad Alhamadi,
Caroline Jay
Abstract:
Websites are frequently used by programmers to support the development process. This paper investigates programmer-Web interactions when coding, and combines observations of behaviour with assessments of the resulting source code. We report on an online observational study with ten undergraduate student programmers as they engaged in programming tasks of varying complexity. Screens were recorded o…
▽ More
Websites are frequently used by programmers to support the development process. This paper investigates programmer-Web interactions when coding, and combines observations of behaviour with assessments of the resulting source code. We report on an online observational study with ten undergraduate student programmers as they engaged in programming tasks of varying complexity. Screens were recorded of participants' activities, and each participated in an interview. Videos and interviews were thematically analysed. Novice programmers employed various strategies for seeking and utilising online knowledge. The resulting source code was examined to determine the extent to which it met requirements and whether it contained errors. The source code analysis revealed that coding with the websites involved more coding time and effort, but increased the possibility of producing correct code. However, coding with websites also introduced instances of either incorrect or non-executable source code.
△ Less
Submitted 20 April, 2023;
originally announced April 2023.
-
Subjective data models in bioinformatics: Do wet-lab and computational biologists comprehend data differently?
Authors:
Yo Yehudi,
Lukas Hughes-Noehrer,
Carole Goble,
Caroline Jay
Abstract:
Biological science produces large amounts of data in a variety of formats, which necessitates the use of computational tools to process, integrate, analyse, and glean insights from the data. Researchers who use computational biology tools range from those who use computers primarily for communication and data lookup, to those who write complex software programs in order to analyse data or make it…
▽ More
Biological science produces large amounts of data in a variety of formats, which necessitates the use of computational tools to process, integrate, analyse, and glean insights from the data. Researchers who use computational biology tools range from those who use computers primarily for communication and data lookup, to those who write complex software programs in order to analyse data or make it easier for others to do so. This research examines how people differ in how they conceptualise the same data, for which we coin the term "subjective data models".
We interviewed 22 people with biological experience and varied levels of computational experience to elicit their perceptions of the same subset of biological data entities. The results suggest that many people had fluid subjective data models that would change depending on the circumstance or tool they were using. Surprisingly, results generally did not seem to cluster around a participant's computational experience/education levels, or the lack thereof. We further found that people did not consistently map entities from an abstract data model to the same identifiers in real-world files, and found that certain data identifier formats were easier for participants to infer meaning from than others.
Real-world implications of these findings suggests that 1) software engineers should design interfaces for task performance and emulate other related popular user interfaces, rather than targeting a person's professional background; 2) when insufficient context is provided, people may guess what data means, whether or not their guesses are correct, emphasising the importance of providing contextual metadata when preparing data for re-use by other, to remove the need for potentially erroneous guesswork.
△ Less
Submitted 25 August, 2022;
originally announced August 2022.
-
Comparing directed networks via denoising graphlet distributions
Authors:
Miguel E. P. Silva,
Robert E. Gaunt,
Luis Ospina-Forero,
Caroline Jay,
Thomas House
Abstract:
Network comparison is a widely-used tool for analyzing complex systems, with applications in varied domains including comparison of protein interactions or highlighting changes in structure of trade networks. In recent years, a number of network comparison methodologies based on the distribution of graphlets (small connected network subgraphs) have been introduced. In particular, NetEmd has recent…
▽ More
Network comparison is a widely-used tool for analyzing complex systems, with applications in varied domains including comparison of protein interactions or highlighting changes in structure of trade networks. In recent years, a number of network comparison methodologies based on the distribution of graphlets (small connected network subgraphs) have been introduced. In particular, NetEmd has recently achieved state of the art performance in undirected networks. In this work, we propose an extension of NetEmd to directed networks and deal with the significant increase in complexity of graphlet structure in the directed case by denoising through linear projections. Simulation results show that our framework is able to improve on the performance of a simple translation of the undirected NetEmd algorithm to the directed case, especially when networks differ in size and density.
△ Less
Submitted 8 March, 2023; v1 submitted 20 July, 2022;
originally announced July 2022.
-
COVID-19: An exploration of consecutive systemic barriers to pathogen-related data sharing during a pandemic
Authors:
Yo Yehudi,
Lukas Hughes-Noehrer,
Carole Goble,
Caroline Jay
Abstract:
In 2020, the COVID-19 pandemic resulted in a rapid response from governments and researchers worldwide. As of late 2023, over millions have died as a result of COVID-19, with many COVID-19 survivors going on to experience long-term effects weeks, months, or years after their illness. Despite this staggering toll, those who work with pandemic-relevant data often face significant systemic barriers t…
▽ More
In 2020, the COVID-19 pandemic resulted in a rapid response from governments and researchers worldwide. As of late 2023, over millions have died as a result of COVID-19, with many COVID-19 survivors going on to experience long-term effects weeks, months, or years after their illness. Despite this staggering toll, those who work with pandemic-relevant data often face significant systemic barriers to accessing, sharing or re-using this data. In this paper we report results of a study, where we interviewed data professionals working with COVID-19-relevant data types including social media, mobility, viral genome, testing, infection, hospital admission, and deaths. These data types are variously used for pandemic spread modelling, healthcare system strain awareness, and devising therapeutic treatments for COVID-19. Barriers to data access, sharing and re-use include the cost of access to data (primarily certain healthcare sources and mobility data from mobile phone carriers), human throughput bottlenecks, unclear pathways to request access to data, unnecessarily strict access controls and data re-use policies, unclear data provenance, inability to link separate data sources that could collectively create a more complete picture, poor adherence to metadata standards, and a lack of computer-suitable data formats.
△ Less
Submitted 22 December, 2023; v1 submitted 24 May, 2022;
originally announced May 2022.
-
Using Interaction Data to Predict Engagement with Interactive Media
Authors:
Jonathan Carlton,
Andy Brown,
Caroline Jay,
John Keane
Abstract:
Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is important in at least two ways: (1) a post-hoc understanding of how engaged audiences are with the content will help production teams learn from experienc…
▽ More
Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is important in at least two ways: (1) a post-hoc understanding of how engaged audiences are with the content will help production teams learn from experience and improve future productions; (2), this type of media has potential for real-time measures of engagement to be used to enhance the user experience by adapting content on-the-fly. Engagement is typically measured by asking samples of users to self-report, which is time consuming and expensive. In some domains, however, interaction data have been used to infer engagement. Fortuitously, the nature of interactive media facilitates a much richer set of interaction data than traditional media; our research aims to understand if these data can be used to infer audience engagement. In this paper, we report a study using data captured from audience interactions with an interactive TV show to model and predict engagement. We find that temporal metrics, including overall time spent on the experience and the interval between events, are predictive of engagement. The results demonstrate that interaction data can be used to infer users' engagement during and after an experience, and the proposed techniques are relevant to better understand audience preference and responses.
△ Less
Submitted 4 August, 2021;
originally announced August 2021.
-
Number and quality of diagrams in scholarly publications is associated with number of citations
Authors:
Guy Clarke Marshall,
Caroline Jay,
Andre Freitas
Abstract:
Diagrams are often used in scholarly communication. We analyse a corpus of diagrams found in scholarly computational linguistics conference proceedings (ACL 2017), and find inclusion of a system diagram to be correlated with higher numbers of citations after 3 years. Inclusion of over three diagrams in this 8-page limit conference was found to correlate with a lower citation count. Focusing on neu…
▽ More
Diagrams are often used in scholarly communication. We analyse a corpus of diagrams found in scholarly computational linguistics conference proceedings (ACL 2017), and find inclusion of a system diagram to be correlated with higher numbers of citations after 3 years. Inclusion of over three diagrams in this 8-page limit conference was found to correlate with a lower citation count. Focusing on neural network system diagrams, we find a correlation between highly cited papers and "good diagramming practice" quantified by level of compliance with a set of diagramming guidelines. Two diagram classification types (one visually based, one mental model based) were not found to correlate with number of citations, but enabled quantification of heterogeneity in those dimensions. Exploring scholarly paper-writing guides, we find diagrams to be a neglected media. This study suggests that diagrams may be a useful source of quality data for predicting citations, and that "graphicacy" is a key skill for scholars with insufficient support at present.
△ Less
Submitted 30 April, 2021;
originally announced April 2021.
-
Why scholars are diagramming neural network models
Authors:
Guy Clarke Marshall,
Caroline Jay,
Andre Freitas
Abstract:
Complex models, such as neural networks (NNs), are comprised of many interrelated components. In order to represent these models, eliciting and characterising the relations between components is essential. Perhaps because of this, diagrams, as "icons of relation", are a prevalent medium for signifying complex models. Diagrams used to communicate NN architectures are currently extremely varied. The…
▽ More
Complex models, such as neural networks (NNs), are comprised of many interrelated components. In order to represent these models, eliciting and characterising the relations between components is essential. Perhaps because of this, diagrams, as "icons of relation", are a prevalent medium for signifying complex models. Diagrams used to communicate NN architectures are currently extremely varied. The diversity in diagrammatic choices provides an opportunity to gain insight into the aspects which are being prioritised for communication. In this philosophical exploration of NN diagrams, we integrate theories of conceptual models, communication theory, and semiotics.
△ Less
Submitted 10 June, 2022; v1 submitted 30 April, 2021;
originally announced April 2021.
-
Structuralist analysis for neural network system diagrams
Authors:
Guy Clarke Marshall,
Caroline Jay,
Andre Freitas
Abstract:
This short paper examines diagrams describing neural network systems in academic conference proceedings. Many aspects of scholarly communication are controlled, particularly with relation to text and formatting, but often diagrams are not centrally curated beyond a peer review. Using a corpus-based approach, we argue that the heterogeneous diagrammatic notations used for neural network systems has…
▽ More
This short paper examines diagrams describing neural network systems in academic conference proceedings. Many aspects of scholarly communication are controlled, particularly with relation to text and formatting, but often diagrams are not centrally curated beyond a peer review. Using a corpus-based approach, we argue that the heterogeneous diagrammatic notations used for neural network systems has implications for signification in this domain. We divide this into (i) what content is being represented and (ii) how relations are encoded. Using a novel structuralist framework, we use a corpus analysis to quantitatively cluster diagrams according to the author's representational choices. This quantitative diagram classification in a heterogeneous domain may provide a foundation for further analysis.
△ Less
Submitted 30 April, 2021;
originally announced April 2021.
-
Understanding Equity, Diversity and Inclusion Challenges Within the Research Software Community
Authors:
Neil P. Chue Hong,
Jeremy Cohen,
Caroline Jay
Abstract:
Research software -- specialist software used to support or undertake research -- is of huge importance to researchers. It contributes to significant advances in the wider world and requires collaboration between people with diverse skills and backgrounds. Analysis of recent survey data provides evidence for a lack of diversity in the Research Software Engineer community. We identify interventions…
▽ More
Research software -- specialist software used to support or undertake research -- is of huge importance to researchers. It contributes to significant advances in the wider world and requires collaboration between people with diverse skills and backgrounds. Analysis of recent survey data provides evidence for a lack of diversity in the Research Software Engineer community. We identify interventions which could address challenges in the wider research software community and highlight areas where the community is becoming more diverse. There are also lessons that are applicable, more generally, to the field of software development around recruitment from other disciplines and the importance of welcoming communities.
△ Less
Submitted 4 April, 2021;
originally announced April 2021.
-
Software must be recognised as an important output of scholarly research
Authors:
Caroline Jay,
Robert Haines,
Daniel S. Katz
Abstract:
Software now lies at the heart of scholarly research. Here we argue that as well as being important from a methodological perspective, software should, in many instances, be recognised as an output of research, equivalent to an academic paper. The article discusses the different roles that software may play in research and highlights the relationship between software and research sustainability an…
▽ More
Software now lies at the heart of scholarly research. Here we argue that as well as being important from a methodological perspective, software should, in many instances, be recognised as an output of research, equivalent to an academic paper. The article discusses the different roles that software may play in research and highlights the relationship between software and research sustainability and reproducibility. It describes the challenges associated with the processes of citing and reviewing software, which differ from those used for papers. We conclude that whilst software outputs do not necessarily fit comfortably within the current publication model, there is a great deal of positive work underway that is likely to make an impact in addressing this.
△ Less
Submitted 15 November, 2020;
originally announced November 2020.
-
A Framework for Improving Scholarly Neural Network Diagrams
Authors:
Guy Clarke Marshall,
André Freitas,
Caroline Jay
Abstract:
Neural networks are a prevalent and effective machine learning component, and their application is leading to significant scientific progress in many domains. As the field of neural network systems is fast growing, it is important to understand how advances are communicated. Diagrams are key to this, appearing in almost all papers describing novel systems. This paper reports on a study into the us…
▽ More
Neural networks are a prevalent and effective machine learning component, and their application is leading to significant scientific progress in many domains. As the field of neural network systems is fast growing, it is important to understand how advances are communicated. Diagrams are key to this, appearing in almost all papers describing novel systems. This paper reports on a study into the use of neural network system diagrams, through interviews, card sorting, and qualitative feedback structured around ecologically-derived examples. We find high diversity of usage, perception and preference in both creation and interpretation of diagrams, examining this in the context of existing design, information visualisation, and user experience guidelines.
This interview study is used to derive a framework for improving existing diagrams. This framework is evaluated through a mixed-methods experimental study, and a ``corpus-based'' approach examining properties of published diagrams linking the framework to citations. The studies suggest that the framework captures aspects relating to communicative efficacy of scholarly NN diagrams, and provides simple steps for their implementation.
△ Less
Submitted 21 November, 2022; v1 submitted 28 August, 2020;
originally announced August 2020.
-
Understanding scholarly Natural Language Processing system diagrams through application of the Richards-Engelhardt framework
Authors:
Guy Clarke Marshall,
Caroline Jay,
André Freitas
Abstract:
We utilise Richards-Engelhardt framework as a tool for understanding Natural Language Processing systems diagrams. Through four examples from scholarly proceedings, we find that the application of the framework to this ecological and complex domain is effective for reflecting on these diagrams. We argue for vocabulary to describe multiple-codings, semiotic variability, and inconsistency or misuse…
▽ More
We utilise Richards-Engelhardt framework as a tool for understanding Natural Language Processing systems diagrams. Through four examples from scholarly proceedings, we find that the application of the framework to this ecological and complex domain is effective for reflecting on these diagrams. We argue for vocabulary to describe multiple-codings, semiotic variability, and inconsistency or misuse of visual encoding principles in diagrams. Further, for application to scholarly Natural Language Processing systems, and perhaps systems diagrams more broadly, we propose the addition of "Grou** by Object" as a new visual encoding principle, and "Emphasising" as a new visual encoding type.
△ Less
Submitted 26 August, 2020;
originally announced August 2020.
-
The Four Pillars of Research Software Engineering
Authors:
J. Cohen,
D. S. Katz,
M. Barker,
N. Chue Hong,
R. Haines,
C. Jay
Abstract:
Building software that can support the huge growth in data and computation required by modern research needs individuals with increasingly specialist skill sets that take time to develop and maintain. The Research Software Engineering movement, which started in the UK and has been built up over recent years, aims to recognise and support these individuals. Why does research software matter to prof…
▽ More
Building software that can support the huge growth in data and computation required by modern research needs individuals with increasingly specialist skill sets that take time to develop and maintain. The Research Software Engineering movement, which started in the UK and has been built up over recent years, aims to recognise and support these individuals. Why does research software matter to professional software development practitioners outside the research community? Research software can have great impact on the wider world and recent progress means the area can now be considered as a more realistic option for a professional software development career. In this article we present a structure, along with supporting evidence of real-world activities, that defines four elements that we believe are key to providing comprehensive and sustainable support for Research Software Engineering. We also highlight ways that the wider developer community can learn from, and engage with, these activities.
△ Less
Submitted 25 January, 2023; v1 submitted 3 February, 2020;
originally announced February 2020.
-
Theory-Software Translation: Research Challenges and Future Directions
Authors:
Caroline Jay,
Robert Haines,
Daniel S. Katz,
Jeffrey Carver,
James C. Phillips,
Anshu Dubey,
Sandra Gesing,
Matthew Turk,
Hui Wan,
Hubertus van Dam,
James Howison,
Vitali Morozov,
Steven R. Brandt
Abstract:
The Theory-Software Translation Workshop, held in New Orleans in February 2019, explored in depth the process of both instantiating theory in software - for example, implementing a mathematical model in code as part of a simulation - and using the outputs of software - such as the behavior of a simulation - to advance knowledge. As computation within research is now ubiquitous, the workshop provid…
▽ More
The Theory-Software Translation Workshop, held in New Orleans in February 2019, explored in depth the process of both instantiating theory in software - for example, implementing a mathematical model in code as part of a simulation - and using the outputs of software - such as the behavior of a simulation - to advance knowledge. As computation within research is now ubiquitous, the workshop provided a timely opportunity to reflect on the particular challenges of research software engineering - the process of develo** and maintaining software for scientific discovery. In addition to the general challenges common to all software development projects, research software additionally must represent, manipulate, and provide data for complex theoretical constructs. Ensuring this process is robust is essential to maintaining the integrity of the science resulting from it, and the workshop highlighted a number of areas where the current approach to research software engineering would benefit from an evidence base that could be used to inform best practice.
The workshop brought together expert research software engineers and academics to discuss the challenges of Theory-Software Translation over a two-day period. This report provides an overview of the workshop activities, and a synthesises of the discussion that was recorded. The body of the report presents a thematic analysis of the challenges of Theory-Software Translation as identified by workshop participants, summarises these into a set of research areas, and provides recommendations for the future direction of this work.
△ Less
Submitted 22 October, 2019;
originally announced October 2019.
-
A Methodology for Using GitLab for Software Engineering Learning Analytics
Authors:
Julio César Cortés Ríos,
Kamilla Kopec-Harding,
Sukru Eraslan,
Christopher Page,
Robert Haines,
Caroline Jay,
Suzanne M. Embury
Abstract:
To bridge the digital skills gap, we need to train more people in Software Engineering techniques. This paper reports on a project exploring the way students solve tasks using collaborative development platforms and version control systems, such as GitLab, to find patterns and evaluation metrics that can be used to improve the course content and reflect on the most common issues the students are f…
▽ More
To bridge the digital skills gap, we need to train more people in Software Engineering techniques. This paper reports on a project exploring the way students solve tasks using collaborative development platforms and version control systems, such as GitLab, to find patterns and evaluation metrics that can be used to improve the course content and reflect on the most common issues the students are facing. In this paper, we explore Learning Analytics approaches that can be used with GitLab and similar tools, and discuss the challenges raised when applying those approaches in Software Engineering Education, with the objective of building a pipeline that supports the full Learning Analytics cycle, from data extraction to data analysis. We focus in particular on the data anonymisation step of the proposed pipeline to explore the available alternatives to satisfy the data protection requirements when handling personal information in academic environments for research purposes.
△ Less
Submitted 15 March, 2019;
originally announced March 2019.
-
What Makes Research Software Sustainable? An Interview Study With Research Software Engineers
Authors:
Mario Rosado de Souza,
Robert Haines,
Markel Vigo,
Caroline Jay
Abstract:
Software is now a vital scientific instrument, providing the tools for data collection and analysis across disciplines from bioinformatics and computational physics, to the humanities. The software used in research is often home-grown and bespoke: it is constructed for a particular project, and rarely maintained beyond this, leading to rapid decay, and frequent `reinvention of the wheel'. Understa…
▽ More
Software is now a vital scientific instrument, providing the tools for data collection and analysis across disciplines from bioinformatics and computational physics, to the humanities. The software used in research is often home-grown and bespoke: it is constructed for a particular project, and rarely maintained beyond this, leading to rapid decay, and frequent `reinvention of the wheel'. Understanding how to develop sustainable research software, such that it is suitable for future reuse, is therefore of interest to both researchers and funders, but how to achieve this remains an open question. Here we report the results of an interview study examining how research software engineers -- the people actively develo** software in an academic research environment -- subjectively define software sustainability. Thematic analysis of the data reveals two interacting dimensions: \emph{intrinsic sustainability}, which relates to internal qualities of software, such as modularity, encapsulation and testability, and \emph{extrinsic sustainability}, concerning cultural and organisational factors, including how software is resourced, supported and shared. Research software engineers believe an increased focus on quality and discoverability are key factors in increasing the sustainability of academic research software.
△ Less
Submitted 14 March, 2019;
originally announced March 2019.
-
The State of Sustainable Research Software: Results from the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1)
Authors:
Daniel S. Katz,
Stephan Druskat,
Robert Haines,
Caroline Jay,
Alexander Struck
Abstract:
This article summarizes motivations, organization, and activities of the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1) held in Manchester, UK in September 2017. The WSSSPE series promotes sustainable research software by positively impacting principles and best practices, careers, learning, and credit. This article discusses the Code of Conduct, idea papers, po…
▽ More
This article summarizes motivations, organization, and activities of the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1) held in Manchester, UK in September 2017. The WSSSPE series promotes sustainable research software by positively impacting principles and best practices, careers, learning, and credit. This article discusses the Code of Conduct, idea papers, position papers, experience papers, demos, and lightning talks presented during the workshop. The main part of the article discusses the speed-blogging groups that formed during the meeting, along with the outputs of those sessions.
△ Less
Submitted 19 July, 2018;
originally announced July 2018.