-
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
Authors:
Jason Wu,
Eldon Schoop,
Alan Leung,
Titus Barik,
Jeffrey P. Bigham,
Jeffrey Nichols
Abstract:
Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expensive human feedback or distilling a proprietary model. In this paper, we explore the use of automated feedback (compilers and multi-modal models) to guide LLMs to generate high-quality UI code. Our method starts with an…
▽ More
Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expensive human feedback or distilling a proprietary model. In this paper, we explore the use of automated feedback (compilers and multi-modal models) to guide LLMs to generate high-quality UI code. Our method starts with an existing LLM and iteratively produces improved models by self-generating a large synthetic dataset using an original model, applying automated tools to aggressively filter, score, and de-duplicate the data into a refined higher quality dataset. The original LLM is improved by finetuning on this refined dataset. We applied our approach to several open-source LLMs and compared the resulting performance to baseline models with both automated metrics and human preferences. Our evaluation shows the resulting models outperform all other downloadable baselines and approach the performance of larger proprietary models.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks
Authors:
Ruijia Cheng,
Titus Barik,
Alan Leung,
Fred Hohman,
Jeffrey Nichols
Abstract:
Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into computational notebooks that augments LLM-based code generati…
▽ More
Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into computational notebooks that augments LLM-based code generation with an additional ephemeral UI step, offering users UI scaffolds as an intermediate stage between user prompts and code generation. We present this workflow in BISCUIT, an extension for JupyterLab that provides users with ephemeral UIs generated by LLMs based on the context of their code and intentions, scaffolding users to understand, guide, and explore with LLM-generated code. Through a user study where 10 novices used BISCUIT for machine learning tutorials, we found that BISCUIT offers users representations of code to aid their understanding, reduces the complexity of prompt engineering, and creates a playground for users to explore different variables and iterate on their ideas.
△ Less
Submitted 11 July, 2024; v1 submitted 10 April, 2024;
originally announced April 2024.
-
Make It Make Sense! Understanding and Facilitating Sensemaking in Computational Notebooks
Authors:
Souti Chattopadhyay,
Zixuan Feng,
Emily Arteaga,
Audrey Au,
Gonzalo Ramos,
Titus Barik,
Anita Sarma
Abstract:
Reusing and making sense of other scientists' computational notebooks. However, making sense of existing notebooks is a struggle, as these reference notebooks are often exploratory, have messy structures, include multiple alternatives, and have little explanation. To help mitigate these issues, we developed a catalog of cognitive tasks associated with the sensemaking process. Utilizing this catalo…
▽ More
Reusing and making sense of other scientists' computational notebooks. However, making sense of existing notebooks is a struggle, as these reference notebooks are often exploratory, have messy structures, include multiple alternatives, and have little explanation. To help mitigate these issues, we developed a catalog of cognitive tasks associated with the sensemaking process. Utilizing this catalog, we introduce Porpoise: an interactive overlay on computational notebooks. Porpoise integrates computational notebook features with digital design, grou** cells into labeled sections that can be expanded, collapsed, or annotated for improved sensemaking.
We investigated data scientists' needs with unfamiliar computational notebooks and investigated the impact of Porpoise adaptations on their comprehension process. Our counterbalanced study with 24 data scientists found Porpoise enhanced code comprehension, making the experience more akin to reading a book, with one participant describing it as It's really like reading a book.
△ Less
Submitted 18 December, 2023;
originally announced December 2023.
-
Half unit cell shift defect induced helical states in Fe-based chalcogenide superconductors
Authors:
Tamoghna Barik,
Jay D. Sau
Abstract:
Recent scanning tunneling spectroscopy along crystalline domain-walls associated with a half unit cell shift have revealed sub-gap density of states that are expected to arise from helical Majorana modes. Such propagating Majorana modes have been proposed to exist on the surface state of topological materials similar to FeTe$_{\text{1-x}}$Se$_\text{x}$ (FTS) along line defects where the supercondu…
▽ More
Recent scanning tunneling spectroscopy along crystalline domain-walls associated with a half unit cell shift have revealed sub-gap density of states that are expected to arise from helical Majorana modes. Such propagating Majorana modes have been proposed to exist on the surface state of topological materials similar to FeTe$_{\text{1-x}}$Se$_\text{x}$ (FTS) along line defects where the superconducting order parameter (OP) is phase shifted by $π$. Here we show that such a $π$ shift in theOP across the half unit-cell shift domain-wall can occur in quite conventional tight-binding models of superconducting FTS as a result of the $s_{\pm}$ pairing symmetry across $Γ$ and M pockets of FTS. The resultant inter-pocket transmission between $Γ$ and M pockets is found to be typically larger than the intra-pocket transmissions. We confirm these conclusions with a calculation based on the Bogoliubov-de-Gennes (BdG) formalism which shows that a $π$-shift across the domain-wall is favored for a large range of model parameters for FTS. We discuss parameter regimes where this mechanism might explain the STS experiments as well as propose to test this explanation by searching for evidence of large inter-pocket scattering.
△ Less
Submitted 24 May, 2023;
originally announced May 2023.
-
Overwatch: Learning Patterns in Code Edit Sequences
Authors:
Yuhao Zhang,
Yasharth Bajpai,
Priyanshu Gupta,
Ameya Ketkar,
Miltiadis Allamanis,
Titus Barik,
Sumit Gulwani,
Arjun Radhakrishna,
Mohammad Raza,
Gustavo Soares,
Ashish Tiwari
Abstract:
Integrated Development Environments (IDEs) provide tool support to automate many source code editing tasks. Traditionally, IDEs use only the spatial context, i.e., the location where the developer is editing, to generate candidate edit recommendations. However, spatial context alone is often not sufficient to confidently predict the developer's next edit, and thus IDEs generate many suggestions at…
▽ More
Integrated Development Environments (IDEs) provide tool support to automate many source code editing tasks. Traditionally, IDEs use only the spatial context, i.e., the location where the developer is editing, to generate candidate edit recommendations. However, spatial context alone is often not sufficient to confidently predict the developer's next edit, and thus IDEs generate many suggestions at a location. Therefore, IDEs generally do not actively offer suggestions and instead, the developer is usually required to click on a specific icon or menu and then select from a large list of potential suggestions. As a consequence, developers often miss the opportunity to use the tool support because they are not aware it exists or forget to use it.
To better understand common patterns in developer behavior and produce better edit recommendations, we can additionally use the temporal context, i.e., the edits that a developer was recently performing. To enable edit recommendations based on temporal context, we present Overwatch, a novel technique for learning edit sequence patterns from traces of developers' edits performed in an IDE. Our experiments show that Overwatch has 78% precision and that Overwatch not only completed edits when developers missed the opportunity to use the IDE tool support but also predicted new edits that have no tool support in the IDE.
△ Less
Submitted 25 July, 2022;
originally announced July 2022.
-
Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements
Authors:
Jason Wu,
Titus Barik,
Xiaoyi Zhang,
Colin Lea,
Jeffrey Nichols,
Jeffrey P. Bigham
Abstract:
Touch is the primary way that users interact with smartphones. However, building mobile user interfaces where touch interactions work well for all users is a difficult problem, because users have different abilities and preferences. We propose a system, Reflow, which automatically applies small, personalized UI adaptations, called refinements -- to mobile app screens to improve touch efficiency. R…
▽ More
Touch is the primary way that users interact with smartphones. However, building mobile user interfaces where touch interactions work well for all users is a difficult problem, because users have different abilities and preferences. We propose a system, Reflow, which automatically applies small, personalized UI adaptations, called refinements -- to mobile app screens to improve touch efficiency. Reflow uses a pixel-based strategy to work with existing applications, and improves touch efficiency while minimally disrupting the design intent of the original application. Our system optimizes a UI by (i) extracting its layout from its screenshot, (ii) refining its layout, and (iii) re-rendering the UI to reflect these modifications. We conducted a user study with 10 participants and a heuristic evaluation with 6 experts and found that applications optimized by Reflow led to, on average, 9% faster selection time with minimal layout disruption. The results demonstrate that Reflow's refinements useful UI adaptations to improve touch interactions.
△ Less
Submitted 15 July, 2022;
originally announced July 2022.
-
Extracting Replayable Interactions from Videos of Mobile App Usage
Authors:
Jieshan Chen,
Amanda Swearngin,
Jason Wu,
Titus Barik,
Jeffrey Nichols,
Xiaoyi Zhang
Abstract:
Screen recordings of mobile apps are a popular and readily available way for users to share how they interact with apps, such as in online tutorial videos, user reviews, or as attachments in bug reports. Unfortunately, both people and systems can find it difficult to reproduce touch-driven interactions from video pixel data alone. In this paper, we introduce an approach to extract and replay user…
▽ More
Screen recordings of mobile apps are a popular and readily available way for users to share how they interact with apps, such as in online tutorial videos, user reviews, or as attachments in bug reports. Unfortunately, both people and systems can find it difficult to reproduce touch-driven interactions from video pixel data alone. In this paper, we introduce an approach to extract and replay user interactions in videos of mobile apps, using only pixel information in video frames. To identify interactions, we apply heuristic-based image processing and convolutional deep learning to segment screen recordings, classify the interaction in each segment, and locate the interaction point. To replay interactions on another device, we match elements on app screens using UI element detection. We evaluate the feasibility of our pixel-based approach using two datasets: the Rico mobile app dataset and a new dataset of 64 apps with both iOS and Android versions. We find that our end-to-end approach can successfully replay a majority of interactions (iOS--84.1%, Android--78.4%) on different devices, which is a step towards supporting a variety of scenarios, including automatically annotating interactions in existing videos, automated UI testing, and creating interactive app tutorials.
△ Less
Submitted 8 July, 2022;
originally announced July 2022.
-
Symphony: Composing Interactive Interfaces for Machine Learning
Authors:
Alex Bäuerle,
Ángel Alexander Cabrera,
Fred Hohman,
Megan Maher,
David Koski,
Xavier Suau,
Titus Barik,
Dominik Moritz
Abstract:
Interfaces for machine learning (ML), information and visualizations about models or data, can help practitioners build robust and responsible ML systems. Despite their benefits, recent studies of ML teams and our interviews with practitioners (n=9) showed that ML interfaces have limited adoption in practice. While existing ML interfaces are effective for specific tasks, they are not designed to b…
▽ More
Interfaces for machine learning (ML), information and visualizations about models or data, can help practitioners build robust and responsible ML systems. Despite their benefits, recent studies of ML teams and our interviews with practitioners (n=9) showed that ML interfaces have limited adoption in practice. While existing ML interfaces are effective for specific tasks, they are not designed to be reused, explored, and shared by multiple stakeholders in cross-functional teams. To enable analysis and communication between different ML practitioners, we designed and implemented Symphony, a framework for composing interactive ML interfaces with task-specific, data-driven components that can be used across platforms such as computational notebooks and web dashboards. We developed Symphony through participatory design sessions with 10 teams (n=31), and discuss our findings from deploying Symphony to 3 production ML projects at Apple. Symphony helped ML practitioners discover previously unknown issues like data duplicates and blind spots in models while enabling them to share insights with other stakeholders.
△ Less
Submitted 17 February, 2022;
originally announced February 2022.
-
On the signatures of non-topological patches on the surface of topological insulators
Authors:
Tamoghna Barik,
Jay D. Sau
Abstract:
The non-trivial topology in the layered $\text{FeTe}_{0.55}\text{Se}_{0.45}$ (FTS) superconductor has been suggested by both theory and experiment to be strongly dependent on the Te concentration. Motivated by this together with the Te fluctuations expected from alloy disorder, we develop a simple layered model for a strong topological insulator that allows us to describe a scenario where topologi…
▽ More
The non-trivial topology in the layered $\text{FeTe}_{0.55}\text{Se}_{0.45}$ (FTS) superconductor has been suggested by both theory and experiment to be strongly dependent on the Te concentration. Motivated by this together with the Te fluctuations expected from alloy disorder, we develop a simple layered model for a strong topological insulator that allows us to describe a scenario where topologically trivial domains permeate the sample. We refer to such a phase as topological domain disordered and study the local density (LDOS) of the topological surface states that can be measured using scanning tunneling spectroscopy (STS) in this phase. We find that topologically trivial domains on the surface, where one would expect the topological surface state to be absent, appear as regions of suppressed LDOS surrounded by domain walls with enhanced LDOS. Furthermore, we show that studying the energy dependence of the STS should allow us to distinguish the topologically trivial parts of the surface from other forms of disorder. Finally, we discuss implications of such local disappearance of the topological surface states for the observation of Majorana modes in vortices.
△ Less
Submitted 30 August, 2021;
originally announced August 2021.
-
Non-equilibrium nature of non-linear optical response: Application to the bulk photo voltaic effect
Authors:
Tamoghna Barik,
Jay D. Sau
Abstract:
The bulk photovoltaic effect is an example of a non-linear optical response that leads to a DC current that is relevant for photo-voltaic applications. In this work, we theoretically study this effect in the presence of electron-phonon interactions. Using the response function formalism we find that the non-linear optical response, in general, contains three operator correlation functions, one of…
▽ More
The bulk photovoltaic effect is an example of a non-linear optical response that leads to a DC current that is relevant for photo-voltaic applications. In this work, we theoretically study this effect in the presence of electron-phonon interactions. Using the response function formalism we find that the non-linear optical response, in general, contains three operator correlation functions, one of which is not ordered in time. This latter correlator cannot be computed from equilibrium field theory. Using a semiclassical approach instead, we show that the bulk photovoltaic effect can be attributed to the dipole moment of the generated excitons. We then confirm the validity of the semiclassical result (which agrees with the non-interacting result) for non-linear DC response from a quantum master equation approach. From this formalism we find that, in contrast to usual linear response, the scattering rate has a strong implicit effect on the non-linear DC response. Most interestingly, the semiclassical treatment shows that the non-linear DC response for spatially inhomogeneous excitation profiles is strongly non-local and must involve the aforementioned out-of-time-ordered correlators that cannot be computed by equilibrium field theory.
△ Less
Submitted 11 September, 2019; v1 submitted 5 August, 2019;
originally announced August 2019.
-
It's Like Python But: Towards Supporting Transfer of Programming Language Knowledge
Authors:
Nischal Shrestha,
Titus Barik,
Chris Parnin
Abstract:
Expertise in programming traditionally assumes a binary novice-expert divide. Learning resources typically target programmers who are learning programming for the first time, or expert programmers for that language. An underrepresented, yet important group of programmers are those that are experienced in one programming language, but desire to author code in a different language. For this scenario…
▽ More
Expertise in programming traditionally assumes a binary novice-expert divide. Learning resources typically target programmers who are learning programming for the first time, or expert programmers for that language. An underrepresented, yet important group of programmers are those that are experienced in one programming language, but desire to author code in a different language. For this scenario, we postulate that an effective form of feedback is presented as a transfer from concepts in the first language to the second. Current programming environments do not support this form of feedback.
In this study, we apply the theory of learning transfer to teach a language that programmers are less familiar with--such as R--in terms of a programming language they already know--such as Python. We investigate learning transfer using a new tool called Transfer Tutor that presents explanations for R code in terms of the equivalent Python code. Our study found that participants leveraged learning transfer as a cognitive strategy, even when unprompted. Participants found Transfer Tutor to be useful across a number of affordances like step** through and highlighting facts that may have been missed or misunderstood. However, participants were reluctant to accept facts without code execution or sometimes had difficulty reading explanations that are verbose or complex. These results provide guidance for future designs and research directions that can support learning transfer when learning new programming languages.
△ Less
Submitted 27 August, 2018;
originally announced August 2018.
-
Learning Quick Fixes from Code Repositories
Authors:
Reudismam Rolim,
Gustavo Soares,
Rohit Gheyi,
Titus Barik,
Loris D'Antoni
Abstract:
Code analyzers such as Error Prone and FindBugs detect code patterns symptomatic of bugs, performance issues, or bad style. These tools express patterns as quick fixes that detect and rewrite unwanted code. However, it is difficult to come up with new quick fixes and decide which ones are useful and frequently appear in real code. We propose to rely on the collective wisdom of programmers and lear…
▽ More
Code analyzers such as Error Prone and FindBugs detect code patterns symptomatic of bugs, performance issues, or bad style. These tools express patterns as quick fixes that detect and rewrite unwanted code. However, it is difficult to come up with new quick fixes and decide which ones are useful and frequently appear in real code. We propose to rely on the collective wisdom of programmers and learn quick fixes from revision histories in software repositories. We present REVISAR, a tool for discovering common Java edit patterns in code repositories. Given code repositories and their revision histories, REVISAR (i) identifies code edits from revisions and (ii) clusters edits into sets that can be described using an edit pattern. The designers of code analyzers can then inspect the patterns and add the corresponding quick fixes to their tools. We ran REVISAR on nine popular GitHub projects, and it discovered 89 useful edit patterns that appeared in 3 or more projects. Moreover, 64% of the discovered patterns did not appear in existing tools. We then conducted a survey with 164 programmers from 124 projects and found that programmers significantly preferred eight out of the nine of the discovered patterns. Finally, we submitted 16 pull requests applying our patterns to 9 projects and, at the time of the writing, programmers accepted 6 (60%) of them. The results of this work aid toolsmiths in discovering quick fixes and making informed decisions about which quick fixes to prioritize based on patterns programmers actually apply in practice.
△ Less
Submitted 7 September, 2018; v1 submitted 10 March, 2018;
originally announced March 2018.
-
Probing Internal Stress and Crystallinity in Wet Foam via Raman Spectroscopy
Authors:
T. K. Barik,
P. Bandyopadhyay,
A. Roy
Abstract:
In this article, we correlate the internal stress and the characteristics of a vibrational mode in wet foam. Using microscope images, we estimate the average size of the bubbles in wet foam, at specific time intervals, over a duration of twenty four hours. Raman spectra are also recorded at the same time intervals, over the same time frame. We show that the internal stress, originated from the m…
▽ More
In this article, we correlate the internal stress and the characteristics of a vibrational mode in wet foam. Using microscope images, we estimate the average size of the bubbles in wet foam, at specific time intervals, over a duration of twenty four hours. Raman spectra are also recorded at the same time intervals, over the same time frame. We show that the internal stress, originated from the macroscopic structural change of foam with ageing, can be related to the observed Raman shift of the low frequency methylene rocking mode of the constituent surfactant molecules in foam. In this report we also show the capability of the Raman spectroscopy to reveal the crystallinity in foamy materials, when studied for a longer period of time.
△ Less
Submitted 4 June, 2009;
originally announced June 2009.
-
Probing liquid surface waves, liquid properties and liquid films with light diffraction
Authors:
Tarun Kr. Barik,
Partha Roy Chaudhuri,
Anushree Roy,
Sayan Kar
Abstract:
Surface waves on liquids act as a dynamical phase grating for incident light. In this article, we revisit the classical method of probing such waves (wavelengths of the order of mm) as well as inherent properties of liquids and liquid films on liquids, using optical diffraction. A combination of simulation and experiment is proposed to trace out the surface wave profiles in various situations (\…
▽ More
Surface waves on liquids act as a dynamical phase grating for incident light. In this article, we revisit the classical method of probing such waves (wavelengths of the order of mm) as well as inherent properties of liquids and liquid films on liquids, using optical diffraction. A combination of simulation and experiment is proposed to trace out the surface wave profiles in various situations (\emph{eg.} for one or more vertical, slightly immersed, electrically driven exciters). Subsequently, the surface tension and the spatial dam** coefficient (related to viscosity) of a variety of liquids are measured carefully in order to gauge the efficiency of measuring liquid properties using this optical probe. The final set of results deal with liquid films where dispersion relations, surface and interface modes, interfacial tension and related issues are investigated in some detail, both theoretically and experimentally. On the whole, our observations and analyses seem to support the claim that this simple, low--cost apparatus is capable of providing a wealth of information on liquids and liquid surface waves in a non--destructive way.
△ Less
Submitted 4 April, 2006; v1 submitted 20 October, 2005;
originally announced October 2005.
-
Diffraction of light by interfering liquid surface waves
Authors:
Tarun Kr. Barik,
Anushree Roy,
Sayan Kar
Abstract:
Interfering liquid surface waves are generated by electrically driven vertical oscillations of two or more equispaced pins immersed in a liquid (water). The corresponding intensity distribution, resulting from diffraction of monochromatic light by the reflection phase grating formed on the liquid surface, is calculated theoretically and found to tally with experiments. The curious features of th…
▽ More
Interfering liquid surface waves are generated by electrically driven vertical oscillations of two or more equispaced pins immersed in a liquid (water). The corresponding intensity distribution, resulting from diffraction of monochromatic light by the reflection phase grating formed on the liquid surface, is calculated theoretically and found to tally with experiments. The curious features of the diffraction pattern and its relation to the interference of waves on the liquid surface are used to measure the amplitude and wavelength of the resultant surface wave along the line joining the two sources of oscillation. Finally, a sample diffraction pattern obtained by optically probing surface regions where interference produces a lattice--like structure is demonstrated and qualitatively explained.
△ Less
Submitted 25 May, 2004;
originally announced May 2004.
