Enabling Generative Design Tools with LLM Agents for Building Novel Devices: A Case Study on Fluidic Computation Interfaces

Specialized design tools play a crucial role in facilitating the prototy** of novel devices (Ion et al., 2017a, b; Lu et al., 2023c; Ou et al., 2016). To streamline the design process, AI has been integrated into such tools, transforming the design landscape through advanced algorithms and computational methods (Gmeiner et al., 2023). This integration is particularly impactful in physical prototy**, where hardware integration is essential. AI algorithms have drastically reduced the costs associated with trial and error for physical objects, increased production efficiency, and broadened the scope of design possibilities. For instance, Zanzibar provides a platform for exploring the design space of physical and digital games (Villar et al., 2018), enabling rapid game development; 3D and 4D printing offer intuitive visualization of precise simulations (An et al., 2018; Peng et al., 2018; Yang et al., 2020; Tao et al., 2023; Sun et al., 2022), even supporting interactive design processes (Wessely et al., 2018). Similar technologies have found wide application in areas such as shape-changing materials (Zhong et al., 2023) and circuit design (Drew et al., 2016; Olberding et al., 2014). However, despite the advancements brought about by ”conventional” AI-integrated tools in improving design efficiency, they still face challenges such as a lack of inspiration and conceptualization of design goals, dependency on users for setting numerous parameters, and a shortage of contextual awareness and proactive suggestions (Gmeiner et al., 2023; Li, 2024).

Meanwhile, LLM has shown promise in addressing these challenges. It has been successfully utilized to enhance creative imaging tools in various domains (Cheng et al., 2023; Duan et al., 2023; Bhavya et al., 2023; Ko et al., 2023; Liu et al., 2023b). By collaborating with LLM, designers can initiate projects with simple requirements and see them through to completion with the aid of LLM. However, despite its potential, in the field of designing interactive hardware systems, which inherently involve input, computation, and output functionalities, and interact contextually with users, LLM has yet to be extensively integrated into specialized design tools. This is potentially due to the following reasons: LLM is trained on large-scale textual data and lacks highly specialized knowledge in newly developed interactive hardware systems to handle these highly specialized tasks. Novel interactive hardware design involves complex combinations of components and functional modules, as well as considerations across multiple aspects. This complexity and diversity are challenging to describe with simple text and require more structured information. Interactive device design requires higher precision, and the results of hardware design need to ensure that the physical implementation works correctly and meets the expected functionality.

Nevertheless, we envision there is a chance to harness LLM’s capabilities to understand and innovate new interactive device technologies. By augmenting conventional specialized design tools with LLM, we can introduce a fresh approach to device prototy**. Leveraging LLM’s strengths in flexibility, adaptability, and creative problem-solving can lower usage barriers and inspire more innovative designs.

Large Language Models, trained on vast dataset to comprehend and manipulate a large number (billions) of parameters (e.g., GPT-4 (Achiam et al., 2023; Andreas, 2022)), has become pivotal in the area as the rise of Transformer (Vaswani et al., 2017) which helps them successfully generate human-like text. Due to their powerful anthropomorphic response capabilities, LLM has been applied to enable a broad range of interactive applications within the Human-Computer Interaction (HCI) community. For instance, some works have focused on enhancing the interaction mechanisms between LLM and users, allowing users to better benefit LLM (Liu et al., 2023a; Dang et al., 2023; Jiang et al., 2023; Zamfirescu-Pereira et al., 2023; Suh et al., 2023; Wang et al., 2023b). Other studies have investigated how LLM can support designers by offering inspiration for CAD projects (Liu et al., 2023b) and enhancing the creative process in animation design (Tseng et al., 2024), thereby directly benefiting users in design and manufacturing fields(Makatura et al., 2023). Additional research has delved into ethical crises and social impacts. For example, ”synthetic lies” discusses the risks of persuasive misinformation generation of LLM (Zhou et al., 2023). Liang et al. have proposed a large-scale empirical analysis on providing review feedback with LLM (Liang et al., 2023). Beyond these areas, LLM has been applied in art creation (Angert et al., 2023), minority groups (Valencia et al., 2023; Huh et al., 2023), public health intervention (Jo et al., 2023), journalism (Petridis et al., 2023), education (Lu et al., 2023a; Peng et al., 2023), and more. However, we still have yet to see the LLM-based generative design tools being fully leveraged to design highly specialized interactive devices.

The agent refers to an entity that is capable of autonomously performing tasks, making decisions, and interacting with other agents or its environment to achieve specific goals (Jennings et al., 1998; Vere and Bickmore, 1990). Within the LLM Agent framework, LLM serves as the agent’s brain with its fundamental abilities such as reasoning, planning and decision-making (Wang et al., 2023a). The agent possesses both long-term memory by leveraging an external knowledge library for fast retrieval and short-term memory through in-context learning in instructions, utilizing tools via external APIs and functions. These abilities endow LLM-agent with powerful and robust capabilities in tackling complex tasks, rendering it highly efficient across diverse applications. The research on LLM-Agent can be generally classified into single-agent and multi-agents. Early work on LLM Agent approaches advocated for the use of a single agent employing tools to tackle complex tasks (bab, 2023; Yao et al., 2022; aut, 2023). However, the intricate logic of complex tasks often led to the single agent experiencing significant hallucinations, making it challenging to resolve complex issues effectively (Chen et al., 2023b). To address this, the multi-agent framework champions role-playing by decomposing complex tasks into multiple, more manageable subtasks and allocating them to different agents, each possessing specialized knowledge in the areas of the assigned subtasks and becomes more focused on resolving these tasks (Chen et al., 2023a; Chan et al., 2023; Park et al., 2023). This approach, which allocates subtasks to collaborative multi-agents, ensures that the generated results are more comprehensive and accurate, behaving with significantly superior capabilities in solving complex tasks far surpassing the abilities of a single agent (Li et al., 2024, 2023). Within the framework of our work, we employed a multi-agent strategy to lay the groundwork for our tool’s design (Fig. 2).

Mechanical computation has garnered increasing attention in recent years, marking its success across various science and engineering fields such as molecular computation (Adleman, 1994), robotics (Rajappan et al., 2021; Wehner et al., 2016), morphological computation (Hauser et al., 2011), and fluidic logic circuits (Mosadegh et al., 2011; Zhang et al., 2017; Rhee and Burns, 2009). Within the HCI device fabrication and design community, there’s a notable trend toward leveraging unconventional computation to enhance computing capabilities. A handful of HCI studies have explored the integration of mechanical computation within interfaces. Ion et al. introduced digital mechanical metamaterials for constructing logic structures (Ion et al., 2017a). Venous Materials (Mor et al., 2020) showcased fluidic mechanisms that react to mechanical inputs from users to produce outputs. Logic Bonbon (Deng et al., 2022) created basic logic gates for crafting desserts with varying flavors based on user inputs. AirLogic (Savage et al., 2022) advanced these fluidic logic concepts by incorporating them into 3D printing processes. The Fluidic Computation Kit (Lu et al., 2023b) delves deeper, offering a systematic exploration of creating advanced mechanical computation devices. It amalgamates diverse force inputs, computation facilitated by various operators, and different output modalities. This breadth makes it particularly interesting, as it encompasses different facets of interactive device construction beyond a singular focus on certain actuation or sensing technology. Given its complexity and innovation, this technology presents unique challenges that may stretch the capabilities of LLM agents. Therefore, we have selected the FCI, drawing from the research of Lu et al., as our entry point to investigate the potential opportunities and challenges of LLM agents in empowering novel device design tools.

The GDT’s architecture, illustrated in Fig. 2, includes elements similar to those found in conventional specialized design tools, such as parametric-driven design, design outcome rendering previews, and simulations. However, unlike those tools that require users to conceive application designs (design goals, DGs) that fit new devices and manually set various parameters or drag-and-drop modules to complete the design, this GDT, augmented with LLM agents, can assist or even take over these tasks. Specifically, the following LLM agents are integrated into this GDT:

- Consultant (Fig. 2.a): This agent is equipped with comprehensive knowledge of FCI’s capabilities and limitations. It can answer questions regarding the FCI, help define design goals, and assist in deciding various details needed during the design phase. It interacts with users via a GUI dialogue box (Fig. 2.i) and provides real-time updates of design details on the GUI for easy access (Fig. 2.h). In addition, the design goal will be displayed at the result window as well (Fig. 2.l)

- Logic Designer, Circuit Engineer, Inspector (Fig. 2.b-d): These three agents are responsible for the design of the computation module, a key component of the FCI tasked with performing logical operations with mechanical circuits. The Logic Designer analyzes the design details provided by the Consultant and outputs computational logic in a prescribed standard format. The Circuit Engineer, familiar with the operators within FCI and capable of ”circuit” design, selects appropriate operators based on the provided logic to complete the FCI circuit design. The Inspector checks and rates the correctness and conciseness of the FCI circuit, deciding whether to send it back to the Circuit Engineer for modifications. Circuit designs approved by the Inspector are used for visualization and simulation on the GUI (Fig. 2.g, k). Users can refer to the visualization results to construct the FCI circuit. Input and output modules are visualized qualitatively.

The computational logic and inspection report will be sent to and displayed at Fig. 2.l.

- I/O Designer (Fig. 2.e): Responsible for designing the I/O module, which includes artifacts like airbags in the FCI for detecting external inputs or providing output feedback. The I/O Designer, also receiving design details from the Consultant, will offer design suggestions for the I/O module based on this information and its knowledge of the FCI. These are primarily qualitative suggestions. But in cases when five predefined shapes are used for the output airbags, quantitative geometry of these shapes (e,g, the diameter of a sphere) will be suggested. Qualitative recommendations are directly reflected in the corresponding section of the GUI (Fig. 2.m). Quantitative suggestions are processed through inverse design algorithms to generate heat-sealing patterns and dimensions for user review via the GUI (Fig. 2.m).

In the next subsection, we will demonstrate the process of completing an FCI design using this FCI-GDT through an example.

In this section, we provide a design walkthrough of using our design tool to create a smart yoga pad for Emily (Fig. 3).

Greet & Introduction (Fig. 3.1). Upon opening the GUI, Emily is warmly greeted by the Consultant, who briefly introduces himself and invites her to inquire about the Fluidic Computation Interface (FCI) and share design ideas. Emily asks about the purpose of FCI, to which the counselor explains it as a non-electronic pneumatics system utilizing fluidic circuits for logic calculations. The FCI comprises Input, Computation, and Output Modules, each serving distinct functions. Potential applications like posture correction and security systems are highlighted, showcasing FCI’s ability to detect force changes and provide various feedback forms. Emily then requests detailed s of FCI applications, leading the counselor to provide instances such as a Posture Correction Chair and Medication Reminder. Each application demonstrates function, detection, and feedback.

Set Design Goal (Fig. 3.2). Following the explanation, the Consultant assists Emily in defining the design goal, assuring her that detailed concept ideas are unnecessary as the bot can help develop them. Emily expresses interest in creating a yoga pad to help practicing yoga and seeks suggestions from the assistant. The Consultant proposes ideas like Posture Detection, Sequence Guide, and Breathing Aid. Inspired by these suggestions, Emily decides to design a yoga pad with posture detection and provide physical assistance for difficult poses. She adds detail: ”If a user cannot reach the hand area of the mat during a pose, the mat will detect this and inflate the hand area to form a supportive structure, similar to a yoga air block, aiding the user in maintaining the correct pose.” When Emily inquired about the specific goal posture, the Consultant also recommended certain applied postures, particularly focusing on common poses that often pose difficulty for beginners and those that can benefit from added support to maintain correct alignment, such as the Triangle Pose. The Consultant confirms the goal and guides Emily to the next step.

Define Input Module (Fig. 3.3). After finalizing the design goal, the Consultant proceeds to help define the Input Module for the yoga pad. The Input Module of the system comprises airbags that detect external forces, causing a change in pressure that alters the input signal from atmospheric (0) to positive (1) pressure. Each input is characterized by three main properties in addition to a comment note:

• •

  Attribute: Defines the state of the signal as Binary (0 or 1), Duration (how long it remains in each state), Frequency (how frequently it transitions), and Edge (transition moment).

• •

  Location: Indicates where the input airbags are placed, such as under the feet, on the sides of the vehicle, or within the seatbelt.

• •

  Manipulation: Describes how the airbags are interacted with, including actions like squeeze, step, press, or twist.

• •

  Note (Optional): Supplements additional information for specific types of input, such as frequency/duration values.

In this walkthrough, the Consultant suggests using two inputs: Input A for detecting hand presence along specific points on the pad corresponding to expected hand placements in posture, and Input B for detecting foot presence to determine if the user is standing on the pad and initiating this posture. These inputs are set to binary attributes, indicating the presence or absence of pressure. Emily asks to add a delay for the input to avoid false activations. The Consultant changes the corresponding attribute of the input and asks to confirm that this meets Emily’s expectations without any additional features.

Define Output Module (Fig. 3.3). Once Emily confirms the former step, the Consultant guides the discussion toward defining the Output Module, presenting various options, including shape-changing, haptic, olfactory and acoustic feedback, which are aligned with the capabilities of the Fluidic Computation Interface (FCI). Given the design goal of providing physical assistance when the user’s hand cannot reach the side edges of the mat, the Consultant suggests one specific output: Output I (Shape-changing Feedback), which involves inflating to form a supportive air block when the hand is not detected but the foot is. The Consultant then asks whether further discussion on this output is needed or if any additional output types should be considered for the yoga mat. Emily is satisfied with the result and confirms it. The Consultant inquires about the shape and size of the output Emily wants and, based on the cues from the yoga bricks, recommends a box measuring 23 * 15 * 7.5 cm.

Define Computation Module (Fig. 3.3). Next, the Consultant outlines the Computation Module, specifying that Output I, the inflation of the supportive air block, is triggered when Input B detects foot pressure but Input A does not detect hand pressure for 30 seconds. This logic ensures that the yoga pad provides assistance when the user is attempting the Triangle Pose and requires support for their hand. Following Emily’s confirmation, the Consultant informs her that the design definition process has concluded. It inquires whether she would prefer to finalize the design in its current state or if there are any adjustments or additions she would like to make.

Preview and interact with the generated device (Fig. 3.4). After reviewing all the design definitions on the UI panels, Emily clicks the ”Result” button to switch to the page displaying the generated FCI model. This interface includes a description, truth table, 3D model of fluid computation kit structures with logic simulation, and suggestions for designing the I/O Module. Emily can click on the highlighted input module model to change its state and observe the corresponding changes of the output module (Fig. 3.4.1). Based on the generated model and description, Emily is ready to connect the entities of the Fluid Computation kits, fabricate inflatable airbags (Fig. 3.4.2), and integrate each module into real space, considering practical application scenarios for their proper arrangement.

We specifically constructed the Consultant agent in GDT to ”include the function, the detection requirements to achieve that function, and the necessary feedback” when proposing design goals. Generally, the GDT consistently provided design goals that included this essential information. The GDT was tasked with proposing one hundred design goals (DGs), delivering ten goals at a time over ten sessions. The complete list of these DGs is available in the Supplementary Material. Firstly, we evaluated the variety of the design goals. The goals were categorized based on application scenarios, as shown in Fig. 4. This categorization spans ten major categories and twenty-four subcategories, demonstrating considerable diversity overall.

Subsequently, we assessed the outcomes, the goals, based on four criteria:

Novelty: This criterion assesses the uniqueness and innovation of the application scenario, focusing on the design’s originality in relation to existing literature and other design goals in the list. For instance, a design such as “…an encrypted door latch that responds to a specific sequence of steps with a particular frequency…(DG2)” lacks novelty, given its similarity to previous works. Conversely, “…to promote an interactive learning environment by detecting edge transitions in pressure on a classroom floor…(DG89)” stands out as novel, distinguished by its uniqueness and absence of similar applications within the experts’ knowledge or the list.

Rationality: Evaluates the logical foundation and appropriateness of the design’s function within its intended scenario. This perspective examines the design goal’s method of providing feedback or interaction, ensuring it is logically coherent and suited for its purpose. For instance, “…ensure driver attentiveness by detecting the frequency of steering wheel manipulation…(DG16)” is deemed irrational, as there is a weak correlation between the mentioned elements. “… to encourage hydration during exercise by detecting the absence of bottle squeezing over time and producing a distinctive olfactory reminder (DG49)” is more rational .

Feasibility: This metric determines the likelihood of the proposed interface being practically realized with the current hardware and technology. It appraises the design’s capability of being developed into a functional, tangible interface. For instance, “…sensing the rising and falling edges of ambient noise pressure on sound-absorbing panels…(DG79)” is considered infeasible, as the technology lacks the capability to discern noise levels.

Complexity: Measures the complexity of the design goals, mainly taking into account the number of inputs, outputs, and computational logic. Experts primarily assess it based on whether a) multiple inputs or outputs need to be employed, and b) the computation potentially requires more than simple one-to-one logic or needs cascading operators. A design is categorized as low complexity if neither condition is met, medium complexity if one condition is met, and high complexity if both are met.

The scoring results for the 100 design goals are shown in Fig. 5. DGs with medium or high levels of novelty accounted for 83%, while the distribution of complexity was relatively even across the three levels. Rationality and Feasibility fundamentally influence whether DGs can be realized, with both criteria showing that approximately 80% of DGs scored medium or higher. For DGs that scored Low in either Rationality or Feasibility, we conducted additional tests. We fed these specific DGs back to the agent, requesting an optimization focused on either Rationality or Feasibility (explaining the meaning of these criteria in the prompt) and then had experts re-evaluate them. As a result, DGs that initially scored Low dropped from 20 to 7 in Rationality and from 23 to 4 in Feasibility.

From the pool of DGs characterized by high novelty and medium or above in Rationality/Feasibility, we selected nine DGs, with three from each complexity level. We tasked the GDT with creating three designs for each DG, resulting in a total of twenty-seven designs. The selected outcomes are illustrated in Fig. 6, 7, 8. Due to space constraints, we detailed nine of these design outcomes. We evaluated the accuracy and highlighted strengths and issues encountered during the design process.

The architecture is illustrated in Fig. 9, leveraging OpenAI APIs (GPT4-preview), Voiceflow, Flask, and Unity in its development. Unity primarily forms the front-end interface, while Voiceflow, Flask, and OpenAI APIs serve as the backbone of the back-end, facilitating the collaborative workflow of LLM agents.

Agents are empowered with Flask and OpenAI APIs. Flask, a lightweight web application framework in Python, facilitates API development, request handling, data post-processing, and serves as an integration layer. The resources—comprising instructions, libraries, and tools—configured for the agents are hosted on Flask and relayed to OpenAI APIs upon agent initialization.

We utilize two distinct OpenAI APIs: GPT and Assistant. Each API demands specific Instruction to define the agent’s core response mechanisms and utilizes user data to prompt responses. The Assistant API, however, distinguishes itself by allowing Tool use, maintaining contextual memory, and the ability to consult a knowledge base, referred to as the Library. This, however, incurs higher token usage. The choice between GPT and Assistant APIs for agent development is dictated by the task’s complexity. For tasks that require basic functionality, such as those handled by the Logic Designer and Inspector, the GPT API is the optimal choice due to its straightforwardness. Conversely, for roles demanding more nuanced understanding and interaction—like the Consultant, I/O Designer, and Circuit Engineer—the Assistant API offers more sophisticated features that are particularly beneficial.

Flask designates a specific port for each agent, acting as the agent’s distinct ”address” within the public network for streamlined access by Voiceflow. Through these dedicated ports, Voiceflow facilitates communication to and from the respective agents, enabling a seamless collaborative workflow among agents. Voiceflow plays a pivotal role in orchestrating the interactions among agents. The front-end interface is developed using Unity (Version 2022.3.15), which monitors the exchanges between Voiceflow and the agents via the Voiceflow API, ensuring that these interactions are effectively presented to the user. In addition to facilitating conversations with users and displaying text-based results, Unity also integrates algorithms for the visualization and simulation of the FCI device.

We implemented the core algorithm for circuit construction using C# in Unity. It is mainly divided into three parts: FCI operator construction, layout wiring, and simulation.

LLM agents bring a vast expanse of knowledge and a unique capacity for generating a diverse array of ideas, complementing human creativity in the design ideation phase. This approach is particularly advantageous when generating innovative solutions across various scenarios, as LLM agents can rapidly access and synthesize information across a wide range of topics, thus expanding perspectives and catalyzing creative thought.

Furthermore, in the DG detailing phase, the synergy between human intuition and LLM’s data-driven suggestions can lead to fascinating outcomes. For example, beyond proposing specific scents for olfactory feedback in FCIs, the LLM agent can delve into its extensive database to suggest essential oil recipes, enriching the design process with detailed and informed choices.

However, this extensive knowledge base and capacity for idea generation can become a drawback if not appropriately managed. It is crucial to clearly communicate the new technology’s design space and capabilities to the LLM agent. Setting constraints is also beneficial; for instance, although FCIs may be compatible with electronic sensors or actuators, the focus is encouraged to be on exploring non-electronic designs. Explicitly informing the agent that FCI is a non-electronic system can greatly reduce the likelihood of it suggesting unsuitable electronic components. This approach ensures that the LLM agent’s contributions are aligned with the project’s goals and adhere to the specified technological framework.

Designing novel devices often involves specialized technical tasks. If corresponding knowledge is well-represented in the LLM’s training data, the agent typically performs competently, as seen in tasks like programming (Hong et al., 2023) and electronic circuit design (Blocklove et al., 2023). If not, the extent to which a LLM agent can handle a specific task would highly depend on how efficiently we can explain the new knowledge to it, and if the agent can leverage its existing ability in the task. For instance, the “circuitry” in FCIs represents a novel approach to mechanical computation, yet it shares analogies with electronic circuits—a domain that has been covered in LLM training. Therefore, after explaining the logic operators used in FCI to the agent, it can combine its extensive knowledge of electronic circuitry to finish designing the computation module of FCIs.

On the other hand, the design intricacies of I/O modules within FCIs illuminate a challenging domain for LLM agents, primarily due to the specialized knowledge required for understanding soft actuators’ complex behaviors and structures could require knowledge about the intricate structures and functionalities of soft actuators, areas with which the LLM can be less acquainted. At present, the tuned agent in our GDT can provide mainly qualitative guidance in these aspects. Although capable of suggesting quantitative dimensions for a few relatively simple shapes that we have explained to it, its ability to tackle tasks that demand a deep understanding of spatial geometry remains constrained. This limitation is true both in introducing new geometric concepts to the agent and in expecting it to design complex geometric shapes or mechanisms.

Efforts to overcome spatial comprehension challenges are crucial, especially since numerous novel device technologies, particularly those involving innovative output actuators, depend heavily on understanding complex spatial and mechanical concepts. To enhance an LLM agent’s capabilities in spatial reasoning and mechanical design, one strategy could involve enriching its training data with s that encompass spatially complex and mechanically detailed scenarios. However, it’s also important to acknowledge that certain domains, such as three-dimensional modeling and design, may inherently lie outside the optimal skill set of LLM due to their primarily text-based knowledge and reasoning capabilities. In such cases, it might be more effective to leverage other types of generative AI that are specifically tailored for 3D conceptualization and design. These specialized AI systems can complement LLM agents by providing expertise in areas where LLM faces limitations, ensuring a more holistic approach to the design and development of novel devices. Recognizing and integrating the strengths of different AI modalities can potentially further enhance the GDT.

Throughout the design task, we noted the agent exhibited great contextual awareness and cognitive capabilities. This keen awareness facilitates a better understanding of the nuances involved in design tasks, enabling the agent to contribute more effectively. We’ve identified three tiers of contextual awareness displayed by the agents within the FCI GDT, each contributing uniquely to the design process:

• •

  Understanding the Design Space: The agent possesses a good awareness of the FIC’s design space if properly tuned. This enables it to not only grasp the broader technological landscape but also to identify and propose relevant and appropriate DGs.

• •

  Reasonably Detailing Post-DG Establishment: After a DG is established, the agent can suggest sensible design details and correct errors based on the context. For instance, it can suggest representative yoga poses and propose corresponding input detection methods; it also recommends feasible parameters like frequency and duration based on the intended detection target or action.

• •

  Strategic Technical Implementation: When determining technical implementation, the agent can suggest practical qualitative and even quantitative solutions based on the DG and design details. s include advising on the shape and processing material requirements for input airbags, or suggesting dimensions for shape-changing output airbags tailored to the usage scenario.

We believe the contextual awareness of LLM agents could positively impact the process of designing novel devices. Our hypothesis suggests that such awareness not only makes the ideation and development phases more efficient but also fosters the creation of solutions that blend innovation with practicality, thereby enhancing the collaborative design performance between AI and human users. This vision emphasizes the necessity for extensive empirical studies in human-AI co-design tasks using our tool to substantiate these potential outcomes. Accordingly, we posit that the agent’s contextual awareness could emerge as a crucial factor in navigating the complexities of novel device design. Should LLM agents demonstrate the capacity to understand, detail, and implement design strategies with precision, they could become invaluable resources in the evolving landscape of novel device creation.

In the implementation of the FCI-GDT, configuring and debugging the agent emerges as a core aspect. Throughout this process, the careful crafting of the Library and clear Instruction is particularly crucial. Theoretically, the configuration process is straightforward, making it relatively simple for those without an extensive programming or development background. Both the library and instructions are mainly written in natural language and can be constructed using our organizational structure as a template (section 6.2.2). However, practical implementation is not without challenges. Creating the library tests developers’ abilities to distill key technological information, while writing instructions examines their skills in conveying information to the agent efficiently and succinctly. Moreover, natural language is not always the most efficient means of conveying a lot of information, differing greatly from the highly abstract, structured, and standardized norms of coding.

Furthermore, the library and instructions guide the agent more as a form of soft constraint, with the agent acting based on its own ”understanding and thinking.” For instance, for the Consultant agent, we defined a dialogue inquiry flow, but it doesn’t necessarily strictly follow this process and can switch freely between different sections. Developers should acknowledge this dynamic, steering away from the inclination to compel the agent into specific actions. Instead, they ought to embrace and leverage the agent’s autonomy, trying to trust the agent’s abilities and granting it a certain freedom in completing tasks. For example, it could be made clear to the agent that it can independently judge the collection of design details and guide users through completing various sections.

The process of debugging agents also greatly differs from software debugging. In the process of debugging an agent, pinpointing problems is often vague, unlike software debugging where specific lines of code with issues is possible to be identified. Developers need to continuously practice to identify the agents’ erroneous behaviors and gradually adjust and improve the library and instructions. For instance, if the agent consistently overlooks certain functions of the device, additional relevant knowledge might need to be supplemented; if an agent’s designs exceed the device’s capabilities, new restrictions could be added; if an agent misuses a certain design space elements due to misunderstanding, it’s necessary to check if the expressions caused confusion, etc.

In summary, the process of configuring and debugging the agent is essentially iterative and optimization-focused, akin to writing and revising a manual. Through precise knowledge libraries and clear operational instructions, as well as meticulous debugging of agent behavior, the performance and applicability of the GDT can be enhanced.

We propose that a similar strategy employed in the development of the FCI-GDT, particularly in terms of task decomposition and agent allocation, could be potentially applied to other research work involving novel devices. To illustrate, we analyze potential approaches for integrating LLM agents into the design tools for several projects as examples.

Digital Mechanical Metamaterials (DMM) (Ion et al., 2017b) are devices created through 3D printing that incorporate mechanical computation. For ideation and detailing phases, a similar Consultant agent could be utilized. However, the design space of this technology has not been systematically summarized in the paper, necessitating its distillation and summarization to clearly explain this technology’s capabilities to the Consultant agent. This includes its fundamental concepts, core framework, input detection and output capabilities, and limitations. For technical implementation, the device could potentially be divided into two main parts: the mechanical circuitry and the metamaterials (Ion et al., 2017a), each assigned to different groups of agents. The design process for mechanical circuitry could leverage the agents’ expertise in electronic circuit design, akin to what is seen in FCI-GDT. However, it’s important to note the key differences between DMM circuitry and that of FCI. DMM circuits operate on non-volatile, binary computing principles, utilizing bistable configurations that do not rely on a constant “power supply”. Instead, they require recharging to recover lost energy after use. These critical differences need to be clearly communicated to the agents. Additionally, it’s crucial to effectively draw analogies between the mechanical operators in DMM systems and the electronic operators in traditional circuits. This approach will facilitate corresponding agents in leveraging their existing circuit design expertise for mechanical circuitry tasks. Synthesizing and visualizing the structure, as well as generating fabricable files for the circuit, could still depend on ”traditional” design tools as outlined in the DMM paper. The metamaterial part could be delegated to another agent or group of agents. Such structures might be unfamiliar to LLM agents, making it unlikely for them to provide precise quantitative suggestions. However, agents could potentially offer reference suggestions, like advising on approximate overall dimensions for the device’s placement or ergonomic shapes and thickness for specific parts, like handles.

LineForm (Nakagaki et al., 2015) represents an actuated curve interface reminiscent of serpentine robotics. Its design space is thoroughly outlined in the associated paper, which could aid a Consultant agent in gras** the underlying technology. The technical implementation primarily emphasizes the programming aspects of the devices, tasks that could potentially be delegated to one group of agents. To effectively program, an agent must first quantitatively describe the target shape which the Consultant agent might have decided to adopt for specific design scenarios. This may involve identifying a set of geometric parameters that can comprehensively define the shapes and explaining these parameters to the agent—along with how altering these parameters influences the shape. The next step could involve assisting another agent in understanding how adjusting each servo’s parameters affects curve control, thereby enabling the agent to program the servos to achieve the desired shape described by the previous agent. If it’s discovered that the final step of achieving and realizing the desired shape can be more efficiently managed using an inverse design algorithm, then a strategic shift could be advantageous. In this case, the agent’s role would mainly be leveraging its contextual awareness and decision-making capabilities to determine the shape parameters based on the design scenario. These parameters would then be inputted into the inverse design algorithm, which is adept at calculating the precise adjustments needed to program the device accordingly.

FoamSense (Nakamaru et al., 2017) introduces a novel soft sensor capable of measuring the deformation states of volumetric soft objects, including compression, bending, twisting, and shearing. The paper meticulously outlines the design process and elements, providing a solid foundation for tuning a Consultant agent to conceptualize and detail the designs. Given the technology’s emphasis on sensor development, two clusters of agents could potentially undertake this task. The first cluster of agents, guided by the Consultant agent’s assembly of requirements—such as the type of manipulation and the specific surface/object integration—would be tasked with estimating the force range, determining the sensor’s optimal shape, selecting suitable materials, and more. The experimental results discussed in the paper could empower the agent to offer quantitative insights regarding these parameters. The second cluster of agents would concentrate on develo** the Arduino code and peripheral support circuits. This area falls within the ”comfort zone” of LLM agents and is theoretically capable of yielding executable outcomes.

Quality control & improvement. While LLMs excel in creative tasks, it’s important to note that creativity often benefits from collaboration between AI and humans. LLMs can generate ideas, suggest possibilities, and provide inspiration, but human judgment, expertise, and subjective taste play crucial roles in refining and realizing creative visions. Besides, LLMs are prone to hallucinations, although we have made efforts to minimize them using multiple agents and confirming with users at each step. Additionally, incorporating feedback from experts and lessons from positive or negative examples in practices could be crucial in refining GPT training. This feedback would be instrumental in enhancing the quality of generated content.

Multimodality. The current version of the tool only supports users in conveying their ideas through textual descriptions, and the generated innovative concept designs are limited to FCI modeling and textual descriptions. It currently lacks support for translating these ideas into sketches, images, models, or videos with real-life scenarios, such as demonstrating the effect of using the Computation Kit on a yoga mat. Future work could explore and integrate advanced technologies and theories, such as leveraging other types of generative AI models. These improvements have the potential to assist users not only in designing and develo** the Fluid Computation device but also in testing, refining, and realizing these concepts in more practical and applicable formats.

User study & Human-AI co-design interaction. Currently, testing of this tool primarily concentrates on objectively evaluating GPT’s performance in designing FCI with minimal human intervention. Future plans include expanding the research to encompass a more diverse user base with varying levels of expertise, to assess how increased human involvement and collaborative design affect performance. This expansion may include role-switching activities between questioning and answering to encourage a more interactive and iterative design methodology, ultimately aiming to understand how these advancements enhance the user experience in design tasks.