Organa: A Robotic Assistant for Automated Chemistry Experimentation and Characterization

Kourosh Darvish

{}^{1,2\dagger*}

, Marta Skreta

{}^{1,2\dagger}

, Yuchi Zhao

{}^{1\dagger}

, Naruki Yoshikawa

{}^{1,2}

,
Sagnik Som

{}^{1}

, Miroslav Bogdanovic

{}^{1}

, Yang Cao

{}^{1}

, Han Hao

{}^{1}

, Hao** Xu

{}^{1,2}

,
Alán Aspuru-Guzik

{}^{1,2\ddagger}

, Animesh Garg

{}^{1,2,3\ddagger}

, Florian Shkurti

{}^{1,2\ddagger}

Abstract

Chemistry experimentation is often resource- and labor-intensive. Despite the many benefits incurred by the integration of advanced and special-purpose lab equipment, many aspects of experimentation are still manually conducted by chemists, for example, polishing an electrode in electrochemistry experiments. Traditional lab automation infrastructure faces challenges when it comes to flexibly adapting to new chemistry experiments. To address this issue, we propose a human-friendly and flexible robotic system, Organa, that automates a diverse set of chemistry experiments. It is capable of interacting with chemists in the lab through natural language, using Large Language Models (LLMs). Organa keeps scientists informed by providing timely reports that incorporate statistical analyses. Additionally, it actively engages with users when necessary for disambiguation or troubleshooting. Organa can reason over user input to derive experiment goals, and plan long sequences of both high-level tasks and low-level robot actions while using feedback from the visual perception of the environment. It also supports scheduling and parallel execution for experiments that require resource allocation and coordination between multiple robots and experiment stations. We show that Organa successfully conducts a diverse set of chemistry experiments, including solubility assessment, pH measurement, recrystallization, and electrochemistry experiments. For the latter, we show that Organa robustly executes a long-horizon plan, comprising 19 steps executed in parallel, to characterize the electrochemical properties of quinone derivatives, a class of molecules used in rechargeable flow batteries. Our user study indicates that Organa significantly improves many aspects of user experience while reducing their physical workload.

Figure 1: Robot setup with Organa’s overall schema. Organa provides a seamless interaction between SDLs and chemists for diverse chemistry experiments. It perceives the objects and the progress of the chemistry task to make an informed decision for the next steps of the chemistry experiment. Informed decisions, guided by user intention and perception, are a key strength. Organa optimizes SDL efficiency through parallel experiment execution, providing timely feedback via reports and analysis, thus kee** users well-informed and involved in high-level decision-making. More details about Organa can be found at https://ac-rad.github.io/organa/.

¹¹footnotetext: University of Toronto, Toronto, ON, Canada²²footnotetext: Vector Institute, Toronto, ON, Canada³³footnotetext: NVIDIA, Santa Clara, CA, USA²²footnotetext: These authors equally contributed to this work.³³footnotetext: These authors equally supervised this work.¹¹footnotetext: Corresponding author; E-mail: [email protected]

Summary

We present Organa: a modular and human-friendly robotic system to enable automation in chemistry labs.

References

[1] M. Christensen, L. P. Yunker, P. Shiri, T. Zepel, P. L. Prieto, S. Grunert, F. Bork, and J. E. Hein, “Automation isn’t automatic,” Chemical Science, vol. 12, no. 47, pp. 15 473–15 490, 2021.
[2] L. M. Roch, F. Häse, C. Kreisbeck, T. Tamayo-Mendoza, L. P. Yunker, J. E. Hein, and A. Aspuru-Guzik, “Chemos: orchestrating autonomous experimentation,” Science Robotics, vol. 3, no. 19, 2018.
[3] S. H. M. Mehr, M. Craven, A. I. Leonov, G. Keenan, and L. Cronin, “A universal system for digitization and automatic execution of the chemical synthesis literature,” Science, vol. 370, no. 6512, pp. 101–108, 2020.
[4] B. Burger, P. M. Maffettone, V. V. Gusev, C. M. Aitchison, Y. Bai, X. Wang, X. Li, B. M. Alston, B. Li, R. Clowes et al., “A mobile robotic chemist,” Nature, vol. 583, no. 7815, pp. 237–241, 2020.
[5] R. Vescovi, T. Ginsburg, K. Hippe, D. Y. Ozgulbas, C. Stone, A. Stroka, R. Butler, B. J. Blaiszik, T. Brettin, K. Chard et al., “Towards a modular architecture for science factories,” Digital Discovery, 2023.
[6] B. P. MacLeod, F. G. Parlane, A. K. Brown, J. E. Hein, and C. P. Berlinguette, “Flexible automation accelerates materials discovery,” Nature Materials, vol. 21, no. 7, pp. 722–726, 2022.
[7] C. Steinruecken, E. Smith, D. Janz, J. Lloyd, and Z. Ghahramani, “The automatic statistician,” Automated machine learning: Methods, systems, challenges, pp. 161–173, 2019.
[8] N. Yoshikawa, M. Skreta, K. Darvish, S. Arellano-Rubach, Z. Ji, L. Bjørn Kristensen, A. Z. Li, Y. Zhao, H. Xu, A. Kuramshin et al., “Large language models for chemistry robotics,” Autonomous Robots, pp. 1–30, 2023.
[9] S. Steiner, J. Wolf, S. Glatzel, A. Andreou, J. M. Granda, G. Keenan, T. Hinkley, G. Aragon-Camarasa, P. J. Kitson, D. Angelone et al., “Organic synthesis in a modular robotic system driven by a chemical programming language,” Science, vol. 363, no. 6423, p. eaav2211, 2019.
[10] I. Oh, M. A. Pence, N. G. Lukhanin, O. Rodríguez, C. M. Schroeder, and J. Rodríguez-López, “The electrolab: An open-source, modular platform for automated characterization of redox-active electrolytes,” Device, vol. 1, no. 5, p. 100103, 2023.
[11] D. Knobbe, H. Zwirnmann, M. Eckhoff, and S. Haddadin, “Core processes in intelligent robotic lab assistants: Flexible liquid handling,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 2335–2342.
[12] H. Fakhruldeen, G. Pizzuto, J. Glowacki, and A. I. Cooper, “Archemist: Autonomous robotic chemistry system architecture,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 6013–6019.
[13] S. Eppel, H. Xu, M. Bismuth, and A. Aspuru-Guzik, “Computer vision for recognition of materials and vessels in chemistry lab settings and the vector-labpics data set,” ACS Central Science, vol. 6, no. 10, pp. 1743–1752, 2020. [Online]. Available: https://doi.org/10.1021/acscentsci.0c00460
[14] R. El-khawaldeh, M. A. Guy, F. Bork, N. Taherimakhsousi, K. N. Jones, J. Hawkins, L. Han, R. P. Pritchard, B. Cole, S. Monfette, and J. E. Hein, “Kee** an “eye” on the experiment: computer vision for real-time monitoring and control,” Chem. Sci., pp. –, 2023. [Online]. Available: http://dx.doi.org/10.1039/D3SC05491H
[15] T. Zepel, V. Lai, L. P. E. Yunker, and J. E. Hein, “Automated liquid-level monitoring and control using computer vision,” ChemRxiv, 2020.
[16] H. Xu, Y. R. Wang, S. Eppel, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Seeing glass: Joint point cloud and depth completion for transparent objects,” arXiv preprint arXiv:2110.00087, 2021.
[17] Y. R. Wang, Y. Zhao, H. Xu, S. Eppel, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Mvtrans: Multi-view perception of transparent objects,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 3771–3778.
[18] N. J. Szymanski, B. Rendy, Y. Fei, R. E. Kumar, T. He, D. Milsted, M. J. McDermott, M. Gallant, E. D. Cubuk, A. Merchant et al., “An autonomous laboratory for the accelerated synthesis of novel materials,” Nature, pp. 1–6, 2023.
[19] R. Duke, S. Mahmoudi, A. P. Kaur, V. Bhat, I. Dingle, N. C. Stumme, S. K. Shaw, D. Eaton, A. Vego, and C. Risko, “Expflow: a graphical user interface for automated reproducible electrochemistry,” Digital Discovery, 2024.
[20] A. Aspuru-Guzik, R. Lindh, and M. Reiher, “The matter simulation (r) evolution,” ACS central science, vol. 4, no. 2, pp. 144–152, 2018.
[21] A. C. Vaucher, P. Schwaller, J. Geluykens, V. H. Nair, A. Iuliano, and T. Laino, “Inferring experimental procedures from text-based representations of chemical reactions,” Nature communications, vol. 12, no. 1, p. 2573, 2021.
[22] Z. Ren, Z. Zhang, Y. Tian, and J. Li, “Crest – copilot for real-world experimental scientist,” ChemRxiv, 2023.
[23] A. M. Bran, S. Cox, A. D. White, and P. Schwaller, “Chemcrow: Augmenting large-language models with chemistry tools,” arXiv preprint arXiv:2304.05376, 2023.
[24] D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes, “Autonomous chemical research with large language models,” Nature, 2023.
[25] C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling, “PDDLStream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning,” in Proceedings of the 30th Int. Conf. on Automated Planning and Scheduling (ICAPS). AAAI Press, 2020, pp. 440–448.
[26] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins, “Pddl-the planning domain definition language,” Technical Report CVC TR98003/DCS TR1165. New Haven, CT: Yale Center for Computational Vision and Control, Tech. Rep. 123, 1998. [Online]. Available: http://www.example.com/advancements_report
[27] M. Toussaint, “Logic-geometric programming: An optimization-based approach to combined task and motion planning.” in IJCAI, 2015, pp. 1930–1936.
[28] M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum, “Differentiable physics and stable modes for tool-use and manipulation planning,” Robotics: Science and Systems Foundation, 2018.
[29] M. Khodeir, B. Agro, and F. Shkurti, “Learning to search in task and motion planning with streams,” IEEE Robotics and Automation Letters, 2023.
[30] B. Kim, L. Shimanuki, L. P. Kaelbling, and T. Lozano-Pérez, “Representation, learning, and planning algorithms for geometric task and motion planning,” The International Journal of Robotics Research, vol. 41, no. 2, pp. 210–231, 2022.
[31] N. Kumar, W. McClinton, R. Chitnis, T. Silver, T. Lozano-Pérez, and L. P. Kaelbling, “Learning efficient abstract planning models that choose what to predict,” in 7th Annual Conference on Robot Learning, 2023.
[32] F. Häse, L. M. Roch, and A. Aspuru-Guzik, “Next-generation experimentation with self-driving laboratories,” Trends in Chemistry, vol. 1, no. 3, pp. 282–291, 2019.
[33] C. D. Hubbs, C. Li, N. V. Sahinidis, I. E. Grossmann, and J. M. Wassick, “A deep reinforcement learning approach for chemical production scheduling,” Computers & Chemical Engineering, vol. 141, p. 106982, 2020.
[34] D. Long, J. Dolejsi, and M. Stolba, “Scheduling problems in pddl,” in Workshop on Knowledge Engineering for Planning and Scheduling, 2023.
[35] M. Fox and D. Long, “Pddl2. 1: An extension to pddl for expressing temporal planning domains,” Journal of artificial intelligence research, vol. 20, pp. 61–124, 2003.
[36] S. Edelkamp, M. Lahijanian, D. Magazzeni, and E. Plaku, “Integrating temporal reasoning and sampling-based motion planning for multigoal problems with dynamics and time windows,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3473–3480, 2018.
[37] J. Chen, B. C. Williams, and C. Fan, “Optimal mixed discrete-continuous planning for linear hybrid systems,” in Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control, ser. HSCC ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3447928.3456654
[38] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=WE_vluYUL-X
[39] M. Skreta, N. Yoshikawa, S. Arellano-Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting,” arXiv preprint arXiv:2303.14100, 2023.
[40] A. Majumdar, F. Xia, B. Ichter, D. Batra, and L. Guibas, “Findthis: Language-driven object disambiguation in indoor environments,” in Conference on Robot Learning. PMLR, 2023, pp. 1335–1347.
[41] M. Helmert, “The fast downward planning system,” Journal of Artificial Intelligence Research, vol. 26, pp. 191–246, 2006.
[42] J. Illingworth and J. Kittler, “A survey of the hough transform,” Computer vision, graphics, and image processing, vol. 44, no. 1, pp. 87–116, 1988.
[43] J. Jiang, G. Cao, J. Deng, T.-T. Do, and S. Luo, “Robotic perception of transparent objects: A review,” IEEE Transactions on Artificial Intelligence, 2023.
[44] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” 2023.
[45] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,” arXiv preprint arXiv:2203.03605, 2022.
[46] Z. Zou, K. Chen, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” Proceedings of the IEEE, vol. 111, no. 3, pp. 257–276, 2023.
[47] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.
[49] Stereolabs, “ZED 2 - AI Stereo Camera,” https://www.stereolabs.com/products/zed-2, accessed: 2023-12-22.
[50] Q.-Y. Zhou, J. Park, and V. Koltun, “Open3D: A modern library for 3D data processing,” arXiv preprint arXiv:1801.09847, 2018.
[51] C. Labrín and F. Urdinez, “Principal component analysis,” in R for Political Data Science. Chapman and Hall/CRC, 2020, pp. 375–393.
[52] P. Beeson and B. Ames, “Trac-ik: An open-source library for improved solving of generic inverse kinematics,” in 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids). IEEE, 2015, pp. 928–935.
[53] Z. Kingston, M. Moll, and L. E. Kavraki, “Exploring implicit spaces for constrained sampling-based planning,” Int. J. Robot. Res., vol. 38, no. 10-11, pp. 1151–1178, 2019.
[54] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning,” Int. J. Robot. Res., vol. 30, no. 7, pp. 846–894, 2011.
[55] “redox potential,” International Union of Pure and Applied Chemistry (IUPAC), 2019. [Online]. Available: https://doi.org/10.1351/goldbook.RT06783
[56] N. A. O. o. J. NAOJ, Handbook of Scientific Tables. WORLD SCIENTIFIC, 2022.
[57] “ph,” International Union of Pure and Applied Chemistry (IUPAC), 2019. [Online]. Available: https://doi.org/10.1351/goldbook.P04524
[58] J. J. Fortman and K. M. Stubbs, “Demonstrations with red cabbage indicator,” Journal of chemical education, vol. 69, no. 1, p. 66, 1992.
[59] B. Huskinson, M. P. Marshak, C. Suh, S. Er, M. R. Gerhardt, C. J. Galvin, X. Chen, A. Aspuru-Guzik, R. G. Gordon, and M. J. Aziz, “A metal-free organic–inorganic aqueous flow battery,” Nature, vol. 505, no. 7482, pp. 195–198, 2014.
[60] A. Khetan, “High-throughput virtual screening of quinones for aqueous redox flow batteries: Status and perspectives,” Batteries, vol. 9, no. 1, p. 24, 2022.
[61] S. Pablo-García et al., 2024, (Manuscript in preparation).
[62] D. M. Heard and A. J. Lennox, “Electrode materials in modern organic electrochemistry,” Angewandte Chemie International Edition, vol. 59, no. 43, pp. 18 866–18 884, 2020.
[63] G. M. Swain, “Solid electrode materials: pretreatment and activation,” in Handbook of electrochemistry. Elsevier, 2007, pp. 111–153.
[64] K. Laws, M. Tze-Kiat Ng, A. Sharma, Y. Jiang, A. J. S. Hammer, and L. Cronin, “An autonomous electrochemical discovery robot that utilises probabilistic algorithms: Probing the redox behaviour of inorganic materials,” ChemElectroChem, p. e202300532, 2023.
[65] N. Yoshikawa et al., 2024, (Manuscript in preparation).
[66] M. Quan, D. Sanchez, M. F. Wasylkiw, and D. K. Smith, “Voltammetry of quinones in unbuffered aqueous solution: reassessing the roles of proton transfer and hydrogen bonding in the aqueous electrochemistry of quinones,” Journal of the American Chemical Society, vol. 129, no. 42, pp. 12 847–12 856, 2007.
[67] K. Darvish, L. Penco, J. Ramos, R. Cisneros, J. Pratt, E. Yoshida, S. Ivaldi, and D. Pucci, “Teleoperation of humanoid robots: A survey,” IEEE Transactions on Robotics, 2023.
[68] S. G. Hart, “Nasa-task load index (nasa-tlx); 20 years later,” in Proceedings of the human factors and ergonomics society annual meeting, vol. 50. Sage publications Sage CA: Los Angeles, CA, 2006, pp. 904–908.
[69] J. Brooke, “Sus: A quick and dirty usability scale,” Usability Eval. Ind., vol. 189, 11 1995.
[70] A. Furnham, “Response bias, social desirability and dissimulation,” Personality and individual differences, vol. 7, no. 3, pp. 385–400, 1986.
[71] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn et al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023.
[72] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone, “Llm+ p: Empowering large language models with optimal planning proficiency,” arXiv preprint arXiv:2304.11477, 2023.
[73] R. El-khawaldeh, M. A. Guy, F. Bork, N. Taherimakhsousi, K. N. Jones, J. Hawkins, L. Han, R. P. Pritchard, B. Cole, S. Monfette et al., “Kee** an “eye” on the experiment: computer vision for real-time monitoring and control,” Chemical Science, 2023.
[74] P. K. Murali, B. Porr, and M. Kaboli, “Touch if it’s transparent! actor: Active tactile-based category-level transparent object reconstruction,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 10 792–10 799.
[75] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks,” in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 8943–8950.
[76] M. Skreta, Z. Zhou, J. L. Yuan, K. Darvish, A. Aspuru-Guzik, and A. Garg, “Replan: Robotic replanning with perception and language models,” 2024.
[77] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Program generation for situated robot task planning using large language models,” Autonomous Robots, pp. 1–14, 2023.
[78] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” arXiv preprint arXiv:2209.07753, 2022.
[79] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023.
[80] M. Khodeir, A. Sonwane, R. Hari, and F. Shkurti, “Policy-guided lazy search with feedback for task and motion planning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 3743–3749.
[81] M. Walker, G. Pizzuto, H. Fakhruldeen, and A. I. Cooper, “Go with the flow: deep learning methods for autonomous viscosity estimations,” Digital Discovery, vol. 2, pp. 1540–1547, 2023. [Online]. Available: http://dx.doi.org/10.1039/D3DD00109A
[82] E. Olson, “Apriltag: A robust and flexible visual fiducial system,” in 2011 IEEE international conference on robotics and automation. IEEE, 2011, pp. 3400–3407.
[83] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
[84] C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y. Xiong, “Tutel: Adaptive mixture-of-experts at scale,” arXiv preprint arXiv:2206.03382, 2022.
[85] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy
[86] Stereolabs, “ZED Mini Camera and SDK Overview,” 2019.
[87] N. Elgrishi, K. J. Rountree, B. D. McCarthy, E. S. Rountree, T. T. Eisenhart, and J. L. Dempsey, “A practical beginner’s guide to cyclic voltammetry,” Journal of chemical education, vol. 95, no. 2, pp. 197–206, 2018.
[88] H. E. Grecco, M. C. Dartiailh, G. Thalhammer-Thurner, T. Bronger, and F. Bauer, “Pyvisa: the python instrumentation package,” Journal of Open Source Software, vol. 8, no. 84, p. 5304, 2023.