-
How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds
Authors:
Prithviraj Ammanabrolu,
Ethan Tien,
Matthew Hausknecht,
Mark O. Riedl
Abstract:
Text-based games are long puzzles or quests, characterized by a sequence of sparse and potentially deceptive rewards. They provide an ideal platform to develop agents that perceive and act upon the world using a combinatorially sized natural language state-action space. Standard Reinforcement Learning agents are poorly equipped to effectively explore such spaces and often struggle to overcome bott…
▽ More
Text-based games are long puzzles or quests, characterized by a sequence of sparse and potentially deceptive rewards. They provide an ideal platform to develop agents that perceive and act upon the world using a combinatorially sized natural language state-action space. Standard Reinforcement Learning agents are poorly equipped to effectively explore such spaces and often struggle to overcome bottlenecks---states that agents are unable to pass through simply because they do not see the right action sequence enough times to be sufficiently reinforced. We introduce Q*BERT, an agent that learns to build a knowledge graph of the world by answering questions, which leads to greater sample efficiency. To overcome bottlenecks, we further introduce MC!Q*BERT an agent that uses an knowledge-graph-based intrinsic motivation to detect bottlenecks and a novel exploration strategy to efficiently learn a chain of policy modules to overcome them. We present an ablation study and results demonstrating how our method outperforms the current state-of-the-art on nine text games, including the popular game, Zork, where, for the first time, a learning agent gets past the bottleneck where the player is eaten by a Grue.
△ Less
Submitted 12 June, 2020;
originally announced June 2020.
-
How To Avoid Being Eaten By a Grue: Exploration Strategies for Text-Adventure Agents
Authors:
Prithviraj Ammanabrolu,
Ethan Tien,
Zhaochen Luo,
Mark O. Riedl
Abstract:
Text-based games -- in which an agent interacts with the world through textual natural language -- present us with the problem of combinatorially-sized action-spaces. Most current reinforcement learning algorithms are not capable of effectively handling such a large number of possible actions per turn. Poor sample efficiency, consequently, results in agents that are unable to pass bottleneck state…
▽ More
Text-based games -- in which an agent interacts with the world through textual natural language -- present us with the problem of combinatorially-sized action-spaces. Most current reinforcement learning algorithms are not capable of effectively handling such a large number of possible actions per turn. Poor sample efficiency, consequently, results in agents that are unable to pass bottleneck states, where they are unable to proceed because they do not see the right action sequence to pass the bottleneck enough times to be sufficiently reinforced. Building on prior work using knowledge graphs in reinforcement learning, we introduce two new game state exploration strategies. We compare our exploration strategies against strong baselines on the classic text-adventure game, Zork1, where prior agent have been unable to get past a bottleneck where the agent is eaten by a Grue.
△ Less
Submitted 19 February, 2020;
originally announced February 2020.
-
Story Realization: Expanding Plot Events into Sentences
Authors:
Prithviraj Ammanabrolu,
Ethan Tien,
Wesley Cheung,
Zhaochen Luo,
William Ma,
Lara J. Martin,
Mark O. Riedl
Abstract:
Neural network based approaches to automated story plot generation attempt to learn how to generate novel plots from a corpus of natural language plot summaries. Prior work has shown that a semantic abstraction of sentences called events improves neural plot generation and and allows one to decompose the problem into: (1) the generation of a sequence of events (event-to-event) and (2) the transfor…
▽ More
Neural network based approaches to automated story plot generation attempt to learn how to generate novel plots from a corpus of natural language plot summaries. Prior work has shown that a semantic abstraction of sentences called events improves neural plot generation and and allows one to decompose the problem into: (1) the generation of a sequence of events (event-to-event) and (2) the transformation of these events into natural language sentences (event-to-sentence). However, typical neural language generation approaches to event-to-sentence can ignore the event details and produce grammatically-correct but semantically-unrelated sentences. We present an ensemble-based model that generates natural language guided by events.We provide results---including a human subjects study---for a full end-to-end automated story generation system showing that our method generates more coherent and plausible stories than baseline approaches.
△ Less
Submitted 21 November, 2019; v1 submitted 8 September, 2019;
originally announced September 2019.
-
Effects of Ox-LDL on Macrophages NAD(P)H Autofluorescence Changes by Two-photon Microscopy
Authors:
Ching-Ting Lin,
En-Kuang Tien,
Szu-Yuan Lee,
Long-Sheng Lu,
Chau-Chung Wu,
Chen-Yuan Dong,
Chii-Wann Lin
Abstract:
Ox-LDL uptakes by macrophage play a critical role in the happening of atherosclerosis. Because of its low damage on observed cells and better signal-to- background ratio, two-photon excitation fluorescence microscopy is used to observe NAD(P)H autofluorescence of macrophage under difference cultured conditions- bare cover glass, coated with fibronectin or poly-D-lysine. The results show that the…
▽ More
Ox-LDL uptakes by macrophage play a critical role in the happening of atherosclerosis. Because of its low damage on observed cells and better signal-to- background ratio, two-photon excitation fluorescence microscopy is used to observe NAD(P)H autofluorescence of macrophage under difference cultured conditions- bare cover glass, coated with fibronectin or poly-D-lysine. The results show that the optimal condition is fibronectin coated surface, on which, macrophages profile can be clearly identified on NAD(P)H autofluorescence images collected by two-photon microscopy. Moreover, different morphology and intensities of autofluorescence under different conditions were observed as well. In the future, effects of ox-LDL on macrophages will be investigated by purposed system to research etiology of atherosclerosis.
△ Less
Submitted 14 August, 2007;
originally announced August 2007.
