-
Potential-Based Reward Sha** For Intrinsic Motivation
Authors:
Grant C. Forbes,
Nitish Gupta,
Leonardo Villalobos-Arias,
Colin M. Potts,
Arnav Jhala,
David L. Roberts
Abstract:
Recently there has been a proliferation of intrinsic motivation (IM) reward-sha** methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward sha**, particularly through potential-based reward sha** (PBRS), has not been ap…
▽ More
Recently there has been a proliferation of intrinsic motivation (IM) reward-sha** methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward sha**, particularly through potential-based reward sha** (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we prove preserves the set of optimal policies under a more general set of functions than has been previously proven. We also present {\em Potential-Based Intrinsic Motivation} (PBIM), a method for converting IM rewards into a potential-based form that is useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey and Cliff Walking environments, we demonstrate that PBIM successfully prevents the agent from converging to a suboptimal policy and can speed up training.
△ Less
Submitted 12 February, 2024;
originally announced February 2024.
-
A Customizable Generator for Comic-Style Visual Narrative
Authors:
Yi-Chun Chen,
Arnav Jhala
Abstract:
We present a theory-inspired visual narrative generator that incorporates comic-authoring idioms, which transfers the conceptual principles of comics into system layers that integrate the theories to create comic content. The generator creates comics through sequential decision-making across layers from panel composition, object positions, panel transitions, and narrative elements. Each layer's de…
▽ More
We present a theory-inspired visual narrative generator that incorporates comic-authoring idioms, which transfers the conceptual principles of comics into system layers that integrate the theories to create comic content. The generator creates comics through sequential decision-making across layers from panel composition, object positions, panel transitions, and narrative elements. Each layer's decisions are based on narrative goals and follow the respective layer idioms of the medium. Cohn's narrative grammar provides the overall story arc. Photographic compositions inspired by the rule of thirds is used to provide panel compositions. McCloud's proposed panel transitions based on focus shifts between scene, character, and temporal changes are encoded in the transition layer. Finally, common overlay symbols (such as the exclamation) are added based on analyzing action verbs using an action-verb ontology. We demonstrate the variety of generated comics through various settings with example outputs. The generator and associated modules could be a useful system for visual narrative authoring and for further research into computational models of visual narrative understanding.
△ Less
Submitted 13 December, 2023;
originally announced January 2024.
-
Panel Transitions for Genre Analysis in Visual Narratives
Authors:
Yi-Chun Chen,
Arnav Jhala
Abstract:
Understanding how humans communicate and perceive narratives is important for media technology research and development. This is particularly important in current times when there are tools and algorithms that are easily available for amateur users to create high-quality content. Narrative media develops over time a set of recognizable patterns of features across similar artifacts. Genre is one su…
▽ More
Understanding how humans communicate and perceive narratives is important for media technology research and development. This is particularly important in current times when there are tools and algorithms that are easily available for amateur users to create high-quality content. Narrative media develops over time a set of recognizable patterns of features across similar artifacts. Genre is one such grou** of artifacts for narrative media with similar patterns, tropes, and story structures. While much work has been done on genre-based classifications in text and video, we present a novel approach to do a multi-modal analysis of genre based on comics and manga-style visual narratives. We present a systematic feature analysis of an annotated dataset that includes a variety of western and eastern visual books with annotations for high-level narrative patterns. We then present a detailed analysis of the contributions of high-level features to genre classification for this medium. We highlight some of the limitations and challenges of our existing computational approaches in modeling subjective labels. Our contributions to the community are: a dataset of annotated manga books, a multi-modal analysis of visual panels and text in a constrained and popular medium through high-level features, and a systematic process for incorporating subjective narrative patterns in computational models.
△ Less
Submitted 14 December, 2023;
originally announced December 2023.
-
CPST: Comprehension-Preserving Style Transfer for Multi-Modal Narratives
Authors:
Yi-Chun Chen,
Arnav Jhala
Abstract:
We investigate the challenges of style transfer in multi-modal visual narratives. Among static visual narratives such as comics and manga, there are distinct visual styles in terms of presentation. They include style features across multiple dimensions, such as panel layout, size, shape, and color. They include both visual and text media elements. The layout of both text and media elements is also…
▽ More
We investigate the challenges of style transfer in multi-modal visual narratives. Among static visual narratives such as comics and manga, there are distinct visual styles in terms of presentation. They include style features across multiple dimensions, such as panel layout, size, shape, and color. They include both visual and text media elements. The layout of both text and media elements is also significant in terms of narrative communication. The sequential transitions between panels are where readers make inferences about the narrative world. These feature differences provide an interesting challenge for style transfer in which there are distinctions between the processing of features for each modality. We introduce the notion of comprehension-preserving style transfer (CPST) in such multi-modal domains. CPST requires not only traditional metrics of style transfer but also metrics of narrative comprehension. To spur further research in this area, we present an annotated dataset of comics and manga and an initial set of algorithms that utilize separate style transfer modules for the visual, textual, and layout parameters. To test whether the style transfer preserves narrative semantics, we evaluate this algorithm through visual story cloze tests inspired by work in computational cognition of narrative systems. Understanding the connection between style and narrative semantics provides insight for applications ranging from informational brochure designs to data storytelling.
△ Less
Submitted 14 December, 2023;
originally announced December 2023.
-
Modeling Risk in Reinforcement Learning: A Literature Map**
Authors:
Leonardo Villalobos-Arias,
Derek Martin,
Abhijeet Krishnan,
Madeleine Gagné,
Colin M. Potts,
Arnav Jhala
Abstract:
Safe reinforcement learning deals with mitigating or avoiding unsafe situations by reinforcement learning (RL) agents. Safe RL approaches are based on specific risk representations for particular problems or domains. In order to analyze agent behaviors, compare safe RL approaches, and effectively transfer techniques between application domains, it is necessary to understand the types of risk speci…
▽ More
Safe reinforcement learning deals with mitigating or avoiding unsafe situations by reinforcement learning (RL) agents. Safe RL approaches are based on specific risk representations for particular problems or domains. In order to analyze agent behaviors, compare safe RL approaches, and effectively transfer techniques between application domains, it is necessary to understand the types of risk specific to safe RL problems. We performed a systematic literature map** with the objective to characterize risk in safe RL. Based on the obtained results, we present definitions, characteristics, and types of risk that hold on multiple application domains. Our literature map** covers literature from the last 5 years (2017-2022), from a variety of knowledge areas (AI, finance, engineering, medicine) where RL approaches emphasize risk representation and management. Our map** covers 72 papers filtered systematically from over thousands of papers on the topic. Our proposed notion of risk covers a variety of representations, disciplinary differences, common training exercises, and types of techniques. We encourage researchers to include explicit and detailed accounts of risk in future safe RL research reports, using this map** as a starting point. With this information, researchers and practitioners could draw stronger conclusions on the effectiveness of techniques on different problems.
△ Less
Submitted 8 December, 2023;
originally announced December 2023.
-
Emergent social NPC interactions in the Social NPCs Skyrim mod and beyond
Authors:
Manuel Guimarães,
Pedro A. Santos,
Arnav Jhala
Abstract:
This work presents an implementation of a social architecture model for authoring Non-Player Character (NPC) in open world games inspired in academic research on agentbased modeling. Believable NPC authoring is burdensome in terms of rich dialogue and responsive behaviors.
We briefly present the characteristics and advantages of using a social agent architecture for this task and describe an imp…
▽ More
This work presents an implementation of a social architecture model for authoring Non-Player Character (NPC) in open world games inspired in academic research on agentbased modeling. Believable NPC authoring is burdensome in terms of rich dialogue and responsive behaviors.
We briefly present the characteristics and advantages of using a social agent architecture for this task and describe an implementation of a social agent architecture CiF-CK released as a mod Social NPCs for The Elder Scrolls V: Skyrim
△ Less
Submitted 20 January, 2023; v1 submitted 27 July, 2022;
originally announced July 2022.
