Showing 1–50 of 61 results for author: Lisý, V

Counteracting Concept Drift by Learning with Future Malware Predictions

Authors: Branislav Bosansky, Lada Hospodkova, Michal Najman, Maria Rigaki, Elnaz Babayeva, Viliam Lisy

Abstract: The accuracy of deployed malware-detection classifiers degrades over time due to changes in data distributions and increasing discrepancies between training and testing data. This phenomenon is known as the concept drift. While the concept drift can be caused by various reasons in general, new malicious files are created by malware authors with a clear intention of avoiding detection. The existenc… ▽ More The accuracy of deployed malware-detection classifiers degrades over time due to changes in data distributions and increasing discrepancies between training and testing data. This phenomenon is known as the concept drift. While the concept drift can be caused by various reasons in general, new malicious files are created by malware authors with a clear intention of avoiding detection. The existence of the intention opens a possibility for predicting such future samples. Including predicted samples in training data should consequently increase the accuracy of the classifiers on new testing data. We compare two methods for predicting future samples: (1) adversarial training and (2) generative adversarial networks (GANs). The first method explicitly seeks for adversarial examples against the classifier that are then used as a part of training data. Similarly, GANs also generate synthetic training data. We use GANs to learn changes in data distributions within different time periods of training data and then apply these changes to generate samples that could be in testing data. We compare these prediction methods on two different datasets: (1) Ember public dataset and (2) the internal dataset of files incoming to Avast. We show that while adversarial training yields more robust classifiers, this method is not a good predictor of future malware in general. This is in contrast with previously reported positive results in different domains (including natural language processing and spam detection). On the other hand, we show that GANs can be successfully used as predictors of future malware. We specifically examine malware families that exhibit significant changes in their data distributions over time and the experimental results confirm that GAN-based predictions can significantly improve the accuracy of the classifier on new, previously unseen data. △ Less

Submitted 14 April, 2024; originally announced April 2024.

Look-ahead Search on Top of Policy Networks in Imperfect Information Games

Authors: Ondrej Kubicek, Neil Burch, Viliam Lisy

Abstract: Search in test time is often used to improve the performance of reinforcement learning algorithms. Performing theoretically sound search in fully adversarial two-player games with imperfect information is notoriously difficult and requires a complicated training process. We present a method for adding test-time search to an arbitrary policy-gradient algorithm that learns from sampled trajectories.… ▽ More Search in test time is often used to improve the performance of reinforcement learning algorithms. Performing theoretically sound search in fully adversarial two-player games with imperfect information is notoriously difficult and requires a complicated training process. We present a method for adding test-time search to an arbitrary policy-gradient algorithm that learns from sampled trajectories. Besides the policy network, the algorithm trains an additional critic network, which estimates the expected values of players following various transformations of the policies given by the policy network. These values are then used for depth-limited search. We show how the values from this critic can create a value function for imperfect information games. Moreover, they can be used to compute the summary statistics necessary to start the search from an arbitrary decision point in the game. The presented algorithm is scalable to very large games since it does not require any search during train time. We evaluate the algorithm's performance when trained along Regularized Nash Dynamics, and we evaluate the benefit of using the search in the standard benchmark game of Leduc hold'em, multiple variants of imperfect information Goofspiel, and Battleships. △ Less

Submitted 6 February, 2024; v1 submitted 23 December, 2023; originally announced December 2023.

Generation of Games for Opponent Model Differentiation

Authors: David Milec, Viliam Lisý, Christopher Kiekintveld

Abstract: Protecting against adversarial attacks is a common multiagent problem. Attackers in the real world are predominantly human actors, and the protection methods often incorporate opponent models to improve the performance when facing humans. Previous results show that modeling human behavior can significantly improve the performance of the algorithms. However, modeling humans correctly is a complex p… ▽ More Protecting against adversarial attacks is a common multiagent problem. Attackers in the real world are predominantly human actors, and the protection methods often incorporate opponent models to improve the performance when facing humans. Previous results show that modeling human behavior can significantly improve the performance of the algorithms. However, modeling humans correctly is a complex problem, and the models are often simplified and assume humans make mistakes according to some distribution or train parameters for the whole population from which they sample. In this work, we use data gathered by psychologists who identified personality types that increase the likelihood of performing malicious acts. However, in the previous work, the tests on a handmade game could not show strategic differences between the models. We created a novel model that links its parameters to psychological traits. We optimized over parametrized games and created games in which the differences are profound. Our work can help with automatic game generation when we need a game in which some models will behave differently and to identify situations in which the models do not align. △ Less

Submitted 28 November, 2023; originally announced November 2023.

Comments: 4 pages

Revisiting the work "Brownian motion with time-dependent friction and single-particle dynamics in liquids" by Lad, Patel, and Pratap [Phys. Rev. E 105, 064107 (2022)]

Authors: Vladimir Lisy, Jana Tothova

Abstract: Recently, Lad, Patel, and Pratap (LP&P) [Phys. Rev. E 105, 064107 (2022)] revisited a microscopic theory of molecular motion in liquids, proposed by Glass and Rice [Phys. Rev. 176, 239 (1968)]. Coming from this theory, LP&P derived a new equation of motion for the velocity autocorrelation function (VAF) and argued that the friction coefficient of particles in liquids should exponentially depend on… ▽ More Recently, Lad, Patel, and Pratap (LP&P) [Phys. Rev. E 105, 064107 (2022)] revisited a microscopic theory of molecular motion in liquids, proposed by Glass and Rice [Phys. Rev. 176, 239 (1968)]. Coming from this theory, LP&P derived a new equation of motion for the velocity autocorrelation function (VAF) and argued that the friction coefficient of particles in liquids should exponentially depend on time. The numerical solution of this equation was fitted to the results of molecular dynamics simulations on different liquids. In our Comment [Phys. Rev. E 108, 036107 (2023)], we showed that this solution, obtained under the condition of zero derivative of the VAF at time t = 0, is physically incorrect. This was evidenced by our exact analytical solution for the VAF, not found by LP&P, and numerically, by using the same method as in the commented work. In the Reply [Phys. Rev. E 108, 036108 (2023)], Lad, Patel, Pratap, and Pandya claimed that our solution does not satisfy all the necessary boundary conditions and is thus not appropriate for the description of atomic dynamics in liquids. Until and unless proven otherwise they do not find any reason for the reconsideration of their theory. Here we give a rebuttal to this Reply and, returning to the original work by LP&P, show that the presented there equation for the VAF is wrong. Due to errors in its derivation, it is, among other inconsistencies, incompatible precisely with the boundary conditions for the VAF which lie in the basis of their theory. △ Less

Submitted 28 January, 2024; v1 submitted 28 October, 2023; originally announced October 2023.

Comments: 11 pages, 4 figures

NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios

Authors: Jaromír Janisch, Tomáš Pevný, Viliam Lisý

Abstract: Current frameworks for training offensive penetration testing agents with deep reinforcement learning struggle to produce agents that perform well in real-world scenarios, due to the reality gap in simulation-based frameworks and the lack of scalability in emulation-based frameworks. Additionally, existing frameworks often use an unrealistic metric that measures the agents' performance on the trai… ▽ More Current frameworks for training offensive penetration testing agents with deep reinforcement learning struggle to produce agents that perform well in real-world scenarios, due to the reality gap in simulation-based frameworks and the lack of scalability in emulation-based frameworks. Additionally, existing frameworks often use an unrealistic metric that measures the agents' performance on the training data. NASimEmu, a new framework introduced in this paper, addresses these issues by providing both a simulator and an emulator with a shared interface. This approach allows agents to be trained in simulation and deployed in the emulator, thus verifying the realism of the used abstraction. Our framework promotes the development of general agents that can transfer to novel scenarios unseen during their training. For the simulation part, we adopt an existing simulator NASim and enhance its realism. The emulator is implemented with industry-level tools, such as Vagrant, VirtualBox, and Metasploit. Experiments demonstrate that a simulation-trained agent can be deployed in emulation, and we show how to use the framework to train a general agent that transfers into novel, structurally different scenarios. NASimEmu is available as open-source. △ Less

Submitted 18 August, 2023; v1 submitted 26 May, 2023; originally announced May 2023.

Comments: NASimEmu is available at https://github.com/jaromiru/NASimEmu and the baseline agents at https://github.com/jaromiru/NASimEmu-agents

Avast-CTU Public CAPE Dataset

Authors: Branislav Bosansky, Dominik Kouba, Ondrej Manhal, Thorsten Sick, Viliam Lisy, Jakub Kroustek, Petr Somol

Abstract: There is a limited amount of publicly available data to support research in malware analysis technology. Particularly, there are virtually no publicly available datasets generated from rich sandboxes such as Cuckoo/CAPE. The benefit of using dynamic sandboxes is the realistic simulation of file execution in the target machine and obtaining a log of such execution. The machine can be infected by ma… ▽ More There is a limited amount of publicly available data to support research in malware analysis technology. Particularly, there are virtually no publicly available datasets generated from rich sandboxes such as Cuckoo/CAPE. The benefit of using dynamic sandboxes is the realistic simulation of file execution in the target machine and obtaining a log of such execution. The machine can be infected by malware hence there is a good chance of capturing the malicious behavior in the execution logs, thus allowing researchers to study such behavior in detail. Although the subsequent analysis of log information is extensively covered in industrial cybersecurity backends, to our knowledge there has been only limited effort invested in academia to advance such log analysis capabilities using cutting edge techniques. We make this sample dataset available to support designing new machine learning methods for malware detection, especially for automatic detection of generic malicious behavior. The dataset has been collected in cooperation between Avast Software and Czech Technical University - AI Center (AIC). △ Less

Submitted 6 September, 2022; originally announced September 2022.

Explaining Classifiers Trained on Raw Hierarchical Multiple-Instance Data

Authors: Tomáš Pevný, Viliam Lisý, Branislav Bošanský, Petr Somol, Michal Pěchouček

Abstract: Learning from raw data input, thus limiting the need for feature engineering, is a component of many successful applications of machine learning methods in various domains. While many problems naturally translate into a vector representation directly usable in standard classifiers, a number of data sources have the natural form of structured data interchange formats (e.g., security logs in JSON/XM… ▽ More Learning from raw data input, thus limiting the need for feature engineering, is a component of many successful applications of machine learning methods in various domains. While many problems naturally translate into a vector representation directly usable in standard classifiers, a number of data sources have the natural form of structured data interchange formats (e.g., security logs in JSON/XML format). Existing methods, such as in Hierarchical Multiple Instance Learning (HMIL), allow learning from such data in their raw form. However, the explanation of the classifiers trained on raw structured data remains largely unexplored. By treating these models as sub-set selections problems, we demonstrate how interpretable explanations, with favourable properties, can be generated using computationally efficient algorithms. We compare to an explanation technique adopted from graph neural networks showing an order of magnitude speed-up and higher-quality explanations. △ Less

Submitted 4 August, 2022; originally announced August 2022.

Continual Depth-limited Responses for Computing Counter-strategies in Sequential Games

Authors: David Milec, Ondřej Kubíček, Viliam Lisý

Abstract: In zero-sum games, the optimal strategy is well-defined by the Nash equilibrium. However, it is overly conservative when playing against suboptimal opponents and it can not exploit their weaknesses. Limited look-ahead game solving in imperfect-information games allows defeating human experts in massive real-world games such as Poker, Liar's Dice, and Scotland Yard. However, since they approximate… ▽ More In zero-sum games, the optimal strategy is well-defined by the Nash equilibrium. However, it is overly conservative when playing against suboptimal opponents and it can not exploit their weaknesses. Limited look-ahead game solving in imperfect-information games allows defeating human experts in massive real-world games such as Poker, Liar's Dice, and Scotland Yard. However, since they approximate Nash equilibrium, they tend to only win slightly against weak opponents. We propose methods combining limited look-ahead solving with an opponent model in order to 1) approximate a best response in large games or 2) compute a robust response with control over the robustness of the response. Both methods can compute the response in real time to previously unseen strategies. We present theoretical guarantees of our methods. We show that existing robust response methods do not work combined with limited look-ahead solving of the shelf, and we propose a novel solution for the issue. Our algorithm performs significantly better than multiple baselines in smaller games and outperforms state-of-the-art methods against SlumBot. △ Less

Submitted 3 April, 2024; v1 submitted 23 December, 2021; originally announced December 2021.

Comments: 16 pages, 15 figures

Revisiting Game Representations: The Hidden Costs of Efficiency in Sequential Decision-making Algorithms

Authors: Vojtěch Kovařík, David Milec, Michal Šustr, Dominik Seitz, Viliam Lisý

Abstract: Recent advancements in algorithms for sequential decision-making under imperfect information have shown remarkable success in large games such as limit- and no-limit poker. These algorithms traditionally formalize the games using the extensive-form game formalism, which, as we show, while theoretically sound, is memory-inefficient and computationally intensive in practice. To mitigate these challe… ▽ More Recent advancements in algorithms for sequential decision-making under imperfect information have shown remarkable success in large games such as limit- and no-limit poker. These algorithms traditionally formalize the games using the extensive-form game formalism, which, as we show, while theoretically sound, is memory-inefficient and computationally intensive in practice. To mitigate these challenges, a popular workaround involves using a specialized representation based on player specific information-state trees. However, as we show, this alternative significantly narrows the set of games that can be represented efficiently. In this study, we identify the set of large games on which modern algorithms have been benchmarked as being naturally represented by Sequential Bayesian Games. We elucidate the critical differences between extensive-form game and sequential Bayesian game representations, both theoretically and empirically. We further argue that the impressive experimental results often cited in the literature may be skewed, as they frequently stem from testing these algorithms only on this restricted class of games. By understanding these nuances, we aim to guide future research in develo** more universally applicable and efficient algorithms for sequential decision-making under imperfect information. △ Less

Submitted 5 December, 2023; v1 submitted 20 December, 2021; originally announced December 2021.

Improving Robustness of Malware Classifiers using Adversarial Strings Generated from Perturbed Latent Representations

Authors: Marek Galovic, Branislav Bosansky, Viliam Lisy

Abstract: In malware behavioral analysis, the list of accessed and created files very often indicates whether the examined file is malicious or benign. However, malware authors are trying to avoid detection by generating random filenames and/or modifying used filenames with new versions of the malware. These changes represent real-world adversarial examples. The goal of this work is to generate realistic ad… ▽ More In malware behavioral analysis, the list of accessed and created files very often indicates whether the examined file is malicious or benign. However, malware authors are trying to avoid detection by generating random filenames and/or modifying used filenames with new versions of the malware. These changes represent real-world adversarial examples. The goal of this work is to generate realistic adversarial examples and improve the classifier's robustness against these attacks. Our approach learns latent representations of input strings in an unsupervised fashion and uses gradient-based adversarial attack methods in the latent domain to generate adversarial examples in the input domain. We use these examples to improve the classifier's robustness by training on the generated adversarial set of strings. Compared to classifiers trained only on perturbed latent vectors, our approach produces classifiers that are significantly more robust without a large trade-off in standard accuracy. △ Less

Submitted 22 October, 2021; originally announced October 2021.

Comments: 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Strategic ML Workshop

Mill.jl and JsonGrinder.jl: automated differentiable feature extraction for learning from raw JSON data

Authors: Simon Mandlik, Matej Racinsky, Viliam Lisy, Tomas Pevny

Abstract: Learning from raw data input, thus limiting the need for manual feature engineering, is one of the key components of many successful applications of machine learning methods. While machine learning problems are often formulated on data that naturally translate into a vector representation suitable for classifiers, there are data sources, for example in cybersecurity, that are naturally represented… ▽ More Learning from raw data input, thus limiting the need for manual feature engineering, is one of the key components of many successful applications of machine learning methods. While machine learning problems are often formulated on data that naturally translate into a vector representation suitable for classifiers, there are data sources, for example in cybersecurity, that are naturally represented in diverse files with a unifying hierarchical structure, such as XML, JSON, and Protocol Buffers. Converting this data to vector (tensor) representation is generally done by manual feature engineering, which is laborious, lossy, and prone to human bias about the importance of particular features. Mill and JsonGrinder is a tandem of libraries, which fully automates the conversion. Starting with an arbitrary set of JSON samples, they create a differentiable machine learning model capable of infer from further JSON samples in their raw form. △ Less

Submitted 19 May, 2021; originally announced May 2021.

Comments: 5 pages, 2 figures, 1 table, submitted to section on one-source software of Journal of Machine Learning Research

Multi-agent Reinforcement Learning in OpenSpiel: A Reproduction Report

Authors: Michael Walton, Viliam Lisy

Abstract: In this report, we present results reproductions for several core algorithms implemented in the OpenSpiel framework for learning in games. The primary contribution of this work is a validation of OpenSpiel's re-implemented search and Reinforcement Learning algorithms against the results reported in their respective originating works. Additionally, we provide complete documentation of hyperparamete… ▽ More In this report, we present results reproductions for several core algorithms implemented in the OpenSpiel framework for learning in games. The primary contribution of this work is a validation of OpenSpiel's re-implemented search and Reinforcement Learning algorithms against the results reported in their respective originating works. Additionally, we provide complete documentation of hyperparameters and source code required to reproduce these experiments easily and exactly. △ Less

Submitted 1 March, 2021; v1 submitted 27 February, 2021; originally announced March 2021.

Complexity and Algorithms for Exploiting Quantal Opponents in Large Two-Player Games

Authors: David Milec, Jakub Černý, Viliam Lisý, Bo An

Abstract: Solution concepts of traditional game theory assume entirely rational players; therefore, their ability to exploit subrational opponents is limited. One type of subrationality that describes human behavior well is the quantal response. While there exist algorithms for computing solutions against quantal opponents, they either do not scale or may provide strategies that are even worse than the enti… ▽ More Solution concepts of traditional game theory assume entirely rational players; therefore, their ability to exploit subrational opponents is limited. One type of subrationality that describes human behavior well is the quantal response. While there exist algorithms for computing solutions against quantal opponents, they either do not scale or may provide strategies that are even worse than the entirely-rational Nash strategies. This paper aims to analyze and propose scalable algorithms for computing effective and robust strategies against a quantal opponent in normal-form and extensive-form games. Our contributions are: (1) we define two different solution concepts related to exploiting quantal opponents and analyze their properties; (2) we prove that computing these solutions is computationally hard; (3) therefore, we evaluate several heuristic approximations based on scalable counterfactual regret minimization (CFR); and (4) we identify a CFR variant that exploits the bounded opponents better than the previously used variants while being less exploitable by the worst-case perfectly-rational opponent. △ Less

Submitted 16 December, 2020; v1 submitted 30 September, 2020; originally announced September 2020.

Comments: 15 pages, 11 figures, submitted to AAAI 2021

Journal ref: Proceedings of the AAAI Conference on Artificial Intelligence, 35(6), 5575-5583 (2021)

Symbolic Relational Deep Reinforcement Learning based on Graph Neural Networks and Autoregressive Policy Decomposition

Authors: Jaromír Janisch, Tomáš Pevný, Viliam Lisý

Abstract: We focus on reinforcement learning (RL) in relational problems that are naturally defined in terms of objects, their relations, and object-centric actions. These problems are characterized by variable state and action spaces, and finding a fixed-length representation, required by most existing RL methods, is difficult, if not impossible. We present a deep RL framework based on graph neural network… ▽ More We focus on reinforcement learning (RL) in relational problems that are naturally defined in terms of objects, their relations, and object-centric actions. These problems are characterized by variable state and action spaces, and finding a fixed-length representation, required by most existing RL methods, is difficult, if not impossible. We present a deep RL framework based on graph neural networks and auto-regressive policy decomposition that naturally works with these problems and is completely domain-independent. We demonstrate the framework's broad applicability in three distinct domains and show impressive zero-shot generalization over different problem sizes. △ Less

Submitted 25 August, 2023; v1 submitted 25 September, 2020; originally announced September 2020.

Comments: code available at https://github.com/jaromiru/sr-drl

Generalized Langevin equations and fluctuation-dissipation theorem for particle-bath systems in electric and magnetic fields

Authors: Vladimir Lisy, Jana Tothova

Abstract: The Brownian motion of a particle immersed in a medium of charged particles is considered when the system is placed in magnetic or electric fields. Coming from the Zwanzig-Caldeira-Legget particle-bath model, we modify it so that not only the charged Brownian particle (BP) but also the bath particles respond to the external fields. For stationary systems the generalized Langevin equations are deri… ▽ More The Brownian motion of a particle immersed in a medium of charged particles is considered when the system is placed in magnetic or electric fields. Coming from the Zwanzig-Caldeira-Legget particle-bath model, we modify it so that not only the charged Brownian particle (BP) but also the bath particles respond to the external fields. For stationary systems the generalized Langevin equations are derived. Arbitrarily time-dependent electric fields do not affect the memory functions, the thermal noise force, and the BP velocity correlation functions. In the case of a constant magnetic field two equations with different memory functions are obtained for the BP motion in the plane perpendicular to the field. As distinct from the previous theories, the random thermal force depends on the field magnitude. Its time correlation function is connected with one of the found memory functions through the familiar second fluctuation-dissipation theorem. △ Less

Submitted 23 September, 2020; originally announced September 2020.

Comments: 8 pages, no figures

Discovering Imperfectly Observable Adversarial Actions using Anomaly Detection

Authors: Olga Petrova, Karel Durkota, Galina Alperovich, Karel Horak, Michal Najman, Branislav Bosansky, Viliam Lisy

Abstract: Anomaly detection is a method for discovering unusual and suspicious behavior. In many real-world scenarios, the examined events can be directly linked to the actions of an adversary, such as attacks on computer networks or frauds in financial operations. While the defender wants to discover such malicious behavior, the attacker seeks to accomplish their goal (e.g., exfiltrating data) while avoidi… ▽ More Anomaly detection is a method for discovering unusual and suspicious behavior. In many real-world scenarios, the examined events can be directly linked to the actions of an adversary, such as attacks on computer networks or frauds in financial operations. While the defender wants to discover such malicious behavior, the attacker seeks to accomplish their goal (e.g., exfiltrating data) while avoiding the detection. To this end, anomaly detectors have been used in a game-theoretic framework that captures these goals of a two-player competition. We extend the existing models to more realistic settings by (1) allowing both players to have continuous action spaces and by assuming that (2) the defender cannot perfectly observe the action of the attacker. We propose two algorithms for solving such games -- a direct extension of existing algorithms based on discretizing the feature space and linear programming and the second algorithm based on constrained learning. Experiments show that both algorithms are applicable for cases with low feature space dimensions but the learning-based method produces less exploitable strategies and it is scalable to higher dimensions. Moreover, we use real-world data to compare our approaches with existing classifiers in a data-exfiltration scenario via the DNS channel. The results show that our models are significantly less exploitable by an informed attacker. △ Less

Submitted 22 April, 2020; originally announced April 2020.

Comments: 9 pages, 3 figures, 3 tables. Extended Abstract of this paper is accepted to AAMAS 2020

MSC Class: 91A80 (Primary); 68T99 (Secondary) ACM Class: I.2.6; G.3

Classification with Costly Features in Hierarchical Deep Sets

Authors: Jaromír Janisch, Tomáš Pevný, Viliam Lisý

Abstract: Classification with Costly Features (CwCF) is a classification problem that includes the cost of features in the optimization criteria. Individually for each sample, its features are sequentially acquired to maximize accuracy while minimizing the acquired features' cost. However, existing approaches can only process data that can be expressed as vectors of fixed length. In real life, the data ofte… ▽ More Classification with Costly Features (CwCF) is a classification problem that includes the cost of features in the optimization criteria. Individually for each sample, its features are sequentially acquired to maximize accuracy while minimizing the acquired features' cost. However, existing approaches can only process data that can be expressed as vectors of fixed length. In real life, the data often possesses rich and complex structure, which can be more precisely described with formats such as XML or JSON. The data is hierarchical and often contains nested lists of objects. In this work, we extend an existing deep reinforcement learning-based algorithm with hierarchical deep sets and hierarchical softmax, so that it can directly process this data. The extended method has greater control over which features it can acquire and, in experiments with seven datasets, we show that this leads to superior performance. To showcase the real usage of the new method, we apply it to a real-life problem of classifying malicious web domains, using an online service. △ Less

Submitted 29 February, 2024; v1 submitted 20 November, 2019; originally announced November 2019.

Comments: formerly Hierarchical Multiple-Instance Data Classification with Costly Features; RL4RealLife @ ICML2021; code available at https://github.com/jaromiru/rcwcf

Classification with Costly Features as a Sequential Decision-Making Problem

Authors: Jaromír Janisch, Tomáš Pevný, Viliam Lisý

Abstract: This work focuses on a specific classification problem, where the information about a sample is not readily available, but has to be acquired for a cost, and there is a per-sample budget. Inspired by real-world use-cases, we analyze average and hard variations of a directly specified budget. We postulate the problem in its explicit formulation and then convert it into an equivalent MDP, that can b… ▽ More This work focuses on a specific classification problem, where the information about a sample is not readily available, but has to be acquired for a cost, and there is a per-sample budget. Inspired by real-world use-cases, we analyze average and hard variations of a directly specified budget. We postulate the problem in its explicit formulation and then convert it into an equivalent MDP, that can be solved with deep reinforcement learning. Also, we evaluate a real-world inspired setting with sparse training dataset with missing features. The presented method performs robustly well in all settings across several distinct datasets, outperforming other prior-art algorithms. The method is flexible, as showcased with all mentioned modifications and can be improved with any domain independent advancement in RL. △ Less

Submitted 5 September, 2019; originally announced September 2019.

Journal ref: Machine Learning (2020): 1-29

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Authors: Vojtěch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, Viliam Lisý

Abstract: Multiagent decision-making in partially observable environments is usually modelled as either an extensive-form game (EFG) in game theory or a partially observable stochastic game (POSG) in multiagent reinforcement learning (MARL). One issue with the current situation is that while most practical problems can be modelled in both formalisms, the relationship of the two models is unclear, which hind… ▽ More Multiagent decision-making in partially observable environments is usually modelled as either an extensive-form game (EFG) in game theory or a partially observable stochastic game (POSG) in multiagent reinforcement learning (MARL). One issue with the current situation is that while most practical problems can be modelled in both formalisms, the relationship of the two models is unclear, which hinders the transfer of ideas between the two communities. A second issue is that while EFGs have recently seen significant algorithmic progress, their classical formalization is unsuitable for efficient presentation of the underlying ideas, such as those around decomposition. To solve the first issue, we introduce factored-observation stochastic games (FOSGs), a minor modification of the POSG formalism which distinguishes between private and public observation and thereby greatly simplifies decomposition. To remedy the second issue, we show that FOSGs and POSGs are naturally connected to EFGs: by "unrolling" a FOSG into its tree form, we obtain an EFG. Conversely, any perfect-recall timeable EFG corresponds to some underlying FOSG in this manner. Moreover, this relationship justifies several minor modifications to the classical EFG formalization that recently appeared as an implicit response to the model's issues with decomposition. Finally, we illustrate the transfer of ideas between EFGs and MARL by presenting three key EFG techniques -- counterfactual regret minimization, sequence form, and decomposition -- in the FOSG framework. △ Less

Submitted 28 September, 2021; v1 submitted 26 June, 2019; originally announced June 2019.

Comments: A 2020 update of the original 2019 version of the paper. (Rewrote the main text and clarified the relationship between FOSGs/POSGs and EFGs. Some of the technical results are now presented in the appendix.)

Value Functions for Depth-Limited Solving in Zero-Sum Imperfect-Information Games

Authors: Vojtěch Kovařík, Dominik Seitz, Viliam Lisý, Jan Rudolf, Shuo Sun, Karel Ha

Abstract: We provide a formal definition of depth-limited games together with an accessible and rigorous explanation of the underlying concepts, both of which were previously missing in imperfect-information games. The definition works for an arbitrary extensive-form game and is not tied to any specific game-solving algorithm. Moreover, this framework unifies and significantly extends three approaches to de… ▽ More We provide a formal definition of depth-limited games together with an accessible and rigorous explanation of the underlying concepts, both of which were previously missing in imperfect-information games. The definition works for an arbitrary extensive-form game and is not tied to any specific game-solving algorithm. Moreover, this framework unifies and significantly extends three approaches to depth-limited solving that previously existed in extensive-form games and multiagent reinforcement learning but were not known to be compatible. A key ingredient of these depth-limited games are value functions. Focusing on two-player zero-sum imperfect-information games, we show how to obtain optimal value functions and prove that public information provides both necessary and sufficient context for computing them. We provide a domain-independent encoding of the domains that allows for approximating value functions even by simple feed-forward neural networks, which are then able to generalize to unseen parts of the game. We use the resulting value network to implement a depth-limited version of counterfactual regret minimization. In three distinct domains, we show that the algorithm's exploitability is roughly linearly dependent on the value network's quality and that it is not difficult to train a value network with which depth-limited CFR's performance is as good as that of CFR with access to the full game. △ Less

Submitted 24 March, 2022; v1 submitted 31 May, 2019; originally announced June 2019.

Comments: The first two authors contributed equally

Problems with the EFG formalism: a solution attempt using observations

Authors: Vojtěch Kovařík, Viliam Lisý

Abstract: We argue that the extensive-form game (EFG) model isn't powerful enough to express all important aspects of imperfect information games, such as those related to decomposition and online game solving. We present a principled attempt to fix the formalism by considering information partitions that correspond to observations. We show that EFGs cannot be "fixed" without additional knowledge about the… ▽ More We argue that the extensive-form game (EFG) model isn't powerful enough to express all important aspects of imperfect information games, such as those related to decomposition and online game solving. We present a principled attempt to fix the formalism by considering information partitions that correspond to observations. We show that EFGs cannot be "fixed" without additional knowledge about the original task, and show how to extend the EFG model under this assumption. However, during our investigation, we ultimately concluded that a better solution is to abandon EFGs entirely and instead adopt a model based on partially observable stochastic games. The primary contribution of the text thus lies in exposing the problems with EFGs and presenting a detailed study on introducing imperfect information by adding observations into an underlying perfect-information model. △ Less

Submitted 14 June, 2019; originally announced June 2019.

Monte Carlo Continual Resolving for Online Strategy Computation in Imperfect Information Games

Authors: Michal Sustr, Vojtech Kovarik, Viliam Lisy

Abstract: Online game playing algorithms produce high-quality strategies with a fraction of memory and computation required by their offline alternatives. Continual Resolving (CR) is a recent theoretically sound approach to online game playing that has been used to outperform human professionals in poker. However, parts of the algorithm were specific to poker, which enjoys many properties not shared by othe… ▽ More Online game playing algorithms produce high-quality strategies with a fraction of memory and computation required by their offline alternatives. Continual Resolving (CR) is a recent theoretically sound approach to online game playing that has been used to outperform human professionals in poker. However, parts of the algorithm were specific to poker, which enjoys many properties not shared by other imperfect information games. We present a domain-independent formulation of CR applicable to any two-player zero-sum extensive-form games that works with an abstract resolving algorithm. We further describe and implement its Monte Carlo variant (MCCR) which uses Monte Carlo Counterfactual Regret Minimization (MCCFR) as a resolver. We prove the correctness of CR and show an $O(T^{-1/2})$-dependence of MCCR's exploitability on the computation time. Furthermore, we present an empirical comparison of MCCR with incremental tree building to Online Outcome Sampling and Information-set MCTS on several domains. △ Less

Submitted 8 March, 2019; v1 submitted 18 December, 2018; originally announced December 2018.

NMR measurements and all-time Brownian movement with memory

Authors: Jana Tothova, Vladimir Lisy

Abstract: In the present work, by using the method of accumulation of phase shifts in the rotating frame, the attenuation function S(t) of the NMR signal from an ensemble of spin-bearing particles in a magnetic-field gradient is expressed through the particle mean square displacement in a form applicable for any kind of stationary stochastic dynamics of spins and for any times. S(t) is evaluated providing t… ▽ More In the present work, by using the method of accumulation of phase shifts in the rotating frame, the attenuation function S(t) of the NMR signal from an ensemble of spin-bearing particles in a magnetic-field gradient is expressed through the particle mean square displacement in a form applicable for any kind of stationary stochastic dynamics of spins and for any times. S(t) is evaluated providing that the random motion of particles can be modeled by the generalized Langevin equation (GLE) with a colored random force driving the particles. The memory integral in this equation is the convolution of the particle velocity or its acceleration with a memory kernel related to the random force by the fluctuation-dissipation theorem. We consider three popular models of the BM with memory: the model of viscoelastic (Maxwell) fluids with the memory exponentially decaying in time, the fractional BM model, and the model of the hydrodynamic BM. In all the cases the solutions of the GLEs are obtained in an exceedingly simple way. The corresponding attenuation functions are then found for the free-induction NMR signal and the pulsed and steady-gradient spin-echo experiments. The results for the free-particle fractional BM compare favorably with experiments acquired in human neuronal tissues and with the observed subdiffusion dynamics in proteins. △ Less

Submitted 24 September, 2018; originally announced September 2018.

Comments: arXiv admin note: substantial text overlap with arXiv:1701.02486, arXiv:1803.01544, arXiv:1803.01554

Journal ref: In: Horizons in World Physics, Editor A. Reimer. Vol. 296 (2018) Chapter 2, p. 59-104, ISBN 978-1-53614-125-2

Analysis of Hannan Consistent Selection for Monte Carlo Tree Search in Simultaneous Move Games

Authors: Vojtěch Kovařík, Viliam Lisý

Abstract: Hannan consistency, or no external regret, is a~key concept for learning in games. An action selection algorithm is Hannan consistent (HC) if its performance is eventually as good as selecting the~best fixed action in hindsight. If both players in a~zero-sum normal form game use a~Hannan consistent algorithm, their average behavior converges to a~Nash equilibrium (NE) of the~game. A similar result… ▽ More Hannan consistency, or no external regret, is a~key concept for learning in games. An action selection algorithm is Hannan consistent (HC) if its performance is eventually as good as selecting the~best fixed action in hindsight. If both players in a~zero-sum normal form game use a~Hannan consistent algorithm, their average behavior converges to a~Nash equilibrium (NE) of the~game. A similar result is known about extensive form games, but the~played strategies need to be Hannan consistent with respect to the~counterfactual values, which are often difficult to obtain. We study zero-sum extensive form games with simultaneous moves, but otherwise perfect information. These games generalize normal form games and they are a special case of extensive form games. We study whether applying HC algorithms in each decision point of these games directly to the~observed payoffs leads to convergence to a~Nash equilibrium. This learning process corresponds to a~class of Monte Carlo Tree Search algorithms, which are popular for playing simultaneous-move games but do not have any known performance guarantees. We show that using HC algorithms directly on the~observed payoffs is not sufficient to guarantee the~convergence. With an~additional averaging over joint actions, the~convergence is guaranteed, but empirically slower. We further define an~additional property of HC algorithms, which is sufficient to guarantee the~convergence without the~averaging and we empirically show that commonly used HC algorithms have this property. △ Less

Submitted 7 July, 2019; v1 submitted 23 April, 2018; originally announced April 2018.

Comments: arXiv admin note: substantial text overlap with arXiv:1509.00149

Automated Construction of Bounded-Loss Imperfect-Recall Abstractions in Extensive-Form Games

Authors: Jiri Cermak, Viliam Lisy, Branislav Bosansky

Abstract: Extensive-form games (EFGs) model finite sequential interactions between players. The amount of memory required to represent these games is the main bottleneck of algorithms for computing optimal strategies and the size of these strategies is often impractical for real-world applications. A common approach to tackle the memory bottleneck is to use information abstraction that removes parts of info… ▽ More Extensive-form games (EFGs) model finite sequential interactions between players. The amount of memory required to represent these games is the main bottleneck of algorithms for computing optimal strategies and the size of these strategies is often impractical for real-world applications. A common approach to tackle the memory bottleneck is to use information abstraction that removes parts of information available to players thus reducing the number of decision points in the game. However, existing information-abstraction techniques are either specific for a particular domain, they do not provide any quality guarantees, or they are applicable to very small subclasses of EFGs. We present domain-independent abstraction methods for creating imperfect recall abstractions in extensive-form games that allow computing strategies that are (near) optimal in the original game. To this end, we introduce two novel algorithms, FPIRA and CFR+IRA, based on fictitious play and counterfactual regret minimization. These algorithms can start with an arbitrary domain specific, or the coarsest possible, abstraction of the original game. The algorithms iteratively detect the missing information they require for computing a strategy for the abstract game that is (near) optimal in the original game. This information is then included back into the abstract game. Moreover, our algorithms are able to exploit imperfect-recall abstractions that allow players to forget even history of their own actions. However, the algorithms require traversing the complete unabstracted game tree. We experimentally show that our algorithms can closely approximate Nash equilibrium of large games using abstraction with as little as 0.9% of information sets of the original game. Moreover, the results suggest that memory savings increase with the increasing size of the original games. △ Less

Submitted 15 April, 2020; v1 submitted 14 March, 2018; originally announced March 2018.

Journal ref: Cermak, Jiri, Viliam Lisy, and Branislav Bosansky. "Automated construction of bounded-loss imperfect-recall abstractions in extensive-form games." Artificial Intelligence 282 (2020): 103248

Attenuation of the NMR signal due to hydrodynamic Brownian motion

Authors: Vladimir Lisy, Jana Tothova

Abstract: Nuclear magnetic resonance (NMR) is a widely used nondestructive method to study random motion of spin-bearing particles in different systems. In the long-time limit the theoretical description of the NMR experiments is well developed and allows proper interpretation of measurements of normal and anomalous diffusion. The traditional description becomes, however, insufficient for the shorter-time d… ▽ More Nuclear magnetic resonance (NMR) is a widely used nondestructive method to study random motion of spin-bearing particles in different systems. In the long-time limit the theoretical description of the NMR experiments is well developed and allows proper interpretation of measurements of normal and anomalous diffusion. The traditional description becomes, however, insufficient for the shorter-time dynamics of the particles. In the present paper, the all-time attenuation function of the NMR signal in a magnetic-field gradient due to the Brownian motion (BM) of particles in incompressible liquids is calculated by using the method of accumulation of phases by a precessing magnetic moment, without reference to a concrete model of the stochastic dynamics. The obtained expressions are then used to evaluate the attenuation within the hydrodynamic theory of the BM. It is shown that the well-known time behavior of the formulas corresponding to the Einstein theory of diffusion in the case of steady gradient and Hahn's echo experiments is reached at times much larger than the characteristic time of the loss of memory in the particle dynamics. At shorter times the attenuation function significantly differs from the classical formulas used to interpret these experiments. △ Less

Submitted 5 March, 2018; originally announced March 2018.

Journal ref: Journal of Molecular Liquids 234 (2017) 182-186

NMR signals within the generalized Langevin model for fractional Brownian motion

Authors: Vladimir Lisy, Jana Tothova

Abstract: The methods of Nuclear Magnetic Resonance belong to the best developed and often used tools for studying random motion of particles in different systems, including soft biological tissues. In the long-time limit the current mathematical description of the experiments allows proper interpretation of measurements of normal and anomalous diffusion. The shorter-time dynamics is however correctly consi… ▽ More The methods of Nuclear Magnetic Resonance belong to the best developed and often used tools for studying random motion of particles in different systems, including soft biological tissues. In the long-time limit the current mathematical description of the experiments allows proper interpretation of measurements of normal and anomalous diffusion. The shorter-time dynamics is however correctly considered only in a few works that do not go beyond the standard memoryless Langevin description of the Brownian motion (BM). In the present work, the attenuation function S(t) for an ensemble of spin-bearing particles in a magnetic-field gradient, expressed in a form applicable for any kind of stationary stochastic dynamics of spins with or without a memory, is calculated in the frame of the model of fractional BM. The solution of the model for particles trapped in a harmonic potential is obtained in an exceedingly simple way and used for the calculation of S(t). In the limit of free particles coupled to a fractal heat bath, the results compare favorably with experiments acquired in human neuronal tissues. The effect of the trap is demonstrated by introducing a simple model for the generalized diffusion coefficient of the particle. △ Less

Submitted 5 March, 2018; originally announced March 2018.

Journal ref: Physica A 494 (2018) 200-208

Classification with Costly Features using Deep Reinforcement Learning

Authors: Jaromír Janisch, Tomáš Pevný, Viliam Lisý

Abstract: We study a classification problem where each feature can be acquired for a cost and the goal is to optimize a trade-off between the expected classification error and the feature cost. We revisit a former approach that has framed the problem as a sequential decision-making problem and solved it by Q-learning with a linear approximation, where individual actions are either requests for feature value… ▽ More We study a classification problem where each feature can be acquired for a cost and the goal is to optimize a trade-off between the expected classification error and the feature cost. We revisit a former approach that has framed the problem as a sequential decision-making problem and solved it by Q-learning with a linear approximation, where individual actions are either requests for feature values or terminate the episode by providing a classification decision. On a set of eight problems, we demonstrate that by replacing the linear approximation with neural networks the approach becomes comparable to the state-of-the-art algorithms developed specifically for this problem. The approach is flexible, as it can be improved with any new reinforcement learning enhancement, it allows inclusion of pre-trained high-performance classifier, and unlike prior art, its performance is robust across all evaluated datasets. △ Less

Submitted 12 November, 2018; v1 submitted 20 November, 2017; originally announced November 2017.

Comments: AAAI 2019

Attenuation of the NMR signal in a field gradient due to stochastic dynamics with memory

Authors: V. Lisy, J. Tothova

Abstract: The attenuation function S(t) for an ensemble of spins in a magnetic-field gradient is calculated by accumulation of the phase shifts in the rotating frame resulting from the displacements of spin-bearing particles. The found S(t), expressed through the particle mean square displacement, is applicable for any kind of stationary stochastic motion of spins, including their non-markovian dynamics wit… ▽ More The attenuation function S(t) for an ensemble of spins in a magnetic-field gradient is calculated by accumulation of the phase shifts in the rotating frame resulting from the displacements of spin-bearing particles. The found S(t), expressed through the particle mean square displacement, is applicable for any kind of stationary stochastic motion of spins, including their non-markovian dynamics with memory. The known expressions valid for normal and anomalous diffusion are obtained as special cases in the long time approximation. The method is also applicable to the NMR pulse sequences based on the refocusing principle. This is demonstrated by describing the Hahn spin echo experiment. The attenuation of the NMR signal is also evaluated providing that the random motion of particle is modeled by the generalized Langevin equation with the memory kernel exponentially decaying in time. △ Less

Submitted 10 January, 2017; originally announced January 2017.

Comments: Journal of Magnetic Resonance, accepted

DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

Authors: Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling

Abstract: Artificial intelligence has seen several breakthroughs in recent years, with games often serving as milestones. A common feature of these games is that players have perfect information. Poker is the quintessential game of imperfect information, and a longstanding challenge problem in artificial intelligence. We introduce DeepStack, an algorithm for imperfect information settings. It combines recur… ▽ More Artificial intelligence has seen several breakthroughs in recent years, with games often serving as milestones. A common feature of these games is that players have perfect information. Poker is the quintessential game of imperfect information, and a longstanding challenge problem in artificial intelligence. We introduce DeepStack, an algorithm for imperfect information settings. It combines recursive reasoning to handle information asymmetry, decomposition to focus computation on the relevant decision, and a form of intuition that is automatically learned from self-play using deep learning. In a study involving 44,000 hands of poker, DeepStack defeated with statistical significance professional poker players in heads-up no-limit Texas hold'em. The approach is theoretically sound and is shown to produce more difficult to exploit strategies than prior approaches. △ Less

Submitted 3 March, 2017; v1 submitted 6 January, 2017; originally announced January 2017.

Equilibrium Approximation Quality of Current No-Limit Poker Bots

Authors: Viliam Lisy, Michael Bowling

Abstract: Approximating a Nash equilibrium is currently the best performing approach for creating poker-playing programs. While for the simplest variants of the game, it is possible to evaluate the quality of the approximation by computing the value of the best response strategy, this is currently not computationally feasible for larger variants of the game, such as heads-up no-limit Texas hold'em. In this… ▽ More Approximating a Nash equilibrium is currently the best performing approach for creating poker-playing programs. While for the simplest variants of the game, it is possible to evaluate the quality of the approximation by computing the value of the best response strategy, this is currently not computationally feasible for larger variants of the game, such as heads-up no-limit Texas hold'em. In this paper, we present a simple and computationally inexpensive Local Best Response method for computing an approximate lower bound on the value of the best response strategy. Using this method, we show that existing poker-playing programs, based on solving abstract games, are remarkably poor Nash equilibrium approximations. △ Less

Submitted 8 January, 2017; v1 submitted 22 December, 2016; originally announced December 2016.

Comments: To appear at AAAI-17 Workshop on Computer Poker and Imperfect Information Games

Comment on "Motional Averaging of Nuclear Resonance in a Field Gradient"

Authors: V. Lisy, J. Tothova

Abstract: In the Letter by Nanette N. Jarenwattananon and Louis-S. Bouchard [PRL 114, 197601 (2015)] an NMR experiment on gases in the presence of a magnetic-field gradient has been considered. As distinct from the traditional description of molecular self-diffusion, the authors calculate the decoherence of the signal taking into account the histories of molecular displacements. For this purpose the general… ▽ More In the Letter by Nanette N. Jarenwattananon and Louis-S. Bouchard [PRL 114, 197601 (2015)] an NMR experiment on gases in the presence of a magnetic-field gradient has been considered. As distinct from the traditional description of molecular self-diffusion, the authors calculate the decoherence of the signal taking into account the histories of molecular displacements. For this purpose the generalized Langevin equation (GLE) is applied. We show that the use of this equation is inappropriate. The calculations performed in the Letter are not correct and do not lead to the reported revised expression for line broadening that takes into account the autocorrelation effects in the diffusion process. The surprising temperature behavior of the observed NMR signal is thus not explained. In particular, the linewidth does not follow the power law f ~ T^(-1/2) at high temperatures. We give also Remarks on the Jarenwattananon and Bouchard Reply [Phys. Rev. Lett. 117, 249702 (2016)]. △ Less

Submitted 10 December, 2016; originally announced December 2016.

Journal ref: Phys. Rev. Lett. 117, 249701 (2016)

Analysis of Hannan Consistent Selection for Monte Carlo Tree Search in Simultaneous Move Games

Authors: Vojtěch Kovařík, Viliam Lisý

Abstract: Monte Carlo Tree Search (MCTS) has recently been successfully used to create strategies for playing imperfect-information games. Despite its popularity, there are no theoretic results that guarantee its convergence to a well-defined solution, such as Nash equilibrium, in these games. We partially fill this gap by analysing MCTS in the class of zero-sum extensive-form games with simultaneous moves… ▽ More Monte Carlo Tree Search (MCTS) has recently been successfully used to create strategies for playing imperfect-information games. Despite its popularity, there are no theoretic results that guarantee its convergence to a well-defined solution, such as Nash equilibrium, in these games. We partially fill this gap by analysing MCTS in the class of zero-sum extensive-form games with simultaneous moves but otherwise perfect information. The lack of information about the opponent's concurrent moves already causes that optimal strategies may require randomization. We present theoretic as well as empirical investigation of the speed and quality of convergence of these algorithms to the Nash equilibria. Primarily, we show that after minor technical modifications, MCTS based on any (approximately) Hannan consistent selection function always converges to an (approximate) subgame perfect Nash equilibrium. Without these modifications, Hannan consistency is not sufficient to ensure such convergence and the selection function must satisfy additional properties, which empirically hold for the most common Hannan consistent algorithms. △ Less

Submitted 1 September, 2015; originally announced September 2015.

Convergence of Monte Carlo Tree Search in Simultaneous Move Games

Authors: Viliam Lisý, Vojtěch Kovařík, Marc Lanctot, Branislav Bošanský

Abstract: We study Monte Carlo tree search (MCTS) in zero-sum extensive-form games with perfect information and simultaneous moves. We present a general template of MCTS algorithms for these games, which can be instantiated by various selection methods. We formally prove that if a selection method is $ε$-Hannan consistent in a matrix game and satisfies additional requirements on exploration, then the MCTS a… ▽ More We study Monte Carlo tree search (MCTS) in zero-sum extensive-form games with perfect information and simultaneous moves. We present a general template of MCTS algorithms for these games, which can be instantiated by various selection methods. We formally prove that if a selection method is $ε$-Hannan consistent in a matrix game and satisfies additional requirements on exploration, then the MCTS algorithm eventually converges to an approximate Nash equilibrium (NE) of the extensive-form game. We empirically evaluate this claim using regret matching and Exp3 as the selection methods on randomly generated games and empirically selected worst case games. We confirm the formal result and show that additional MCTS variants also converge to approximate NE on the evaluated games. △ Less

Submitted 5 November, 2013; v1 submitted 31 October, 2013; originally announced October 2013.

Comments: NIPS 2013 paper including appendix

Journal ref: Advances in Neural Information Processing Systems 26, pp 2112-2120, 2013

On the colour of thermal noise in fluids

Authors: Jana Tothova, Lukas Glod, Vladimir Lisy

Abstract: In the paper by Franosch et al., Nature 478, 85 (2011), the positional fluctuations of Brownian microspheres in fluids were studied by confining the particles in an optical trap. Experimental access to short timescales has revealed a resonance peak in the spectrum of these fluctuations, in contrast to the commonly assumed overdamped motion. This work is also interesting as the first measurement of… ▽ More In the paper by Franosch et al., Nature 478, 85 (2011), the positional fluctuations of Brownian microspheres in fluids were studied by confining the particles in an optical trap. Experimental access to short timescales has revealed a resonance peak in the spectrum of these fluctuations, in contrast to the commonly assumed overdamped motion. This work is also interesting as the first measurement of the "colour" of thermal noise driving the Brownian particles through collisions with the fluid molecules. The obtained results are described by the hydrodynamic theory of the Brownian motion in harmonic potentials. In the present work we show that the correlation properties of the thermal noise significantly differ from those determined in the discussed work. △ Less

Submitted 15 April, 2013; originally announced April 2013.

Comments: 5 pages, 2 figs. A comment submitted to and rejected by Nature

On the correlation properties of thermal noise in fluids

Authors: Vladimir Lisy, Jana Tothova, Lukas Glod

Abstract: The properties of the thermal force driving micron particles in incompressible fluids are studied within the hydrodynamic theory of the Brownian motion. It is shown that the assumption used for the hydrodynamic Langevin equation in its usual form, according to which the random force at a time t and the velocity of the particle at the initial time equal to zero are uncorrelated, leads to super-diff… ▽ More The properties of the thermal force driving micron particles in incompressible fluids are studied within the hydrodynamic theory of the Brownian motion. It is shown that the assumption used for the hydrodynamic Langevin equation in its usual form, according to which the random force at a time t and the velocity of the particle at the initial time equal to zero are uncorrelated, leads to super-diffusion of the particle. To obtain the correct Einstein diffusion at long times, the mentioned hypothesis must be abandoned, which however does not contradict causality. The corresponding correlations are explicitly evaluated. We consider also the "color" of thermal noise, recently measured experimentally (Th. Franosch et al., Nature 478, 85 (2011)), and correct the interpretation of these experiments. The time correlation functions for the thermal random force are obtained using the exact solution of the Langevin equation, and on the basis of the theorem that in the linear response theory connects the mobility of the particle and its velocity autocorrelation function. △ Less

Submitted 13 October, 2012; originally announced October 2012.

Comments: Int. J. Thermophys. 2012, in press

Brownian oscillators driven by correlated noise in a moving trap

Authors: Lukas Glod, Gabriela Vasziova, Jana Tothova, Vladimir Lisy

Abstract: Brownian oscillator, i.e. a micron-sized or smaller particle trapped in a thermally fluctuating environment is studied. The confining harmonic potential can move with a constant velocity. As distinct from the standard Langevin theory, the chaotic force driving the particle is correlated in time. The dynamics of the particle is described by the generalized Langevin equation with the inertial term,… ▽ More Brownian oscillator, i.e. a micron-sized or smaller particle trapped in a thermally fluctuating environment is studied. The confining harmonic potential can move with a constant velocity. As distinct from the standard Langevin theory, the chaotic force driving the particle is correlated in time. The dynamics of the particle is described by the generalized Langevin equation with the inertial term, a coloured noise force, and a memory integral. We consider two kinds of the memory in the system. The first one corresponds to the exponentially correlated noise in a weakly viscoelastic fluid and in the second case the memory naturally arises within the Navier-Stokes hydrodynamics. Exact analytical solutions are obtained in both the cases using a simple and effective method not applied so far in this kind of problems. △ Less

Submitted 20 February, 2012; originally announced February 2012.

Comments: Presented at the 17th International Conference Applied Physics of Condensed Matter (APCOM 2011), Novy Smokovec, High Tatras, Slovakia, June 22 - 24, 2011

Journal ref: Journal of Electrical Engineering 63 (2012) 53-58

On interpretation of force measurements in fluids: regular and thermal forces

Authors: Jana Tothova, Lukas Glod, Gabriela Vasziova, Vladimir Lisy

Abstract: The paper is devoted to the problem of the determination of regular and thermal forces acting on microscopic and smaller objects in fluids. One of the methods how regular forces are determined is the measurement of the drift velocity of Brownian particles. We have obtained an exact expression for this velocity within the hydrodynamic theory of the Brownian motion. It is shown that the influence of… ▽ More The paper is devoted to the problem of the determination of regular and thermal forces acting on microscopic and smaller objects in fluids. One of the methods how regular forces are determined is the measurement of the drift velocity of Brownian particles. We have obtained an exact expression for this velocity within the hydrodynamic theory of the Brownian motion. It is shown that the influence of the inertial and memory effects can be significant in the force determination when the experimental times are sufficiently short. In the second part of the work, within the same theory, we study the properties of the thermal force driving the particles in incompressible fluids. We show that the usual assumption for the Kubo's generalized Langevin equation (called the "fundamental hypothesis") that the thermal force at a time t and the velocity of the particle in preceding times are uncorrelated, leads to an unexpected super-diffusion of the particle. To obtain the Einstein diffusion at long times, the mentioned hypothesis must be abandoned, which however does not contradict to causality. Finally, we consider the "color" of thermal noise, recently measured experimentally [Th. Franosch et al., Nature 478, 85 (2011)], and correct the interpretation of these experiments. △ Less

Submitted 20 February, 2012; originally announced February 2012.

Comments: Presented at the 13th International Workshop on Theoretical Physics: Small Triangle Meeting, Stara Lesna, High Tatras, Slovakia, Nov. 13 - 16, 2011

Brownian motion of charged particles driven by correlated noise

Authors: V. Lisy, J. Tothova

Abstract: Stochastic motion of charged particles in the magnetic field was first studied almost half a century ago in the classical works by Taylor and Kursunoglu in connection with the diffusion of electrons and ions in plasma. In their works the long-time limits of the mean square displacement (MSD) of the particles have been found. Later Furuse on the basis of standard Langevin theory generalized their r… ▽ More Stochastic motion of charged particles in the magnetic field was first studied almost half a century ago in the classical works by Taylor and Kursunoglu in connection with the diffusion of electrons and ions in plasma. In their works the long-time limits of the mean square displacement (MSD) of the particles have been found. Later Furuse on the basis of standard Langevin theory generalized their results for arbitrary times. The currently observed revival of these problems is mainly related to memory effects in the diffusion of particles, which appear when colored random forces act on the particles from their surroundings. In the present work an exact analytical solution of the generalized Langevin equation has been found for the motion of the particle in an external magnetic field when the random force is exponentially correlated in the time. The obtained MSD of the particle motion across the field contains a term proportional to the time, a constant term, and contributions exponentially decaying in the time. The results are more general than the previous results from the literature and are obtained in a considerably simpler way applicable to many other problems of the Brownian motion with memory. △ Less

Submitted 10 July, 2011; originally announced July 2011.

Comments: 14 pages, 5 figures

An old efficient approach to anomalous Brownian motion

Authors: V. Lisy, J. Tothova

Abstract: A number of random processes in various fields of science is described by phenomenological equations containing a stochastic force, the best known example being the Langevin equation (LE) for the Brownian motion (BM) of particles. Long ago Vladimirsky (1942) proposed a simple method for solving such equations. The method, based on the classical Gibbs statistics, consists in converting the stochast… ▽ More A number of random processes in various fields of science is described by phenomenological equations containing a stochastic force, the best known example being the Langevin equation (LE) for the Brownian motion (BM) of particles. Long ago Vladimirsky (1942) proposed a simple method for solving such equations. The method, based on the classical Gibbs statistics, consists in converting the stochastic LE into a deterministic one, and is applicable to linear equations with any kind of memory. When the memory effects are taken into account in the description of the BM, the mean square displacement of the particle at long times can exhibit an "anomalous" (different from that in the Einstein theory) time dependence. In the present paper we show how some general properties of such anomalous BM can be easily derived using the Vladimirsky approach. The method can be effectively used in solving many of the problems currently considered in the literature. We apply it to the description of the BM when the memory kernel in the Volterra-type integro-differential LE exponentially decreases with the time. The problem of the hydrodynamic BM of a charged particle in an external magnetic field is also solved. △ Less

Submitted 5 June, 2010; originally announced June 2010.

Comments: 8 pages, no figures

Rheology of dilute polymer solutions with time-dependent screening of hydrodynamic interactions

Authors: V. Lisy, J. Tothova, B. Brutovsky

Abstract: The screening of hydrodynamic interactions (HI) essentially affects macroscopic properties of polymeric solutions. This screening depends not only on the polymer concentration but has a dynamic nature. In the present work, a bead-spring theory is developed, in which this phenomenon is described for solutions of nonentangled polymer coils. The equation of motion for the beads of a test polymer is… ▽ More The screening of hydrodynamic interactions (HI) essentially affects macroscopic properties of polymeric solutions. This screening depends not only on the polymer concentration but has a dynamic nature. In the present work, a bead-spring theory is developed, in which this phenomenon is described for solutions of nonentangled polymer coils. The equation of motion for the beads of a test polymer is solved together with the Brinkman's equation for the solvent velocity that takes into account the presence of other coils in solution. The time correlation functions for the polymer normal modes are found. A tendency to the screening of HI is demonstrated on the coil diffusion as well as on the relaxation of its internal modes. With the growing concentration of the coils they both show a transition to the exact Rouse behavior. The viscosity of the solution and some other observable quantities are calculated. As the time increaes, the time-dependent quantities change their behavior from the Rouse regime through the Zimm one again to the Rouse dynamics at long times. △ Less

Submitted 2 November, 2007; v1 submitted 31 October, 2007; originally announced October 2007.

Comments: A modified version of this paper will appear in Intern. Rev. Phys

Effects of hydrodynamic noise on the diffusion of polymers in dilute solutions

Authors: V. Lisy, J. Tothova, A. V. Zatovsky

Abstract: The Rouse-Zimm equation for the position vectors of beads map** the polymer is generalized by taking into account the viscous aftereffect and the hydrodynamic noise. For the noise, the random fluctuations of the hydrodynamic tensor of stresses are responsible. The preaveraging of the Oseen tensor for the nonstationary Navier-Stokes equation allowed us to relate the time correlation functions o… ▽ More The Rouse-Zimm equation for the position vectors of beads map** the polymer is generalized by taking into account the viscous aftereffect and the hydrodynamic noise. For the noise, the random fluctuations of the hydrodynamic tensor of stresses are responsible. The preaveraging of the Oseen tensor for the nonstationary Navier-Stokes equation allowed us to relate the time correlation functions of the Fourier components of the bead position to the correlation functions of the hydrodynamic field created by the noise. The velocity autocorrelation function of the center of inertia of the polymer coil is considered in detail for both the short and long times when it behaves according to the t^(-3/2) law and does not depend on any polymer parameters. The diffusion coefficient of the polymer is close to that from the Zimm theory, with corrections depending on the ratio between the size of the bead and the size of the whole coil. △ Less

Submitted 26 September, 2007; originally announced September 2007.

Comments: 10 pages, no figures

Simple derivation of the first cumulant for the Rouse chain

Authors: V. Lisy, B. Brutovsky, J. Tothova

Abstract: A simple analytic expression for the first cumulant of the dynamic structure factor of a polymer coil in the Rouse model is derived. The obtained formula is exact within the usual assumption of the continuum distribution of beads along the chain. It reflects the contributions to the scattering of light or neutrons from both the internal motion of the polymer and its diffusion, and is valid in th… ▽ More A simple analytic expression for the first cumulant of the dynamic structure factor of a polymer coil in the Rouse model is derived. The obtained formula is exact within the usual assumption of the continuum distribution of beads along the chain. It reflects the contributions to the scattering of light or neutrons from both the internal motion of the polymer and its diffusion, and is valid in the whole region of the wave-vector change at the scattering. △ Less

Submitted 26 September, 2007; originally announced September 2007.

Comments: 4 pages, 1 figure

Inverse Geometric Approach to the Simulation of the Circular Growth. The Case of Multicellular Tumor Spheroids

Authors: Branislav Brutovsky, Denis Horvath, Vladimir Lisy

Abstract: We demonstrate the power of the genetic algorithms to construct the cellular automata model simulating the growth of 2-dimensional close-to-circular clusters revealing the desired properties, such as the growth rate and, at the same time, the fractal behavior of their contours. The possible application of the approach in the field of tumor modeling is outlined. We demonstrate the power of the genetic algorithms to construct the cellular automata model simulating the growth of 2-dimensional close-to-circular clusters revealing the desired properties, such as the growth rate and, at the same time, the fractal behavior of their contours. The possible application of the approach in the field of tumor modeling is outlined. △ Less

Submitted 30 July, 2007; v1 submitted 24 April, 2007; originally announced April 2007.

Addendum to "Monomer motion in single- and double-stranded DNA coils" [arXiv: cond-mat/0509399]

Authors: J. Tothova, B. Brutovsky, V. Lisy

Abstract: In our work [J. Tothova et al., cond-mat/0509399] the first observation of the kinetics of individual polymer monomers using the fluorescence correlation technique [R. Shusterman et al., Phys. Rev. Lett. 92, 048303 (2004)] has been interpreted within the joint Rouse-Zimm theory. Optimizing the theory to the experimental data the phenomenological parameters for the statistical-mechanical descript… ▽ More In our work [J. Tothova et al., cond-mat/0509399] the first observation of the kinetics of individual polymer monomers using the fluorescence correlation technique [R. Shusterman et al., Phys. Rev. Lett. 92, 048303 (2004)] has been interpreted within the joint Rouse-Zimm theory. Optimizing the theory to the experimental data the phenomenological parameters for the statistical-mechanical description of the universal behavior of double- and single-stranded DNA and the dominant types of their dynamics have been determined. Recently, these data have been corrected [R. Shusterman et al., Phys. Rev. Lett. 98, 029901 (2007)]. In this Addendum the fits of the theory to the new data are presented. The main conclusions of our preceding work remain unchanged. Moreover, the new data allow a significantly better agreement with the theory than the previous ones. △ Less

Submitted 22 January, 2007; originally announced January 2007.

Comments: 3 pages, 2 figures

Simulation of Multicellular Tumor Spheroids Growth Dynamics

Authors: Branislav Brutovsky, Denis Horvath, Vladimir Lisy

Abstract: The inverse geometric approach to the modeling of the growth of circular objects revealing required features, such as the velocity of the growth and fractal behavior of their contours, is presented. It enables to reproduce some of the recent findings in morphometry of tumors with the possible implications for cancer research. The technique is based on cellular automata paradigm with the transiti… ▽ More The inverse geometric approach to the modeling of the growth of circular objects revealing required features, such as the velocity of the growth and fractal behavior of their contours, is presented. It enables to reproduce some of the recent findings in morphometry of tumors with the possible implications for cancer research. The technique is based on cellular automata paradigm with the transition rules selected by optimization procedure performed by the genetic algorithms. △ Less

Submitted 18 October, 2006; originally announced October 2006.

Comments: 6 pages, 7 figures

Comment on "Monomer Dynamics in Double- and Single-Stranded DNA Polymers"

Authors: J. Tothova, B. Brutovsky, V. Lisy

Abstract: It is discussed that the kinetics observed by Shusterman et al. [Phys. Rev. Lett. 92, 048303] for long dsDNA is not the Rouse one and, in fact, the macromolecule behaves (approximately) as the Zimm polymer. It is discussed that the kinetics observed by Shusterman et al. [Phys. Rev. Lett. 92, 048303] for long dsDNA is not the Rouse one and, in fact, the macromolecule behaves (approximately) as the Zimm polymer. △ Less

Submitted 15 September, 2005; originally announced September 2005.

Comments: 1 page

The dynamics of polymers in solution with hydrodynamic memory

Authors: V. Lisy, J. Tothova, B. Brutovsky, A. V Zatovsky

Abstract: The theory of the dynamics of polymers in solution is developed coming from the hydrodynamic theory of the Brownian motion (BM) and the Rouse-Zimm (RZ) model. It is shown that the time correlation functions describing the polymer motion essentially differ from those in the previous RZ models based on the Einstein theory of BM. The MSD of the polymer coil is at short times proportional to t^2 (in… ▽ More The theory of the dynamics of polymers in solution is developed coming from the hydrodynamic theory of the Brownian motion (BM) and the Rouse-Zimm (RZ) model. It is shown that the time correlation functions describing the polymer motion essentially differ from those in the previous RZ models based on the Einstein theory of BM. The MSD of the polymer coil is at short times proportional to t^2 (instead of t). At long times it contains additional (to the Einstein term) contributions, the leading of which is ~ t^{1/2}. The relaxation of the internal normal modes of the polymer differs from the traditional exponential decay. This is displayed in the tails of their correlation functions, the longest-lived being ~ t^{-3/2} in the Rouse limit and t^{-5/2} in the Zimm case when the hydrodynamic interaction is strong. It is discussed that the found peculiarities, in particular a slower diffusion of the coil, should be observable in dynamic scattering experiments. The dynamic structure factor and the first cumulant of the polymer coil are calculated. The theory is extended to the situation when the dynamics of the studied polymer is influenced by the presence of other polymers in dilute solution. △ Less

Submitted 15 September, 2005; originally announced September 2005.

Comments: 40 pages. Submitted to Soft Condensed Matter: New Research (NOVA SCIENCE, New York)

Monomer motion in single- and double-stranded DNA coils

Authors: J. Tothova, B. Brutovsky, V. Lisy

Abstract: The dynamics of flexible polymers in dilute solution is usually described in terms of the pure Rouse or Zimm bead-spring models assuming continuous distribution of the internal relaxation modes. We show that this approach may lead to misleading interpretation of experimental data. The more correct description should come from the joint Rouse-Zimm (RZ) theory that contains the Rouse and Zimm mode… ▽ More The dynamics of flexible polymers in dilute solution is usually described in terms of the pure Rouse or Zimm bead-spring models assuming continuous distribution of the internal relaxation modes. We show that this approach may lead to misleading interpretation of experimental data. The more correct description should come from the joint Rouse-Zimm (RZ) theory that contains the Rouse and Zimm models as limiting cases. The internal modes are discrete with respect to the mode number, and the type of the bead motion changes in the time from the Rouse to Zimm behavior. We demonstrate this interpreting the recent first observation of the kinetics of individual polymer monomers using the fluorescence correlation technique [R. Shusterman et al., Phys. Rev. Lett. 92, 048303 (2004)]. Optimizing the RZ theory to the data on double- and single-stranded DNA coils (dsDNA and ssDNA) the parameters for the statistical-mechanical description of the behavior of these polymers have been determined. The calculations indicate that dsDNA follows mainly the classical Zimm-type kinetics rather than the Rouse one as it was originally proposed. Single-stranded DNA also behaves predominantly as the Zimm polymer. For dsDNA the Kuhn length agrees with the commonly accepted value in the literature while in the case of ssDNA it takes a value much larger than it is usually cited in the literature. △ Less

Submitted 15 September, 2005; originally announced September 2005.

Comments: 13 pages

The Rouse-Zimm-Brinkman theory of the dynamics of polymers in dilute solutions

Authors: V. Lisy, J. Tothova, A. V. Zatovsky

Abstract: We propose a theory of the dynamics of polymers in dilute solution, in which the popular Zimm and Rouse models are limiting cases of infinitely large and small draining parameter. The equation of motion for the polymer segments beads) is solved together with Brinkman's equation for the solvent velocity that takes into account the presence of other polymer coils in the solution. The equation for… ▽ More We propose a theory of the dynamics of polymers in dilute solution, in which the popular Zimm and Rouse models are limiting cases of infinitely large and small draining parameter. The equation of motion for the polymer segments beads) is solved together with Brinkman's equation for the solvent velocity that takes into account the presence of other polymer coils in the solution. The equation for the polymer normal modes is obtained and the relevant time correlation functions are found. A tendency to the time-dependent hydrodynamic screening is demonstrated on the diffusion of the polymers as well as on the relaxation of their internal modes. With the growing concentration of the coils in solution they both show a transition to the (exactly) Rouse behavior. The shear viscosity of the solution, the Huggins coefficient and other quantities are calculated and shown to be notably different from the known results. △ Less

Submitted 15 September, 2005; originally announced September 2005.

Comments: The work presented at the Annual Conference in Ukraine "Statistical Physics 2005: Modern Problems and New Applications", August 28-30, 2005, Lviv, Ukraine