-
Deep Back-Filling: a Split Window Technique for Deep Online Cluster Job Scheduling
Authors:
Lingfei Wang,
Aaron Harwood,
Maria A. Rodriguez
Abstract:
Job scheduling is a critical component of workload management systems that can significantly influence system performance, e.g., in HPC clusters. The scheduling objectives are often mixed, such as maximizing resource utilization and minimizing job waiting time. An increasing number of researchers are moving from heuristic-based approaches to Deep Reinforcement Learning approaches in order to optim…
▽ More
Job scheduling is a critical component of workload management systems that can significantly influence system performance, e.g., in HPC clusters. The scheduling objectives are often mixed, such as maximizing resource utilization and minimizing job waiting time. An increasing number of researchers are moving from heuristic-based approaches to Deep Reinforcement Learning approaches in order to optimize scheduling objectives. However, the job scheduler's state space is partially observable to a DRL-based agent because the job queue is practically unbounded. The agent's observation of the state space is constant in size since the input size of the neural networks is predefined. All existing solutions to this problem intuitively allow the agent to observe a fixed window size of jobs at the head of the job queue. In our research, we have seen that such an approach can lead to "window staleness" where the window becomes full of jobs that can not be scheduled until the cluster has completed sufficient work. In this paper, we propose a novel general technique that we call \emph{split window}, which allows the agent to observe both the head \emph{and tail} of the queue. With this technique, the agent can observe all arriving jobs at least once, which completely eliminates the window staleness problem. By leveraging the split window, the agent can significantly reduce the average job waiting time and average queue length, alternatively allowing the use of much smaller windows and, therefore, faster training times. We show a range of simulation results using HPC job scheduling trace data that supports the effectiveness of our technique.
△ Less
Submitted 18 January, 2024;
originally announced January 2024.
-
An Energy-aware and Fault-tolerant Deep Reinforcement Learning based approach for Multi-agent Patrolling Problems
Authors:
Chenhao Tong,
Aaron Harwood,
Maria A. Rodriguez,
Richard O. Sinnott
Abstract:
Autonomous vehicles are suited for continuous area patrolling problems. However, finding an optimal patrolling strategy can be challenging for many reasons. Firstly, patrolling environments are often complex and can include unknown environmental factors, such as wind or landscape. Secondly, autonomous vehicles can have failures or hardware constraints, such as limited battery life. Importantly, pa…
▽ More
Autonomous vehicles are suited for continuous area patrolling problems. However, finding an optimal patrolling strategy can be challenging for many reasons. Firstly, patrolling environments are often complex and can include unknown environmental factors, such as wind or landscape. Secondly, autonomous vehicles can have failures or hardware constraints, such as limited battery life. Importantly, patrolling large areas often requires multiple agents that need to collectively coordinate their actions. In this work, we consider these limitations and propose an approach based on model-free, deep multi-agent reinforcement learning. In this approach, the agents are trained to patrol an environment with various unknown dynamics and factors. They can automatically recharge themselves to support continuous collective patrolling. A distributed homogeneous multi-agent architecture is proposed, where all patrolling agents execute identical policies locally based on their local observations and shared location information. This architecture provides a patrolling system that can tolerate agent failures and allow supplementary agents to be added to replace failed agents or to increase the overall patrol performance. The solution is validated through simulation experiments from multiple perspectives, including the overall patrol performance, the efficiency of battery recharging strategies, the overall fault tolerance, and the ability to cooperate with supplementary agents.
△ Less
Submitted 8 June, 2023; v1 submitted 15 December, 2022;
originally announced December 2022.
-
Where did you tweet from? Inferring the origin locations of tweets based on contextual information
Authors:
Rabindra Lamsal,
Aaron Harwood,
Maria Rodriguez Read
Abstract:
Public conversations on Twitter comprise many pertinent topics including disasters, protests, politics, propaganda, sports, climate change, epidemics/pandemic outbreaks, etc., that can have both regional and global aspects. Spatial discourse analysis rely on geographical data. However, today less than 1% of tweets are geotagged; in both cases--point location or bounding place information. A major…
▽ More
Public conversations on Twitter comprise many pertinent topics including disasters, protests, politics, propaganda, sports, climate change, epidemics/pandemic outbreaks, etc., that can have both regional and global aspects. Spatial discourse analysis rely on geographical data. However, today less than 1% of tweets are geotagged; in both cases--point location or bounding place information. A major issue with tweets is that Twitter users can be at location A and exchange conversations specific to location B, which we call the Location A/B problem. The problem is considered solved if location entities can be classified as either origin locations (Location As) or non-origin locations (Location Bs). In this work, we propose a simple yet effective framework--the True Origin Model--to address the problem that uses machine-level natural language understanding to identify tweets that conceivably contain their origin location information. The model achieves promising accuracy at country (80%), state (67%), city (58%), county (56%) and district (64%) levels with support from a Location Extraction Model as basic as the CoNLL-2003-based RoBERTa. We employ a tweet contexualizer (locBERT) which is one of the core components of the proposed model, to investigate multiple tweets' distributions for understanding Twitter users' tweeting behavior in terms of mentioning origin and non-origin locations. We also highlight a major concern with the currently regarded gold standard test set (ground truth) methodology, introduce a new data set, and identify further research avenues for advancing the area.
△ Less
Submitted 17 November, 2022;
originally announced November 2022.
-
Socially Enhanced Situation Awareness from Microblogs using Artificial Intelligence: A Survey
Authors:
Rabindra Lamsal,
Aaron Harwood,
Maria Rodriguez Read
Abstract:
The rise of social media platforms provides an unbounded, infinitely rich source of aggregate knowledge of the world around us, both historic and real-time, from a human perspective. The greatest challenge we face is how to process and understand this raw and unstructured data, go beyond individual observations and see the "big picture"--the domain of Situation Awareness. We provide an extensive s…
▽ More
The rise of social media platforms provides an unbounded, infinitely rich source of aggregate knowledge of the world around us, both historic and real-time, from a human perspective. The greatest challenge we face is how to process and understand this raw and unstructured data, go beyond individual observations and see the "big picture"--the domain of Situation Awareness. We provide an extensive survey of Artificial Intelligence research, focusing on microblog social media data with applications to Situation Awareness, that gives the seminal work and state-of-the-art approaches across six thematic areas: Crime, Disasters, Finance, Physical Environment, Politics, and Health and Population. We provide a novel, unified methodological perspective, identify key results and challenges, and present ongoing research directions.
△ Less
Submitted 13 September, 2022;
originally announced September 2022.
-
Twitter conversations predict the daily confirmed COVID-19 cases
Authors:
Rabindra Lamsal,
Aaron Harwood,
Maria Rodriguez Read
Abstract:
As of writing this paper, COVID-19 (Coronavirus disease 2019) has spread to more than 220 countries and territories. Following the outbreak, the pandemic's seriousness has made people more active on social media, especially on the microblogging platforms such as Twitter and Weibo. The pandemic-specific discourse has remained on-trend on these platforms for months now. Previous studies have confirm…
▽ More
As of writing this paper, COVID-19 (Coronavirus disease 2019) has spread to more than 220 countries and territories. Following the outbreak, the pandemic's seriousness has made people more active on social media, especially on the microblogging platforms such as Twitter and Weibo. The pandemic-specific discourse has remained on-trend on these platforms for months now. Previous studies have confirmed the contributions of such socially generated conversations towards situational awareness of crisis events. The early forecasts of cases are essential to authorities to estimate the requirements of resources needed to cope with the outgrowths of the virus. Therefore, this study attempts to incorporate the public discourse in the design of forecasting models particularly targeted for the steep-hill region of an ongoing wave. We propose a sentiment-involved topic-based latent variables search methodology for designing forecasting models from publicly available Twitter conversations. As a use case, we implement the proposed methodology on Australian COVID-19 daily cases and Twitter conversations generated within the country. Experimental results: (i) show the presence of latent social media variables that Granger-cause the daily COVID-19 confirmed cases, and (ii) confirm that those variables offer additional prediction capability to forecasting models. Further, the results show that the inclusion of social media variables introduces 48.83--51.38% improvements on RMSE over the baseline models. We also release the large-scale COVID-19 specific geotagged global tweets dataset, MegaGeoCOV, to the public anticipating that the geotagged data of this scale would aid in understanding the conversational dynamics of the pandemic through other spatial and temporal contexts.
△ Less
Submitted 13 September, 2022; v1 submitted 21 June, 2022;
originally announced June 2022.
-
Unified Collision Model of Coherent and Measurement-based Quantum Feedback
Authors:
Alfred Harwood,
Matteo Brunelli,
Alessio Serafini
Abstract:
We introduce a general framework, based on collision models and discrete CP-maps, to describe on an equal footing coherent and measurement-based feedback control of quantum mechanical systems. We apply our framework to prominent tasks in quantum control, ranging from cooling to Hamiltonian control. Unlike other proposed comparisons, where coherent feedback always proves superior, we find that eith…
▽ More
We introduce a general framework, based on collision models and discrete CP-maps, to describe on an equal footing coherent and measurement-based feedback control of quantum mechanical systems. We apply our framework to prominent tasks in quantum control, ranging from cooling to Hamiltonian control. Unlike other proposed comparisons, where coherent feedback always proves superior, we find that either measurements or coherent manipulations of the controller can be advantageous depending on the task at hand. Measurement-based feedback is typically superior in cooling, whilst coherent feedback is better at assisting quantum operations. Furthermore, we show that both coherent and measurement-based feedback loops allow one to simulate arbitrary Hamiltonian evolutions, and discuss their respective effectiveness in this regard.
△ Less
Submitted 1 April, 2022;
originally announced April 2022.
-
Measuring the stability of fundamental constants with a network of clocks
Authors:
G. Barontini,
L. Blackburn,
V. Boyer,
F. Butuc-Mayer,
X. Calmet,
J. R. Crespo Lopez-Urrutia,
E. A. Curtis,
B. Darquie,
J. Dunningham,
N. J. Fitch,
E. M. Forgan,
K. Georgiou,
P. Gill,
R. M. Godun,
J. Goldwin,
V. Guarrera,
A. C. Harwood,
I. R. Hill,
R. J. Hendricks,
M. Jeong,
M. Y. H. Johnson,
M. Keller,
L. P. Kozhiparambil Sajith,
F. Kuipers,
H. S. Margolis
, et al. (19 additional authors not shown)
Abstract:
The detection of variations of fundamental constants of the Standard Model would provide us with compelling evidence of new physics, and could lift the veil on the nature of dark matter and dark energy. In this work, we discuss how a network of atomic and molecular clocks can be used to look for such variations with unprecedented sensitivity over a wide range of time scales. This is precisely the…
▽ More
The detection of variations of fundamental constants of the Standard Model would provide us with compelling evidence of new physics, and could lift the veil on the nature of dark matter and dark energy. In this work, we discuss how a network of atomic and molecular clocks can be used to look for such variations with unprecedented sensitivity over a wide range of time scales. This is precisely the goal of the recently launched QSNET project: A network of clocks for measuring the stability of fundamental constants. QSNET will include state-of-the-art atomic clocks, but will also develop next-generation molecular and highly charged ion clocks with enhanced sensitivity to variations of fundamental constants. We describe the technological and scientific aims of QSNET and evaluate its expected performance. We show that in the range of parameters probed by QSNET, either we will discover new physics, or we will impose new constraints on violations of fundamental symmetries and a range of theories beyond the Standard Model, including dark matter and dark energy models.
△ Less
Submitted 11 May, 2022; v1 submitted 20 December, 2021;
originally announced December 2021.
-
Real-time Spatio-temporal Event Detection on Geotagged Social Media
Authors:
Yasmeen George,
Shanika Karunasekera,
Aaron Harwood,
Kwan Hui Lim
Abstract:
A key challenge in mining social media data streams is to identify events which are actively discussed by a group of people in a specific local or global area. Such events are useful for early warning for accident, protest, election or breaking news. However, neither the list of events nor the resolution of both event time and space is fixed or known beforehand. In this work, we propose an online…
▽ More
A key challenge in mining social media data streams is to identify events which are actively discussed by a group of people in a specific local or global area. Such events are useful for early warning for accident, protest, election or breaking news. However, neither the list of events nor the resolution of both event time and space is fixed or known beforehand. In this work, we propose an online spatio-temporal event detection system using social media that is able to detect events at different time and space resolutions. First, to address the challenge related to the unknown spatial resolution of events, a quad-tree method is exploited in order to split the geographical space into multiscale regions based on the density of social media data. Then, a statistical unsupervised approach is performed that involves Poisson distribution and a smoothing method for highlighting regions with unexpected density of social posts. Further, event duration is precisely estimated by merging events happening in the same region at consecutive time intervals. A post processing stage is introduced to filter out events that are spam, fake or wrong. Finally, we incorporate simple semantics by using social media entities to assess the integrity, and accuracy of detected events. The proposed method is evaluated using different social media datasets: Twitter and Flickr for different cities: Melbourne, London, Paris and New York. To verify the effectiveness of the proposed method, we compare our results with two baseline algorithms based on fixed split of geographical space and clustering method. For performance evaluation, we manually compute recall and precision. We also propose a new quality measure named strength index, which automatically measures how accurate the reported event is.
△ Less
Submitted 23 June, 2021;
originally announced June 2021.
-
User Identification across Social Networking Sites using User Profiles and Posting Patterns
Authors:
Prashant Solanki,
Kwan Hui Lim,
Aaron Harwood
Abstract:
With the prevalence of online social networking sites (OSNs) and mobile devices, people are increasingly reliant on a variety of OSNs for kee** in touch with family and friends, and using it as a source of information. For example, a user might utilise multiple OSNs for different purposes, such as using Flickr to share holiday pictures with family and friends, and Twitter to post short messages…
▽ More
With the prevalence of online social networking sites (OSNs) and mobile devices, people are increasingly reliant on a variety of OSNs for kee** in touch with family and friends, and using it as a source of information. For example, a user might utilise multiple OSNs for different purposes, such as using Flickr to share holiday pictures with family and friends, and Twitter to post short messages about their thoughts. Identifying the same user across multiple OSNs is an important task as this allows us to understand the usage patterns of users among different OSNs, make recommendations when a user registers for a new OSN, and various other useful applications. To address this problem, we proposed an algorithm based on the multilayer perceptron using various types of features, namely: (i) user profile, such as name, location, description; (ii) temporal distribution of user generated content; and (iii) embedding based on user name, real name and description. Using a Twitter and Flickr dataset of users and their posting activities, we perform an empirical study on how these features affect the performance of user identification across the two OSNs and discuss our main findings based on the different features.
△ Less
Submitted 22 June, 2021;
originally announced June 2021.
-
Cavity optomechanics assisted by optical coherent feedback
Authors:
Alfred Harwood,
Matteo Brunelli,
Alessio Serafini
Abstract:
We consider a wide family of optical coherent feedback loops acting on an optomechanical system operating in the linearized regime. We assess the efficacy of such loops in improving key operations, such as cooling, steady-state squeezing and entanglement, as well as optical to mechanical state transfer. We find that mechanical sideband cooling can be enhanced through passive, interferometric coher…
▽ More
We consider a wide family of optical coherent feedback loops acting on an optomechanical system operating in the linearized regime. We assess the efficacy of such loops in improving key operations, such as cooling, steady-state squeezing and entanglement, as well as optical to mechanical state transfer. We find that mechanical sideband cooling can be enhanced through passive, interferometric coherent feedback, achieving lower steady-state occupancies and considerably speeding up the cooling process; we also quantify the detrimental effect of non-zero delay times on the cooling performance. Steady state entanglement generation in the blue sideband can also be assisted by passive interferometric feedback, which allows one to stabilise otherwise unstable systems, though active feedback (including squeezing elements) does not help to this aim. We show that active feedback loops only allow for the generation of optical, but not mechanical squeezing. Finally, we prove that passive feedback can assist state transfer at transient times for red-sideband driven systems in the strong coupling regime.
△ Less
Submitted 29 January, 2021; v1 submitted 23 November, 2020;
originally announced November 2020.
-
Mining Influentials and their Bot Activities on Twitter Campaigns
Authors:
Shanika Karunasekera,
Kwan Hui Lim,
Aaron Harwood
Abstract:
Twitter is increasingly used for political, advertising and marketing campaigns, where the main aim is to influence users to support specific causes, individuals or groups. We propose a novel methodology for mining and analyzing Twitter campaigns, which includes: (i) collecting tweets and detecting topics relating to a campaign; (ii) mining important campaign topics using scientometrics measures;…
▽ More
Twitter is increasingly used for political, advertising and marketing campaigns, where the main aim is to influence users to support specific causes, individuals or groups. We propose a novel methodology for mining and analyzing Twitter campaigns, which includes: (i) collecting tweets and detecting topics relating to a campaign; (ii) mining important campaign topics using scientometrics measures; (iii) modelling user interests using hashtags and topical entropy; (iv) identifying influential users using an adapted PageRank score; and (v) various metrics and visualization techniques for identifying bot-like activities. While this methodology is generalizable to multiple campaign types, we demonstrate its effectiveness on the 2017 German federal election.
△ Less
Submitted 22 November, 2020;
originally announced November 2020.
-
Optimal Multi-Level Interval-based Checkpointing for Exascale Stream Processing Systems
Authors:
Sachini Jayasekara,
Aaron Harwood,
Shanika Karunasekera
Abstract:
State-of-the-art stream processing platforms make use of checkpointing to support fault tolerance, where a "checkpoint tuple" flows through the topology to all operators, indicating a checkpoint and triggering a checkpoint operation. The checkpoint will enable recovering from any kind of failure, be it as localized as a process fault or as wide spread as power supply loss to an entire rack of mach…
▽ More
State-of-the-art stream processing platforms make use of checkpointing to support fault tolerance, where a "checkpoint tuple" flows through the topology to all operators, indicating a checkpoint and triggering a checkpoint operation. The checkpoint will enable recovering from any kind of failure, be it as localized as a process fault or as wide spread as power supply loss to an entire rack of machines. As we move towards Exascale computing, it is becoming clear that this kind of "single-level" checkpointing is too inefficient to scale. Some HPC researchers are now investigating multi-level checkpointing, where checkpoint operations at each level are tailored to specific kinds of failure to address the inefficiencies of single-level checkpointing. Multi-level checkpointing has been shown in practice to be superior, giving greater efficiency in operation over single-level checkpointing. However, to date there is no theoretical basis that provides optimal parameter settings for an interval-based coordinated multi-level checkpointing approach. This paper presents a theoretical framework for determining optimal parameter settings in an interval-based multi-level periodic checkpointing system, that is applicable to stream processing. Our approach is stochastic, where at a given checkpoint interval, a level is selected with some probability for checkpointing. We derive the optimal checkpoint interval and associated optimal checkpoint probabilities for a multi-level checkpointing system, that considers failure rates, checkpoint costs, restart costs and possible failure during restarting, at every level. We confirm our results with stochastic simulation and practical experimentation.
△ Less
Submitted 15 December, 2019;
originally announced December 2019.
-
A Utilization Model for Optimization of Checkpoint Intervals in Distributed Stream Processing Systems
Authors:
Sachini Jayasekara,
Aaron Harwood,
Shanika Karunasekera
Abstract:
State-of-the-art distributed stream processing systems such as Apache Flink and Storm have recently included checkpointing to provide fault-tolerance for stateful applications. This is a necessary eventuality as these systems head into the Exascale regime, and is evidently more efficient than replication as state size grows. However current systems use a nominal value for the checkpoint interval,…
▽ More
State-of-the-art distributed stream processing systems such as Apache Flink and Storm have recently included checkpointing to provide fault-tolerance for stateful applications. This is a necessary eventuality as these systems head into the Exascale regime, and is evidently more efficient than replication as state size grows. However current systems use a nominal value for the checkpoint interval, indicative of assuming roughly 1 failure every 19 days, that does not take into account the salient aspects of the checkpoint process, nor the system scale, which can readily lead to inefficient system operation. To address this shortcoming, we provide a rigorous derivation of utilization -- the fraction of total time available for the system to do useful work -- that incorporates checkpoint interval, failure rate, checkpoint cost, failure detection and restart cost, depth of the system topology and message delay. Our model yields an elegant expression for utilization and provides an optimal checkpoint interval given these parameters, interestingly showing it to be dependent only on checkpoint cost and failure rate. We confirm the accuracy and efficacy of our model through experiments with Apache Flink, where we obtain improvements in system utilization for every case, especially as the system size increases. Our model provides a solid theoretical basis for the analysis and optimization of more elaborate checkpointing approaches.
△ Less
Submitted 26 November, 2019;
originally announced November 2019.
-
Ultimate squeezing through coherent quantum feedback: A fair comparison with measurement-based schemes
Authors:
Alfred Harwood,
Alessio Serafini
Abstract:
We develop a general framework to describe interferometric coherent feedback loops and prove that, under any such scheme, the steady-state squeezing of a bosonic mode subject to a rotating wave coupling with a white noise environment and to any quadratic Hamiltonian must abide by a noise-dependent bound that reduces to the 3dB limit at zero temperature. Such a finding is contrasted, at fixed dynam…
▽ More
We develop a general framework to describe interferometric coherent feedback loops and prove that, under any such scheme, the steady-state squeezing of a bosonic mode subject to a rotating wave coupling with a white noise environment and to any quadratic Hamiltonian must abide by a noise-dependent bound that reduces to the 3dB limit at zero temperature. Such a finding is contrasted, at fixed dynamical parameters, with the performance of homodyne continuous monitoring of the output modes. The latter allows one to beat coherent feedback and the 3dB limit under certain dynamical conditions, which will be determined exactly.
△ Less
Submitted 16 October, 2019;
originally announced October 2019.
-
Multi-spatial Scale Event Detection from Geo-tagged Tweet Streams via Power-law Verification
Authors:
Yi Han,
Shanika Karunasekera,
Christopher Leckie,
Aaron Harwood
Abstract:
Compared with traditional news media, social media nowadays provides a richer and more timely source of news. We are interested in multi-spatial level event detection from geo-tagged tweet streams. Specifically, in this paper we (1) examine the statistical characteristic for the time series of the number of geo-tagged tweets posted from specific regions during a short time interval, e.g., ten seco…
▽ More
Compared with traditional news media, social media nowadays provides a richer and more timely source of news. We are interested in multi-spatial level event detection from geo-tagged tweet streams. Specifically, in this paper we (1) examine the statistical characteristic for the time series of the number of geo-tagged tweets posted from specific regions during a short time interval, e.g., ten seconds or one minute; (2) verify from over thirty datasets that while almost all such time series exhibit self-similarity, those that correspond to events, especially short-term and unplanned outbursts, follow a power-law distribution; (3) demonstrate that these findings can be applied to facilitate event detection from tweet streams. We propose two algorithms---Power-law basic and Power-law advanced, where Power-law basic only checks the existence of power-law distributions in the time series from tweet streams at multi-spatial scales, without looking into the content of each tweet, and Power-law advanced integrates power-law verification with semantic analysis via word embedding. Our experiments on multiple datasets show that by considering spatio-temporal statistical distributions of tweets alone, the seemingly naive algorithm of Power-law basic achieves comparable results with more advanced event detection methods, while the semantic analysis enhanced version, Power-law advanced, can significantly increase both the precision and the recall.
△ Less
Submitted 21 August, 2019; v1 submitted 12 June, 2019;
originally announced June 2019.
-
Geometry of Interest (GOI): Spatio-Temporal Destination Extraction and Partitioning in GPS Trajectory Data
Authors:
Seyed Morteza Mousavi,
Aaron Harwood,
Shanika Karunasekera,
Mojtaba Maghrebi
Abstract:
Nowadays large amounts of GPS trajectory data is being continuously collected by GPS-enabled devices such as vehicles navigation systems and mobile phones. GPS trajectory data is useful for applications such as traffic management, location forecasting, and itinerary planning. Such applications often need to extract the time-stamped Sequence of Visited Locations (SVLs) of the mobile objects. The ne…
▽ More
Nowadays large amounts of GPS trajectory data is being continuously collected by GPS-enabled devices such as vehicles navigation systems and mobile phones. GPS trajectory data is useful for applications such as traffic management, location forecasting, and itinerary planning. Such applications often need to extract the time-stamped Sequence of Visited Locations (SVLs) of the mobile objects. The nearest neighbor query (NNQ) is the most applied method for labeling the visited locations based on the IDs of the POIs in the process of SVL generation. NNQ in some scenarios is not accurate enough. To improve the quality of the extracted SVLs, instead of using NNQ, we label the visited locations as the IDs of the POIs which geometrically intersect with the GPS observations. Intersection operator requires the accurate geometry of the points of interest which we refer to them as the Geometries of Interest (GOIs). In some application domains (e.g. movement trajectories of animals), adequate information about the POIs and their GOIs may not be available a priori, or they may not be publicly accessible and, therefore, they need to be derived from GPS trajectory data. In this paper we propose a novel method for estimating the POIs and their GOIs, which consists of three phases: (i) extracting the geometries of the stay regions; (ii) constructing the geometry of destination regions based on the extracted stay regions; and (iii) constructing the GOIs based on the geometries of the destination regions. Using the geometric similarity to known GOIs as the major evaluation criterion, the experiments we performed using long-term GPS trajectory data show that our method outperforms the existing approaches.
△ Less
Submitted 16 May, 2016; v1 submitted 13 March, 2016;
originally announced March 2016.
-
An Adaptive Checkpointing Scheme for Peer-to-Peer Based Volunteer Computing Work Flows
Authors:
Lei Ni,
Aaron Harwood
Abstract:
Volunteer Computing, sometimes called Public Resource Computing, is an emerging computational model that is very suitable for work-pooled parallel processing. As more complex grid applications make use of work flows in their design and deployment it is reasonable to consider the impact of work flow deployment over a Volunteer Computing infrastructure. In this case, the inter work flow I/O can le…
▽ More
Volunteer Computing, sometimes called Public Resource Computing, is an emerging computational model that is very suitable for work-pooled parallel processing. As more complex grid applications make use of work flows in their design and deployment it is reasonable to consider the impact of work flow deployment over a Volunteer Computing infrastructure. In this case, the inter work flow I/O can lead to a significant increase in I/O demands at the work pool server. A possible solution is the use of a Peer-to- Peer based parallel computing architecture to off-load this I/O demand to the workers; where the workers can fulfill some aspects of work flow coordination and I/O checking, etc. However, achieving robustness in such a large scale system is a challenging hurdle towards the decentralized execution of work flows and general parallel processes. To increase robustness, we propose and show the merits of using an adaptive checkpoint scheme that efficiently checkpoints the status of the parallel processes according to the estimation of relevant network and peer parameters. Our scheme uses statistical data observed during runtime to dynamically make checkpoint decisions in a completely de- centralized manner. The results of simulation show support for our proposed approach in terms of reduced required runtime.
△ Less
Submitted 26 November, 2007;
originally announced November 2007.
-
Mean Field Models of Message Throughput in Dynamic Peer-to-Peer Systems
Authors:
Aaron Harwood,
Olga Ohrimenko
Abstract:
The churn rate of a peer-to-peer system places direct limitations on the rate at which messages can be effectively communicated to a group of peers. These limitations are independent of the topology and message transmission latency. In this paper we consider a peer-to-peer network, based on the Engset model, where peers arrive and depart independently at random. We show how the arrival and depar…
▽ More
The churn rate of a peer-to-peer system places direct limitations on the rate at which messages can be effectively communicated to a group of peers. These limitations are independent of the topology and message transmission latency. In this paper we consider a peer-to-peer network, based on the Engset model, where peers arrive and depart independently at random. We show how the arrival and departure rates directly limit the capacity for message streams to be broadcast to all other peers, by deriving mean field models that accurately describe the system behavior. Our models cover the unit and more general k buffer cases, i.e. where a peer can buffer at most k messages at any one time, and we give results for both single and multi-source message streams. We define coverage rate as peer-messages per unit time, i.e. the rate at which a number of peers receive messages, and show that the coverage rate is limited by the churn rate and buffer size. Our theory introduces an Instantaneous Message Exchange (IME) model and provides a template for further analysis of more complicated systems. Using the IME model, and assuming random processes, we have obtained very accurate equations of the system dynamics in a variety of interesting cases, that allow us to tune a peer-to-peer system. It remains to be seen if we can maintain this accuracy for general processes and when applying a non-instantaneous model.
△ Less
Submitted 14 May, 2007;
originally announced May 2007.
-
A Peer-to-Peer Browsable File Index using a Popularity Based Global Namespace
Authors:
Thomas Jacobs,
Aaron Harwood
Abstract:
The distribution of files using decentralized, peer-to-peer (P2P) systems, has significant advantages over centralized approaches. It is however more difficult to settle on the best approach for file sharing. Most file sharing systems are based on query string searches, leading to a relatively simple but inefficient broadcast or to an efficient but relatively complicated index in a structured en…
▽ More
The distribution of files using decentralized, peer-to-peer (P2P) systems, has significant advantages over centralized approaches. It is however more difficult to settle on the best approach for file sharing. Most file sharing systems are based on query string searches, leading to a relatively simple but inefficient broadcast or to an efficient but relatively complicated index in a structured environment. In this paper we use a browsable peer-to-peer file index consisting of files which serve as directory nodes, interconnecting to form a directory network. We implemented the system based on BitTorrent and Kademlia. The directory network inherits all of the advantages of decentralization and provides browsable, efficient searching. To avoid conflict between users in the P2P system while also imposing no additional restrictions, we allow multiple versions of each directory node to simultaneously exist -- using popularity as the basis for default browsing behavior. Users can freely add files and directory nodes to the network. We show, using a simulation of user behavior and file quality, that the popularity based system consistently leads users to a high quality directory network; above the average quality of user updates. Q
△ Less
Submitted 29 January, 2007;
originally announced January 2007.
-
Towards Parallel Computing on the Internet: Applications, Architectures, Models and Programming Tools
Authors:
Elankovan Sundararajan,
Aaron Harwood
Abstract:
The development of Internet wide resources for general purpose parallel computing poses the challenging task of matching computation and communication complexity. A number of parallel computing models exist that address this for traditional parallel architectures, and there are a number of emerging models that attempt to do this for large scale Internet-based systems like computational grids. In…
▽ More
The development of Internet wide resources for general purpose parallel computing poses the challenging task of matching computation and communication complexity. A number of parallel computing models exist that address this for traditional parallel architectures, and there are a number of emerging models that attempt to do this for large scale Internet-based systems like computational grids. In this survey we cover the three fundamental aspects -- application, architecture and model, and we show how they have been developed over the last decade. We also cover programming tools that are currently being used for parallel programming in computational grids. The trend in conventional computational models are to put emphasis on efficient communication between participating nodes by adapting different types of communication to network conditions. Effects of dynamism and uncertainties that arise in large scale systems are evidently important to understand and yet there is currently little work that addresses this from a parallel computing perspective.
△ Less
Submitted 24 December, 2006; v1 submitted 21 December, 2006;
originally announced December 2006.
-
Lossy Bulk Synchronous Parallel Processing Model for Very Large Scale Grids
Authors:
Elankovan Sundararajan,
Aaron Harwood,
Kotagiri Ramamohanarao
Abstract:
The performance of a parallel algorithm in a very large scale grid is significantly influenced by the underlying Internet protocols and inter-connectivity. Many grid programming platforms use TCP due to its reliability, usually with some optimizations to reduce its costs. However, TCP does not perform well in a high bandwidth and high delay network environment. On the other hand, UDP is the fast…
▽ More
The performance of a parallel algorithm in a very large scale grid is significantly influenced by the underlying Internet protocols and inter-connectivity. Many grid programming platforms use TCP due to its reliability, usually with some optimizations to reduce its costs. However, TCP does not perform well in a high bandwidth and high delay network environment. On the other hand, UDP is the fastest protocol available because it omits connection setup process, acknowledgments and retransmissions sacrificing reliable transfer. Many new bulk data transfer schemes using UDP for data transmission such as RBUDP, Tsunami, and SABUL have been introduced and shown to have better performance compared to TCP. In this paper, we consider the use of UDP and examine the relationship between packet loss and speedup with respect to the number of grid nodes. Our measurement suggests that packet loss rates between 5%-15% on average are not uncommon between PlanetLab nodes that are widely distributed over the Internet. We show that transmitting multiple copies of same packet produces higher speedup. We show the minimum number of packet duplication required to maximize the possible speedup for a given number of nodes using a BSP based model. Our work demonstrates that by using an appropriate number of packet copies, we can increase performance of parallel program.
△ Less
Submitted 20 December, 2006; v1 submitted 19 November, 2006;
originally announced November 2006.
-
A Taxonomy of Peer-to-Peer Based Complex Queries: a Grid perspective
Authors:
Rajiv Ranjan,
Aaron Harwood,
Rajkumar Buyya
Abstract:
Grid superscheduling requires support for efficient and scalable discovery of resources. Resource discovery activities involve searching for the appropriate resource types that match the user's job requirements. To accomplish this goal, a resource discovery system that supports the desired look-up operation is mandatory. Various kinds of solutions to this problem have been suggested, including t…
▽ More
Grid superscheduling requires support for efficient and scalable discovery of resources. Resource discovery activities involve searching for the appropriate resource types that match the user's job requirements. To accomplish this goal, a resource discovery system that supports the desired look-up operation is mandatory. Various kinds of solutions to this problem have been suggested, including the centralised and hierarchical information server approach. However, both of these approaches have serious limitations in regards to scalability, fault-tolerance and network congestion. To overcome these limitations, organising resource information using Peer-to-Peer (P2P) network model has been proposed. Existing approaches advocate an extension to structured P2P protocols, to support the Grid resource information system (GRIS). In this paper, we identify issues related to the design of such an efficient, scalable, fault-tolerant, consistent and practical GRIS system using a P2P network model. We compile these issues into various taxonomies in sections III and IV. Further, we look into existing works that apply P2P based network protocols to GRIS. We think that this taxonomy and its map** to relevant systems would be useful for academic and industry based researchers who are engaged in the design of scalable Grid systems.
△ Less
Submitted 30 October, 2006;
originally announced October 2006.
-
A Case for Cooperative and Incentive-Based Coupling of Distributed Clusters
Authors:
Rajiv Ranjan,
Aaron Harwood,
Rajkumar Buyya
Abstract:
Research interest in Grid computing has grown significantly over the past five years. Management of distributed resources is one of the key issues in Grid computing. Central to management of resources is the effectiveness of resource allocation as it determines the overall utility of the system. The current approaches to superscheduling in a grid environment are non-coordinated since application…
▽ More
Research interest in Grid computing has grown significantly over the past five years. Management of distributed resources is one of the key issues in Grid computing. Central to management of resources is the effectiveness of resource allocation as it determines the overall utility of the system. The current approaches to superscheduling in a grid environment are non-coordinated since application level schedulers or brokers make scheduling decisions independently of the others in the system. Clearly, this can exacerbate the load sharing and utilization problems of distributed resources due to suboptimal schedules that are likely to occur. To overcome these limitations, we propose a mechanism for coordinated sharing of distributed clusters based on computational economy. The resulting environment, called \emph{Grid-Federation}, allows the transparent use of resources from the federation when local resources are insufficient to meet its users' requirements. The use of computational economy methodology in coordinating resource allocation not only facilitates the QoS based scheduling, but also enhances utility delivered by resources.
△ Less
Submitted 15 May, 2006;
originally announced May 2006.
-
SLA-Based Coordinated Superscheduling Scheme and Performance for Computational Grids
Authors:
Rajiv Ranjan,
Aaron Harwood,
Rajkumar Buyya
Abstract:
The Service Level Agreement~(SLA) based grid superscheduling approach promotes coordinated resource sharing. Superscheduling is facilitated between administratively and topologically distributed grid sites by grid schedulers such as Resource brokers. In this work, we present a market-based SLA coordination mechanism. We based our SLA model on a well known \emph{contract net protocol}.
The key…
▽ More
The Service Level Agreement~(SLA) based grid superscheduling approach promotes coordinated resource sharing. Superscheduling is facilitated between administratively and topologically distributed grid sites by grid schedulers such as Resource brokers. In this work, we present a market-based SLA coordination mechanism. We based our SLA model on a well known \emph{contract net protocol}.
The key advantages of our approach are that it allows:~(i) resource owners to have finer degree of control over the resource allocation that was previously not possible through traditional mechanism; and (ii) superschedulers to bid for SLA contracts in the contract net with focus on completing the job within the user specified deadline. In this work, we use simulation to show the effectiveness of our proposed approach.
△ Less
Submitted 15 May, 2006;
originally announced May 2006.
-
Diffusive Load Balancing of Loosely-Synchronous Parallel Programs over Peer-to-Peer Networks
Authors:
Scott Douglas,
Aaron Harwood
Abstract:
The use of under-utilized Internet resources is widely recognized as a viable form of high performance computing. Sustained processing power of roughly 40T FLOPS using 4 million volunteered Internet hosts has been reported for embarrassingly parallel problems. At the same time, peer-to-peer (P2P) file sharing networks, with more than 50 million participants, have demonstrated the capacity for sc…
▽ More
The use of under-utilized Internet resources is widely recognized as a viable form of high performance computing. Sustained processing power of roughly 40T FLOPS using 4 million volunteered Internet hosts has been reported for embarrassingly parallel problems. At the same time, peer-to-peer (P2P) file sharing networks, with more than 50 million participants, have demonstrated the capacity for scale in distributed systems. This paper contributes a study of load balancing techniques for a general class of loosely-synchronous parallel algorithms when executed over a P2P network. We show that decentralized, diffusive load balancing can be effective at balancing load and is facilitated by the dynamic properties of P2P. While a moderate degree of dynamicity can benefit load balancing, significant dynamicity hinders the parallel program performance due to the need for increased load migration. To the best of our knowledge this study provides new insight into the performance of loosely-synchronous parallel programs over the Internet.
△ Less
Submitted 5 October, 2004;
originally announced October 2004.
-
Analytical formulations of Peer-to-Peer Connection Efficiency
Authors:
Aaron Harwood
Abstract:
Use of Peer-to-Peer (P2P) service networks introduces a new communication paradigm because peers are both clients and servers and so each peer may provide/request services to/from other peers. Empirical studies of P2P networks have been undertaken and reveal useful characteristics. However there is to date little analytical work to describe P2P networks with respect to their communication paradi…
▽ More
Use of Peer-to-Peer (P2P) service networks introduces a new communication paradigm because peers are both clients and servers and so each peer may provide/request services to/from other peers. Empirical studies of P2P networks have been undertaken and reveal useful characteristics. However there is to date little analytical work to describe P2P networks with respect to their communication paradigm and their interconnections. This paper provides an analytical formulation and optimisation of peer connection efficiency, in terms of minimising the fraction of wasted connection time. Peer connection efficiency is analysed for both a uni- and multi-connected peer. Given this fundamental optimisation, the paper optimises the number of connections that peers should make use of as a function of network load, in terms of minimising the total queue size that requests in the P2P network experience. The results of this paper provide a basis for engineering high performance P2P interconnection networks. The optimisations are useful for reducing bandwidth and power consumption, e.g. in the case of peers being mobile devices with a limited power supply. Also these results could be used to determine when a (virtual) circuit should be switched to support a connection.
△ Less
Submitted 13 February, 2003;
originally announced February 2003.
