Showing 1–50 of 79 results for author: Mahdavi, M

Prize-Collecting Steiner Tree: A 1.79 Approximation

Authors: Ali Ahmadi, Iman Gholami, MohammadTaghi Hajiaghayi, Peyman Jabbarzade, Mohammad Mahdavi

Abstract: Prize-Collecting Steiner Tree (PCST) is a generalization of the Steiner Tree problem, a fundamental problem in computer science. In the classic Steiner Tree problem, we aim to connect a set of vertices known as terminals using the minimum-weight tree in a given weighted graph. In this generalized version, each vertex has a penalty, and there is flexibility to decide whether to connect each vertex… ▽ More Prize-Collecting Steiner Tree (PCST) is a generalization of the Steiner Tree problem, a fundamental problem in computer science. In the classic Steiner Tree problem, we aim to connect a set of vertices known as terminals using the minimum-weight tree in a given weighted graph. In this generalized version, each vertex has a penalty, and there is flexibility to decide whether to connect each vertex or pay its associated penalty, making the problem more realistic and practical. Both the Steiner Tree problem and its Prize-Collecting version had long-standing $2$-approximation algorithms, matching the integrality gap of the natural LP formulations for both. This barrier for both problems has been surpassed, with algorithms achieving approximation factors below $2$. While research on the Steiner Tree problem has led to a series of reductions in the approximation ratio below $2$, culminating in a $\ln(4)+ε$ approximation by Byrka, Grandoni, Rothvoß, and Sanità, the Prize-Collecting version has not seen improvements in the past 15 years since the work of Archer, Bateni, Hajiaghayi, and Karloff, which reduced the approximation factor for this problem from $2$ to $1.9672$. Interestingly, even the Prize-Collecting TSP approximation, which was first improved below $2$ in the same paper, has seen several advancements since then. In this paper, we reduce the approximation factor for the PCST problem substantially to 1.7994 via a novel iterative approach. △ Less

Submitted 6 May, 2024; originally announced May 2024.

On the Generalization Ability of Unsupervised Pretraining

Authors: Yuyang Deng, Junyuan Hong, Jiayu Zhou, Mehrdad Mahdavi

Abstract: Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. However, a rigorous understanding of how the representation function learned on an unlabeled dataset affects the generalization of the fine-tuned model is lacking. Existing theoretical research does not adequately account for the heterogeneity of the distri… ▽ More Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. However, a rigorous understanding of how the representation function learned on an unlabeled dataset affects the generalization of the fine-tuned model is lacking. Existing theoretical research does not adequately account for the heterogeneity of the distribution and tasks in pre-training and fine-tuning stage. To bridge this gap, this paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase, ultimately affecting the generalization capabilities of the fine-tuned model on downstream tasks. We apply our theoretical framework to analyze generalization bound of two distinct scenarios: Context Encoder pre-training with deep neural networks and Masked Autoencoder pre-training with deep transformers, followed by fine-tuning on a binary classification task. Finally, inspired by our findings, we propose a novel regularization method during pre-training to further enhances the generalization of fine-tuned model. Overall, our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms. △ Less

Submitted 11 March, 2024; originally announced March 2024.

On the Generalization Capability of Temporal Graph Learning Algorithms: Theoretical Insights and a Simpler Method

Authors: Weilin Cong, Jian Kang, Hanghang Tong, Mehrdad Mahdavi

Abstract: Temporal Graph Learning (TGL) has become a prevalent technique across diverse real-world applications, especially in domains where data can be represented as a graph and evolves over time. Although TGL has recently seen notable progress in algorithmic solutions, its theoretical foundations remain largely unexplored. This paper aims at bridging this gap by investigating the generalization ability o… ▽ More Temporal Graph Learning (TGL) has become a prevalent technique across diverse real-world applications, especially in domains where data can be represented as a graph and evolves over time. Although TGL has recently seen notable progress in algorithmic solutions, its theoretical foundations remain largely unexplored. This paper aims at bridging this gap by investigating the generalization ability of different TGL algorithms (e.g., GNN-based, RNN-based, and memory-based methods) under the finite-wide over-parameterized regime. We establish the connection between the generalization error of TGL algorithms and "the number of layers/steps" in the GNN-/RNN-based TGL methods and "the feature-label alignment (FLA) score", where FLA can be used as a proxy for the expressive power and explains the performance of memory-based methods. Guided by our theoretical analysis, we propose Simplified-Temporal-Graph-Network, which enjoys a small generalization error, improved overall performance, and lower model complexity. Extensive experiments on real-world datasets demonstrate the effectiveness of our method. Our theoretical findings and proposed algorithm offer essential insights into TGL from a theoretical standpoint, laying the groundwork for the designing practical TGL algorithms in future studies. △ Less

Submitted 26 February, 2024; originally announced February 2024.

Distributed Personalized Empirical Risk Minimization

Authors: Yuyang Deng, Mohammad Mahdi Kamani, Pouria Mahdavinia, Mehrdad Mahdavi

Abstract: This paper advocates a new paradigm Personalized Empirical Risk Minimization (PERM) to facilitate learning from heterogeneous data sources without imposing stringent constraints on computational resources shared by participating devices. In PERM, we aim to learn a distinct model for each client by learning who to learn with and personalizing the aggregation of local empirical losses by effectively… ▽ More This paper advocates a new paradigm Personalized Empirical Risk Minimization (PERM) to facilitate learning from heterogeneous data sources without imposing stringent constraints on computational resources shared by participating devices. In PERM, we aim to learn a distinct model for each client by learning who to learn with and personalizing the aggregation of local empirical losses by effectively estimating the statistical discrepancy among data distributions, which entails optimal statistical accuracy for all local distributions and overcomes the data heterogeneity issue. To learn personalized models at scale, we propose a distributed algorithm that replaces the standard model averaging with model shuffling to simultaneously optimize PERM objectives for all devices. This also allows us to learn distinct model architectures (e.g., neural networks with different numbers of parameters) for different clients, thus confining underlying memory and compute resources of individual clients. We rigorously analyze the convergence of the proposed algorithm and conduct experiments that corroborate the effectiveness of the proposed paradigm. △ Less

Submitted 26 October, 2023; originally announced October 2023.

Stochastic Quantum Sampling for Non-Logconcave Distributions and Estimating Partition Functions

Authors: Guneykan Ozgul, Xiantao Li, Mehrdad Mahdavi, Chunhao Wang

Abstract: We present quantum algorithms for sampling from non-logconcave probability distributions in the form of $π(x) \propto \exp(-βf(x))$. Here, $f$ can be written as a finite sum $f(x):= \frac{1}{N}\sum_{k=1}^N f_k(x)$. Our approach is based on quantum simulated annealing on slowly varying Markov chains derived from unadjusted Langevin algorithms, removing the necessity for function evaluations which c… ▽ More We present quantum algorithms for sampling from non-logconcave probability distributions in the form of $π(x) \propto \exp(-βf(x))$. Here, $f$ can be written as a finite sum $f(x):= \frac{1}{N}\sum_{k=1}^N f_k(x)$. Our approach is based on quantum simulated annealing on slowly varying Markov chains derived from unadjusted Langevin algorithms, removing the necessity for function evaluations which can be computationally expensive for large data sets in mixture modeling and multi-stable systems. We also incorporate a stochastic gradient oracle that implements the quantum walk operators inexactly by only using mini-batch gradients. As a result, our stochastic gradient based algorithm only accesses small subsets of data points in implementing the quantum walk. One challenge of quantizing the resulting Markov chains is that they do not satisfy the detailed balance condition in general. Consequently, the mixing time of the algorithm cannot be expressed in terms of the spectral gap of the transition density, making the quantum algorithms nontrivial to analyze. To overcome these challenges, we first build a hypothetical Markov chain that is reversible, and also converges to the target distribution. Then, we quantified the distance between our algorithm's output and the target distribution by using this hypothetical chain as a bridge to establish the total complexity. Our quantum algorithms exhibit polynomial speedups in terms of both dimension and precision dependencies when compared to the best-known classical algorithms. △ Less

Submitted 17 October, 2023; originally announced October 2023.

Comments: 32 pages

Regret Analysis of Repeated Delegated Choice

Authors: MohammadTaghi Hajiaghayi, Mohammad Mahdavi, Keivan Rezaei, Suho Shin

Abstract: We present a study on a repeated delegated choice problem, which is the first to consider an online learning variant of Kleinberg and Kleinberg, EC'18. In this model, a principal interacts repeatedly with an agent who possesses an exogenous set of solutions to search for efficient ones. Each solution can yield varying utility for both the principal and the agent, and the agent may propose a soluti… ▽ More We present a study on a repeated delegated choice problem, which is the first to consider an online learning variant of Kleinberg and Kleinberg, EC'18. In this model, a principal interacts repeatedly with an agent who possesses an exogenous set of solutions to search for efficient ones. Each solution can yield varying utility for both the principal and the agent, and the agent may propose a solution to maximize its own utility in a selfish manner. To mitigate this behavior, the principal announces an eligible set which screens out a certain set of solutions. The principal, however, does not have any information on the distribution of solutions in advance. Therefore, the principal dynamically announces various eligible sets to efficiently learn the distribution. The principal's objective is to minimize cumulative regret compared to the optimal eligible set in hindsight. We explore two dimensions of the problem setup, whether the agent behaves myopically or strategizes across the rounds, and whether the solutions yield deterministic or stochastic utility. Our analysis mainly characterizes some regimes under which the principal can recover the sublinear regret, thereby shedding light on the rise and fall of the repeated delegation procedure in various regimes. △ Less

Submitted 13 February, 2024; v1 submitted 7 October, 2023; originally announced October 2023.

Understanding Deep Gradient Leakage via Inversion Influence Functions

Authors: Haobo Zhang, Junyuan Hong, Yuyang Deng, Mehrdad Mahdavi, Jiayu Zhou

Abstract: Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors. This attack casts significant privacy challenges on distributed learning from clients with sensitive data, where clients are required to share gradients. Defending against such attacks requires but lacks an understanding of when and how privacy leakage happens, mostly because of th… ▽ More Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors. This attack casts significant privacy challenges on distributed learning from clients with sensitive data, where clients are required to share gradients. Defending against such attacks requires but lacks an understanding of when and how privacy leakage happens, mostly because of the black-box nature of deep networks. In this paper, we propose a novel Inversion Influence Function (I$^2$F) that establishes a closed-form connection between the recovered images and the private gradients by implicitly solving the DGL problem. Compared to directly solving DGL, I$^2$F is scalable for analyzing deep networks, requiring only oracle access to gradients and Jacobian-vector products. We empirically demonstrate that I$^2$F effectively approximated the DGL generally on different model architectures, datasets, modalities, attack implementations, and perturbation-based defenses. With this novel tool, we provide insights into effective gradient perturbation directions, the unfairness of privacy protection, and privacy-preferred model initialization. Our codes are provided in https://github.com/illidanlab/inversion-influence-function. △ Less

Submitted 8 January, 2024; v1 submitted 22 September, 2023; originally announced September 2023.

Comments: 24 pages, 18 figures, accepted to NeurIPS2023

Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptation

Authors: Yuyang Deng, Ilja Kuzborskij, Mehrdad Mahdavi

Abstract: We consider the problem of learning a model from multiple heterogeneous sources with the goal of performing well on a new target distribution. The goal of learner is to mix these data sources in a target-distribution aware way and simultaneously minimize the empirical risk on the mixed source. The literature has made some tangible advancements in establishing theory of learning on mixture domain.… ▽ More We consider the problem of learning a model from multiple heterogeneous sources with the goal of performing well on a new target distribution. The goal of learner is to mix these data sources in a target-distribution aware way and simultaneously minimize the empirical risk on the mixed source. The literature has made some tangible advancements in establishing theory of learning on mixture domain. However, there are still two unsolved problems. Firstly, how to estimate the optimal mixture of sources, given a target domain; Secondly, when there are numerous target domains, how to solve empirical risk minimization (ERM) for each target using possibly unique mixture of data sources in a computationally efficient manner. In this paper we address both problems efficiently and with guarantees. We cast the first problem, mixture weight estimation, as a convex-nonconcave compositional minimax problem, and propose an efficient stochastic algorithm with provable stationarity guarantees. Next, for the second problem, we identify that for certain regimes, solving ERM for each target domain individually can be avoided, and instead parameters for a target optimal model can be viewed as a non-linear function on a space of the mixture coefficients. Building upon this, we show that in the offline setting, a GD-trained overparameterized neural network can provably learn such function to predict the model of target domain instead of solving a designated ERM problem. Finally, we also consider an online setting and propose a label efficient online algorithm, which predicts parameters for new targets given an arbitrary sequence of mixing coefficients, while enjoying regret guarantees. △ Less

Submitted 12 November, 2023; v1 submitted 19 September, 2023; originally announced September 2023.

2-Approximation for Prize-Collecting Steiner Forest

Authors: Ali Ahmadi, Iman Gholami, MohammadTaghi Hajiaghayi, Peyman Jabbarzade, Mohammad Mahdavi

Abstract: Approximation algorithms for the prize-collecting Steiner forest problem (PCSF) have been a subject of research for over three decades, starting with the seminal works of Agrawal, Klein, and Ravi and Goemans and Williamson on Steiner forest and prize-collecting problems. In this paper, we propose and analyze a natural deterministic algorithm for PCSF that achieves a $2$-approximate solution in pol… ▽ More Approximation algorithms for the prize-collecting Steiner forest problem (PCSF) have been a subject of research for over three decades, starting with the seminal works of Agrawal, Klein, and Ravi and Goemans and Williamson on Steiner forest and prize-collecting problems. In this paper, we propose and analyze a natural deterministic algorithm for PCSF that achieves a $2$-approximate solution in polynomial time. This represents a significant improvement compared to the previously best known algorithm with a $2.54$-approximation factor developed by Hajiaghayi and Jain in 2006. Furthermore, K{ö}nemann, Olver, Pashkovich, Ravi, Swamy, and Vygen have established an integrality gap of at least $9/4$ for the natural LP relaxation for PCSF. However, we surpass this gap through the utilization of a combinatorial algorithm and a novel analysis technique. Since $2$ is the best known approximation guarantee for Steiner forest problem, which is a special case of PCSF, our result matches this factor and closes the gap between the Steiner forest problem and its generalized version, PCSF. △ Less

Submitted 6 May, 2024; v1 submitted 10 September, 2023; originally announced September 2023.

Introducing a New Evaluation Criteria for EMD-Base Steganography Method

Authors: Hanieh Rafiee, Mojtaba Mahdavi, AhmadReza NaghshNilchi

Abstract: Steganography is a technique to hide the presence of secret communication. When one of the communication elements is under the influence of the enemy, it can be used. The main measure to evaluate steganography methods in a certain capacity is security. Therefore, in a certain capacity, reducing the amount of changes in the cover media, creates a higher embedding efficiency and thus more security o… ▽ More Steganography is a technique to hide the presence of secret communication. When one of the communication elements is under the influence of the enemy, it can be used. The main measure to evaluate steganography methods in a certain capacity is security. Therefore, in a certain capacity, reducing the amount of changes in the cover media, creates a higher embedding efficiency and thus more security of an steganography method. Mostly, security and capacity are in conflict with each other, the increase of one lead to the decrease of the other. The presence of a single criterion that represents security and capacity at the same time be useful in comparing steganography methods. EMD and the relevant methods are a group of steganography techniques, which optimize the amount of changes resulting from embedding (security). The present paper is aimed to provide an evaluation criterion for this group of steganography methods. In this study, after a general review and comparison of EMD-based steganography techniques, we present a method to compare them exactly, from the perspective of embedding efficiency. First, a formula is presented to determine the value of embedding efficiency, which indicates the effect of one or more changes on one or more pixels. The results demonstrate that the proposed embedding efficiency formula shows the performance of the methods better when several changes are made on a pixel compared to the existing criteria. In the second step, we have obtained an upper bound, which determines the best efficiency for each certain capacity. Finally, based on the introduced bound, another evaluation criterion for a better comparison of the methods is presented. △ Less

Submitted 15 August, 2023; originally announced August 2023.

EchoGLAD: Hierarchical Graph Neural Networks for Left Ventricle Landmark Detection on Echocardiograms

Authors: Masoud Mokhtari, Mobina Mahdavi, Hooman Vaseli, Christina Luong, Purang Abolmaesumi, Teresa S. M. Tsang, Renjie Liao

Abstract: The functional assessment of the left ventricle chamber of the heart requires detecting four landmark locations and measuring the internal dimension of the left ventricle and the approximate mass of the surrounding muscle. The key challenge of automating this task with machine learning is the sparsity of clinical labels, i.e., only a few landmark pixels in a high-dimensional image are annotated, l… ▽ More The functional assessment of the left ventricle chamber of the heart requires detecting four landmark locations and measuring the internal dimension of the left ventricle and the approximate mass of the surrounding muscle. The key challenge of automating this task with machine learning is the sparsity of clinical labels, i.e., only a few landmark pixels in a high-dimensional image are annotated, leading many prior works to heavily rely on isotropic label smoothing. However, such a label smoothing strategy ignores the anatomical information of the image and induces some bias. To address this challenge, we introduce an echocardiogram-based, hierarchical graph neural network (GNN) for left ventricle landmark detection (EchoGLAD). Our main contributions are: 1) a hierarchical graph representation learning framework for multi-resolution landmark detection via GNNs; 2) induced hierarchical supervision at different levels of granularity using a multi-level loss. We evaluate our model on a public and a private dataset under the in-distribution (ID) and out-of-distribution (OOD) settings. For the ID setting, we achieve the state-of-the-art mean absolute errors (MAEs) of 1.46 mm and 1.86 mm on the two datasets. Our model also shows better OOD generalization than prior works with a testing MAE of 4.3 mm. △ Less

Submitted 23 July, 2023; originally announced July 2023.

Comments: To be published in MICCAI 2023

On the Hardness of Robustness Transfer: A Perspective from Rademacher Complexity over Symmetric Difference Hypothesis Space

Authors: Yuyang Deng, Nidham Gazagnadou, Junyuan Hong, Mehrdad Mahdavi, Lingjuan Lyu

Abstract: Recent studies demonstrated that the adversarially robust learning under $\ell_\infty$ attack is harder to generalize to different domains than standard domain adaptation. How to transfer robustness across different domains has been a key question in domain adaptation field. To investigate the fundamental difficulty behind adversarially robust domain adaptation (or robustness transfer), we propose… ▽ More Recent studies demonstrated that the adversarially robust learning under $\ell_\infty$ attack is harder to generalize to different domains than standard domain adaptation. How to transfer robustness across different domains has been a key question in domain adaptation field. To investigate the fundamental difficulty behind adversarially robust domain adaptation (or robustness transfer), we propose to analyze a key complexity measure that controls the cross-domain generalization: the adversarial Rademacher complexity over {\em symmetric difference hypothesis space} $\mathcal{H} Δ\mathcal{H}$. For linear models, we show that adversarial version of this complexity is always greater than the non-adversarial one, which reveals the intrinsic hardness of adversarially robust domain adaptation. We also establish upper bounds on this complexity measure. Then we extend them to the ReLU neural network class by upper bounding the adversarial Rademacher complexity in the binary classification setting. Finally, even though the robust domain adaptation is provably harder, we do find positive relation between robust learning and standard domain adaptation. We explain \emph{how adversarial training helps domain adaptation in terms of standard risk}. We believe our results initiate the study of the generalization theory of adversarially robust domain adaptation, and could shed lights on distributed adversarially robust learning from heterogeneous sources, e.g., federated learning scenario. △ Less

Submitted 23 February, 2023; originally announced February 2023.

Do We Really Need Complicated Model Architectures For Temporal Networks?

Authors: Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, Mehrdad Mahdavi

Abstract: Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, a conceptually and technically simple architecture that consists… ▽ More Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, a conceptually and technically simple architecture that consists of three components: (1) a link-encoder that is only based on multi-layer perceptrons (MLP) to summarize the information from temporal links, (2) a node-encoder that is only based on neighbor mean-pooling to summarize node information, and (3) an MLP-based link classifier that performs link prediction based on the outputs of the encoders. Despite its simplicity, GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance. These results motivate us to rethink the importance of simpler model architecture. △ Less

Submitted 22 February, 2023; originally announced February 2023.

Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projection

Authors: Weilin Cong, Mehrdad Mahdavi

Abstract: As privacy protection receives much attention, unlearning the effect of a specific node from a pre-trained graph learning model has become equally important. However, due to the node dependency in the graph-structured data, representation unlearning in Graph Neural Networks (GNNs) is challenging and less well explored. In this paper, we fill in this gap by first studying the unlearning problem in… ▽ More As privacy protection receives much attention, unlearning the effect of a specific node from a pre-trained graph learning model has become equally important. However, due to the node dependency in the graph-structured data, representation unlearning in Graph Neural Networks (GNNs) is challenging and less well explored. In this paper, we fill in this gap by first studying the unlearning problem in linear-GNNs, and then introducing its extension to non-linear structures. Given a set of nodes to unlearn, we propose PROJECTOR that unlearns by projecting the weight parameters of the pre-trained model onto a subspace that is irrelevant to features of the nodes to be forgotten. PROJECTOR could overcome the challenges caused by node dependency and enjoys a perfect data removal, i.e., the unlearned model parameters do not contain any information about the unlearned node features which is guaranteed by algorithmic construction. Empirical results on real-world datasets illustrate the effectiveness and efficiency of PROJECTOR. △ Less

Submitted 17 February, 2023; originally announced February 2023.

Tight Analysis of Extra-gradient and Optimistic Gradient Methods For Nonconvex Minimax Problems

Authors: Pouria Mahdavinia, Yuyang Deng, Haochuan Li, Mehrdad Mahdavi

Abstract: Despite the established convergence theory of Optimistic Gradient Descent Ascent (OGDA) and Extragradient (EG) methods for the convex-concave minimax problems, little is known about the theoretical guarantees of these methods in nonconvex settings. To bridge this gap, for the first time, this paper establishes the convergence of OGDA and EG methods under the nonconvex-strongly-concave (NC-SC) and… ▽ More Despite the established convergence theory of Optimistic Gradient Descent Ascent (OGDA) and Extragradient (EG) methods for the convex-concave minimax problems, little is known about the theoretical guarantees of these methods in nonconvex settings. To bridge this gap, for the first time, this paper establishes the convergence of OGDA and EG methods under the nonconvex-strongly-concave (NC-SC) and nonconvex-concave (NC-C) settings by providing a unified analysis through the lens of single-call extra-gradient methods. We further establish lower bounds on the convergence of GDA/OGDA/EG, shedding light on the tightness of our analysis. We also conduct experiments supporting our theoretical results. We believe our results will advance the theoretical understanding of OGDA and EG methods for solving complicated nonconvex minimax real-world problems, e.g., Generative Adversarial Networks (GANs) or robust neural networks training. △ Less

Submitted 17 October, 2022; originally announced October 2022.

A Novel Hybrid Backscatter and Conventional Algorithm for Multi-Hop IoT Networks

Authors: Mahmoud Raeisi, Mehdi Mahdavi, Ali Mohammad Doost Hosseini

Abstract: This paper investigates a multi-hop cognitive radio network in terms of end-to-end bit delivery. The network exploits backscatter communication (BackCom) and harvest-then-transmit (HTT) mode in a hybrid manner. Such a network can be used in internet of things (IoT) applications in which IoT users coexist with a primary network (PN) and use the primary spectrum to transmit data in both BackCom and… ▽ More This paper investigates a multi-hop cognitive radio network in terms of end-to-end bit delivery. The network exploits backscatter communication (BackCom) and harvest-then-transmit (HTT) mode in a hybrid manner. Such a network can be used in internet of things (IoT) applications in which IoT users coexist with a primary network (PN) and use the primary spectrum to transmit data in both BackCom and HTT modes. Besides, such users can harvest energy from the primary signals. A novel hybrid backscatter and conventional transmission (HBCT) algorithm is proposed in order to maximize end-to-end bit delivery by jointly optimizing time and power allocations. For this goal, we formulate a non-convex optimization problem. Next, we transform the problem into a convex one and develop a new analytical formulation by which we calculate the optimal power and time allocation in closed-form equations. The numerical results demonstrate the superiority of HBCT compared with current schemes. △ Less

Submitted 1 May, 2022; originally announced May 2022.

Journal ref: Transactions on Emerging Telecommunications Technologies, Volume 33, Issue 12, 2022, 19

Learning Distributionally Robust Models at Scale via Composite Optimization

Authors: Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, Amin Karbasi

Abstract: To train machine learning models that are robust to distribution shifts in the data, distributionally robust optimization (DRO) has been proven very effective. However, the existing approaches to learning a distributionally robust model either require solving complex optimization problems such as semidefinite programming or a first-order method whose convergence scales linearly with the number of… ▽ More To train machine learning models that are robust to distribution shifts in the data, distributionally robust optimization (DRO) has been proven very effective. However, the existing approaches to learning a distributionally robust model either require solving complex optimization problems such as semidefinite programming or a first-order method whose convergence scales linearly with the number of data samples -- which hinders their scalability to large datasets. In this paper, we show how different variants of DRO are simply instances of a finite-sum composite optimization for which we provide scalable methods. We also provide empirical results that demonstrate the effectiveness of our proposed algorithm with respect to the prior art in order to learn robust models from very large datasets. △ Less

Submitted 17 March, 2022; originally announced March 2022.

Comments: Accepted to ICLR2022 as a conference paper. International Conference on Learning Representations (2022)

DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability

Authors: Weilin Cong, Yanhong Wu, Yuandong Tian, Mengting Gu, Yinglong Xia, Chun-cheng Jason Chen, Mehrdad Mahdavi

Abstract: Transformers have achieved great success in several domains, including Natural Language Processing and Computer Vision. However, its application to real-world graphs is less explored, mainly due to its high computation cost and its poor generalizability caused by the lack of enough training data in the graph domain. To fill in this gap, we propose a scalable Transformer-like dynamic graph learning… ▽ More Transformers have achieved great success in several domains, including Natural Language Processing and Computer Vision. However, its application to real-world graphs is less explored, mainly due to its high computation cost and its poor generalizability caused by the lack of enough training data in the graph domain. To fill in this gap, we propose a scalable Transformer-like dynamic graph learning method named Dynamic Graph Transformer (DyFormer) with spatial-temporal encoding to effectively learn graph topology and capture implicit links. To achieve efficient and scalable training, we propose temporal-union graph structure and its associated subgraph-based node sampling strategy. To improve the generalization ability, we introduce two complementary self-supervised pre-training tasks and show that jointly optimizing the two pre-training tasks results in a smaller Bayesian error rate via an information-theoretic analysis. Extensive experiments on the real-world datasets illustrate that DyFormer achieves a consistent 1%-3% AUC gain (averaged over all time steps) compared with baselines on all benchmarks. △ Less

Submitted 29 January, 2023; v1 submitted 19 November, 2021; originally announced November 2021.

Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

Authors: Morteza Ramezani, Weilin Cong, Mehrdad Mahdavi, Mahmut T. Kandemir, Anand Sivasubramaniam

Abstract: Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GN… ▽ More Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability. To overcome these issues, we propose a communication-efficient distributed GNN training technique named $\text{Learn Locally, Correct Globally}$ (LLCG). To reduce the communication and memory overhead, each local machine in LLCG first trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging. However, ignoring node dependency could result in significant performance degradation. To solve the performance degradation, we propose to apply $\text{Global Server Corrections}$ on the server to refine the locally learned models. We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error. However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate. Extensive experiments on real-world datasets show that LLCG can significantly improve the efficiency without hurting the performance. △ Less

Submitted 13 March, 2022; v1 submitted 15 November, 2021; originally announced November 2021.

Comments: The Tenth International Conference on Learning Representations (ICLR 2022)

On Provable Benefits of Depth in Training Graph Convolutional Networks

Authors: Weilin Cong, Morteza Ramezani, Mehrdad Mahdavi

Abstract: Graph Convolutional Networks (GCNs) are known to suffer from performance degradation as the number of layers increases, which is usually attributed to over-smoothing. Despite the apparent consensus, we observe that there exists a discrepancy between the theoretical understanding of over-smoothing and the practical capabilities of GCNs. Specifically, we argue that over-smoothing does not necessaril… ▽ More Graph Convolutional Networks (GCNs) are known to suffer from performance degradation as the number of layers increases, which is usually attributed to over-smoothing. Despite the apparent consensus, we observe that there exists a discrepancy between the theoretical understanding of over-smoothing and the practical capabilities of GCNs. Specifically, we argue that over-smoothing does not necessarily happen in practice, a deeper model is provably expressive, can converge to global optimum with linear convergence rate, and achieve very high training accuracy as long as properly trained. Despite being capable of achieving high training accuracy, empirical results show that the deeper models generalize poorly on the testing stage and existing theoretical understanding of such behavior remains elusive. To achieve better understanding, we carefully analyze the generalization capability of GCNs, and show that the training strategies to achieve high training accuracy significantly deteriorate the generalization capability of GCNs. Motivated by these findings, we propose a decoupled structure for GCNs that detaches weight matrices from feature propagation to preserve the expressive power and ensure good generalization performance. We conduct empirical evaluations on various synthetic and real-world datasets to validate the correctness of our theory. △ Less

Submitted 28 October, 2021; originally announced October 2021.

Meta-learning with an Adaptive Task Scheduler

Authors: Huaxiu Yao, Yu Wang, Ying Wei, Peilin Zhao, Mehrdad Mahdavi, Defu Lian, Chelsea Finn

Abstract: To benefit the learning of a new task, meta-learning has been proposed to transfer a well-generalized meta-model learned from various meta-training tasks. Existing meta-learning algorithms randomly sample meta-training tasks with a uniform probability, under the assumption that tasks are of equal importance. However, it is likely that tasks are detrimental with noise or imbalanced given a limited… ▽ More To benefit the learning of a new task, meta-learning has been proposed to transfer a well-generalized meta-model learned from various meta-training tasks. Existing meta-learning algorithms randomly sample meta-training tasks with a uniform probability, under the assumption that tasks are of equal importance. However, it is likely that tasks are detrimental with noise or imbalanced given a limited number of meta-training tasks. To prevent the meta-model from being corrupted by such detrimental tasks or dominated by tasks in the majority, in this paper, we propose an adaptive task scheduler (ATS) for the meta-training process. In ATS, for the first time, we design a neural scheduler to decide which meta-training tasks to use next by predicting the probability being sampled for each candidate task, and train the scheduler to optimize the generalization capacity of the meta-model to unseen tasks. We identify two meta-model-related factors as the input of the neural scheduler, which characterize the difficulty of a candidate task to the meta-model. Theoretically, we show that a scheduler taking the two factors into account improves the meta-training loss and also the optimization landscape. Under the setting of meta-learning with noise and limited budgets, ATS improves the performance on both miniImageNet and a real-world drug discovery benchmark by up to 13% and 18%, respectively, compared to state-of-the-art task schedulers. △ Less

Submitted 26 October, 2021; originally announced October 2021.

Comments: Accepted by NeurIPS 2021

Local SGD Optimizes Overparameterized Neural Networks in Polynomial Time

Authors: Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi

Abstract: In this paper we prove that Local (S)GD (or FedAvg) can optimize deep neural networks with Rectified Linear Unit (ReLU) activation function in polynomial time. Despite the established convergence theory of Local SGD on optimizing general smooth functions in communication-efficient distributed optimization, its convergence on non-smooth ReLU networks still eludes full theoretical understanding. The… ▽ More In this paper we prove that Local (S)GD (or FedAvg) can optimize deep neural networks with Rectified Linear Unit (ReLU) activation function in polynomial time. Despite the established convergence theory of Local SGD on optimizing general smooth functions in communication-efficient distributed optimization, its convergence on non-smooth ReLU networks still eludes full theoretical understanding. The key property used in many Local SGD analysis on smooth function is gradient Lipschitzness, so that the gradient on local models will not drift far away from that on averaged model. However, this decent property does not hold in networks with non-smooth ReLU activation function. We show that, even though ReLU network does not admit gradient Lipschitzness property, the difference between gradients on local models and average model will not change too much, under the dynamics of Local SGD. We validate our theoretical results via extensive experiments. This work is the first to show the convergence of Local SGD on non-smooth functions, and will shed lights on the optimization theory of federated training of deep neural networks. △ Less

Submitted 22 February, 2022; v1 submitted 22 July, 2021; originally announced July 2021.

Image content dependent semi-fragile watermarking with localized tamper detection

Authors: Samira Hosseini, Mojtaba Mahdavi

Abstract: Content-independent watermarks and block-wise independency can be considered as vulnerabilities in semi-fragile watermarking methods. In this paper to achieve the objectives of semi-fragile watermarking techniques, a method is proposed to not have the mentioned shortcomings. In the proposed method, the watermark is generated by relying on image content and a key. Furthermore, the embedding scheme… ▽ More Content-independent watermarks and block-wise independency can be considered as vulnerabilities in semi-fragile watermarking methods. In this paper to achieve the objectives of semi-fragile watermarking techniques, a method is proposed to not have the mentioned shortcomings. In the proposed method, the watermark is generated by relying on image content and a key. Furthermore, the embedding scheme causes the watermarked blocks to become dependent on each other, using a key. In the embedding phase, the image is partitioned into non-overlap** blocks. In order to detect and separate the different types of attacks more precisely, the proposed method embeds three copies of each watermark bit into LWT coefficients of each 4x4 block. In the authentication phase, by voting between the extracted bits the error maps are created; these maps indicate image authenticity and reveal the modified regions. Also, in order to automate the authentication, the images are classified into four categories using seven features. Classification accuracy in the experiments is 97.97 percent. It is noted that our experiments demonstrate that the proposed method is robust against JPEG compression and is competitive with a state-of-the-art semi-fragile watermarking method, in terms of robustness and semi-fragility. △ Less

Submitted 27 June, 2021; originally announced June 2021.

Comments: 32 pages, 11 figures, 5 tables

Pareto Efficient Fairness in Supervised Learning: From Extraction to Tracing

Authors: Mohammad Mahdi Kamani, Rana Forsati, James Z. Wang, Mehrdad Mahdavi

Abstract: As algorithmic decision-making systems are becoming more pervasive, it is crucial to ensure such systems do not become mechanisms of unfair discrimination on the basis of gender, race, ethnicity, religion, etc. Moreover, due to the inherent trade-off between fairness measures and accuracy, it is desirable to learn fairness-enhanced models without significantly compromising the accuracy. In this pa… ▽ More As algorithmic decision-making systems are becoming more pervasive, it is crucial to ensure such systems do not become mechanisms of unfair discrimination on the basis of gender, race, ethnicity, religion, etc. Moreover, due to the inherent trade-off between fairness measures and accuracy, it is desirable to learn fairness-enhanced models without significantly compromising the accuracy. In this paper, we propose Pareto efficient Fairness (PEF) as a suitable fairness notion for supervised learning, that can ensure the optimal trade-off between overall loss and other fairness criteria. The proposed PEF notion is definition-agnostic, meaning that any well-defined notion of fairness can be reduced to the PEF notion. To efficiently find a PEF classifier, we cast the fairness-enhanced classification as a bilevel optimization problem and propose a gradient-based method that can guarantee the solution belongs to the Pareto frontier with provable guarantees for convex and non-convex objectives. We also generalize the proposed algorithmic solution to extract and trace arbitrary solutions from the Pareto frontier for a given preference over accuracy and fairness measures. This approach is generic and can be generalized to any multicriteria optimization problem to trace points on the Pareto frontier curve, which is interesting by its own right. We empirically demonstrate the effectiveness of the PEF solution and the extracted Pareto frontier on real-world datasets compared to state-of-the-art methods. △ Less

Submitted 4 April, 2021; originally announced April 2021.

On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction

Authors: Weilin Cong, Morteza Ramezani, Mehrdad Mahdavi

Abstract: Graph Convolutional Networks (GCNs) have achieved impressive empirical advancement across a wide variety of semi-supervised node classification tasks. Despite their great success, training GCNs on large graphs suffers from computational and memory issues. A potential path to circumvent these obstacles is sampling-based methods, where at each layer a subset of nodes is sampled. Although recent stud… ▽ More Graph Convolutional Networks (GCNs) have achieved impressive empirical advancement across a wide variety of semi-supervised node classification tasks. Despite their great success, training GCNs on large graphs suffers from computational and memory issues. A potential path to circumvent these obstacles is sampling-based methods, where at each layer a subset of nodes is sampled. Although recent studies have empirically demonstrated the effectiveness of sampling-based methods, these works lack theoretical convergence guarantees under realistic settings and cannot fully leverage the information of evolving parameters during optimization. In this paper, we describe and analyze a general doubly variance reduction schema that can accelerate any sampling method under the memory budget. The motivating impetus for the proposed schema is a careful analysis of the variance of sampling methods where it is shown that the induced variance can be decomposed into node embedding approximation variance (zeroth-order variance) during forward propagation and layerwise-gradient variance (first-order variance) during backward propagation. We theoretically analyze the convergence of the proposed schema and show that it enjoys an $\mathcal{O}(1/T)$ convergence rate. We complement our theoretical results by integrating the proposed schema in different sampling methods and applying them to different large real-world graphs. △ Less

Submitted 1 November, 2021; v1 submitted 3 March, 2021; originally announced March 2021.

Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency

Authors: Yuyang Deng, Mehrdad Mahdavi

Abstract: Local SGD is a promising approach to overcome the communication overhead in distributed learning by reducing the synchronization frequency among worker nodes. Despite the recent theoretical advances of local SGD in empirical risk minimization, the efficiency of its counterpart in minimax optimization remains unexplored. Motivated by large scale minimax learning problems, such as adversarial robust… ▽ More Local SGD is a promising approach to overcome the communication overhead in distributed learning by reducing the synchronization frequency among worker nodes. Despite the recent theoretical advances of local SGD in empirical risk minimization, the efficiency of its counterpart in minimax optimization remains unexplored. Motivated by large scale minimax learning problems, such as adversarial robust learning and training generative adversarial networks (GANs), we propose local Stochastic Gradient Descent Ascent (local SGDA), where the primal and dual variables can be trained locally and averaged periodically to significantly reduce the number of communications. We show that local SGDA can provably optimize distributed minimax problems in both homogeneous and heterogeneous data with reduced number of communications and establish convergence rates under strongly-convex-strongly-concave and nonconvex-strongly-concave settings. In addition, we propose a novel variant local SGDA+, to solve nonconvex-nonconcave problems. We give corroborating empirical evidence on different distributed minimax problems. △ Less

Submitted 25 February, 2021; originally announced February 2021.

Comments: This paper has been accepted to AISTATS 2021

Distributionally Robust Federated Averaging

Authors: Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi

Abstract: In this paper, we study communication efficient distributed algorithms for distributionally robust federated learning via periodic averaging with adaptive sampling. In contrast to standard empirical risk minimization, due to the minimax structure of the underlying optimization problem, a key difficulty arises from the fact that the global parameter that controls the mixture of local losses can onl… ▽ More In this paper, we study communication efficient distributed algorithms for distributionally robust federated learning via periodic averaging with adaptive sampling. In contrast to standard empirical risk minimization, due to the minimax structure of the underlying optimization problem, a key difficulty arises from the fact that the global parameter that controls the mixture of local losses can only be updated infrequently on the global stage. To compensate for this, we propose a Distributionally Robust Federated Averaging (DRFA) algorithm that employs a novel snapshotting scheme to approximate the accumulation of history gradients of the mixing parameter. We analyze the convergence rate of DRFA in both convex-linear and nonconvex-linear settings. We also generalize the proposed idea to objectives with regularization on the mixture parameter and propose a proximal variant, dubbed as DRFA-Prox, with provable convergence rates. We also analyze an alternative optimization method for regularized cases in strongly-convex-strongly-concave and non-convex (under PL condition)-strongly-concave settings. To the best of our knowledge, this paper is the first to solve distributionally robust federated learning with reduced communication, and to analyze the efficiency of local descent methods on distributed minimax problems. We give corroborating experimental evidence for our theoretical results in federated learning settings. △ Less

Submitted 24 February, 2021; originally announced February 2021.

Comments: Published in NeurIPS 2020: https://proceedings.neurips.cc/paper/2020/hash/ac450d10e166657ec8f93a1b65ca1b14-Abstract.html

Journal ref: Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, 2020

Communication-efficient k-Means for Edge-based Machine Learning

Authors: Hanlin Lu, Ting He, Shiqiang Wang, Changchang Liu, Mehrdad Mahdavi, Vijaykrishnan Narayanan, Kevin S. Chan, Stephen Pasteris

Abstract: We consider the problem of computing the k-means centers for a large high-dimensional dataset in the context of edge-based machine learning, where data sources offload machine learning computation to nearby edge servers. k-Means computation is fundamental to many data analytics, and the capability of computing provably accurate k-means centers by leveraging the computation power of the edge server… ▽ More We consider the problem of computing the k-means centers for a large high-dimensional dataset in the context of edge-based machine learning, where data sources offload machine learning computation to nearby edge servers. k-Means computation is fundamental to many data analytics, and the capability of computing provably accurate k-means centers by leveraging the computation power of the edge servers, at a low communication and computation cost to the data sources, will greatly improve the performance of these analytics. We propose to let the data sources send small summaries, generated by joint dimensionality reduction (DR), cardinality reduction (CR), and quantization (QT), to support approximate k-means computation at reduced complexity and communication cost. By analyzing the complexity, the communication cost, and the approximation error of k-means algorithms based on carefully designed composition of DR/CR/QT methods, we show that: (i) it is possible to compute near-optimal k-means centers at a near-linear complexity and a constant or logarithmic communication cost, (ii) the order of applying DR and CR significantly affects the complexity and the communication cost, and (iii) combining DR/CR methods with a properly configured quantizer can further reduce the communication cost without compromising the other performance metrics. Our theoretical analysis has been validated through experiments based on real datasets. △ Less

Submitted 21 January, 2022; v1 submitted 8 February, 2021; originally announced February 2021.

Blockchain for steganography: advantages, new algorithms and open challenges

Authors: Omid Torki, Maede Ashouri-Talouki, Mojtaba Mahdavi

Abstract: Steganography is a solution for covert communication and blockchain is a p2p network for data transmission, so the benefits of blockchain can be used in steganography. In this paper, we discuss the advantages of blockchain in steganography, which include the ability to embed hidden data without manual change in the original data, as well as the readiness of the blockchain platform for data transmi… ▽ More Steganography is a solution for covert communication and blockchain is a p2p network for data transmission, so the benefits of blockchain can be used in steganography. In this paper, we discuss the advantages of blockchain in steganography, which include the ability to embed hidden data without manual change in the original data, as well as the readiness of the blockchain platform for data transmission and storage, which eliminates the need for the Steganographer to design and implement a new platform for data transmission and storage. We have proposed two algorithms for steganography in blockchain, the first one is a high-capacity algorithm for the key and the steganography algorithm exchange and switching, and the second one is a medium-capacity algorithm for embedding hidden data. Also, by reviewing the previous three steganography schemes in blockchain, we have examined their drawback and have showed that none of them are practical schemes for steganography in blockchain. Then, we have explained the challenges of steganography in blockchain from the steganographers and steganalyzers point of view. △ Less

Submitted 8 January, 2021; originally announced January 2021.

Online Structured Meta-learning

Authors: Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi, Zhenhui Li, Richard Socher, Caiming Xiong

Abstract: Learning quickly is of great importance for machine intelligence deployed in online platforms. With the capability of transferring knowledge from learned tasks, meta-learning has shown its effectiveness in online scenarios by continuously updating the model with the learned prior. However, current online meta-learning algorithms are limited to learn a globally-shared meta-learner, which may lead t… ▽ More Learning quickly is of great importance for machine intelligence deployed in online platforms. With the capability of transferring knowledge from learned tasks, meta-learning has shown its effectiveness in online scenarios by continuously updating the model with the learned prior. However, current online meta-learning algorithms are limited to learn a globally-shared meta-learner, which may lead to sub-optimal results when the tasks contain heterogeneous information that are distinct by nature and difficult to share. We overcome this limitation by proposing an online structured meta-learning (OSML) framework. Inspired by the knowledge organization of human and hierarchical feature representation, OSML explicitly disentangles the meta-learner as a meta-hierarchical graph with different knowledge blocks. When a new task is encountered, it constructs a meta-knowledge pathway by either utilizing the most relevant knowledge blocks or exploring new blocks. Through the meta-knowledge pathway, the model is able to quickly adapt to the new task. In addition, new knowledge is further incorporated into the selected blocks. Experiments on three datasets demonstrate the effectiveness and interpretability of our proposed framework in the context of both homogeneous and heterogeneous tasks. △ Less

Submitted 22 October, 2020; originally announced October 2020.

Comments: Accepted by NeurIPS 2020

Inductive Reachability Witnesses

Authors: Ali Asadi, Krishnendu Chatterjee, Hongfei Fu, Amir Kafshdar Goharshady, Mohammad Mahdavi

Abstract: In this work, we consider the fundamental problem of reachability analysis over imperative programs with real variables. The reachability property requires that a program can reach certain target states during its execution. Previous works that tackle reachability analysis are either unable to handle programs consisting of general loops (e.g. symbolic execution), or lack completeness guarantees (e… ▽ More In this work, we consider the fundamental problem of reachability analysis over imperative programs with real variables. The reachability property requires that a program can reach certain target states during its execution. Previous works that tackle reachability analysis are either unable to handle programs consisting of general loops (e.g. symbolic execution), or lack completeness guarantees (e.g. abstract interpretation), or are not automated (e.g. incorrectness logic/reverse Hoare logic). In contrast, we propose a novel approach for reachability analysis that can handle general programs, is (semi-)complete, and can be entirely automated for a wide family of programs. Our approach extends techniques from both invariant generation and ranking-function synthesis to reachability analysis through the notion of (Universal) Inductive Reachability Witnesses (IRWs/UIRWs). While traditional invariant generation uses over-approximations of reachable states, we consider the natural dual problem of under-approximating the set of program states that can reach a target state. We then apply an argument similar to ranking functions to ensure that all states in our under-approximation can indeed reach the target set in finitely many steps. △ Less

Submitted 28 July, 2020; originally announced July 2020.

Full Quaternion Representation of Color images: A Case Study on QSVD-based Color Image Compression

Authors: Alireza Parchami, Mojtaba Mahdavi

Abstract: For many years, channels of a color image have been processed individually, or the image has been converted to grayscale one with respect to color image processing. Pure quaternion representation of color images solves this issue as it allows images to be processed in a holistic space. Nevertheless, it brings additional costs due to the extra fourth dimension. In this paper, we propose an approach… ▽ More For many years, channels of a color image have been processed individually, or the image has been converted to grayscale one with respect to color image processing. Pure quaternion representation of color images solves this issue as it allows images to be processed in a holistic space. Nevertheless, it brings additional costs due to the extra fourth dimension. In this paper, we propose an approach for representing color images with full quaternion numbers that enables us to process color images holistically without additional cost in time, space and computation. With taking auto- and cross-correlation of color channels into account, an autoencoder neural network is used to generate a global model for transforming a color image into a full quaternion matrix. To evaluate the model, we use UCID dataset, and the results indicate that the model has an acceptable performance on color images. Moreover, we propose a compression method based on the generated model and QSVD as a case study. The method is compared with the same compression method using pure quaternion representation and is assessed with UCID dataset. The results demonstrate that the compression method using the proposed full quaternion representation fares better than the other in terms of time, quality, and size of compressed files. △ Less

Submitted 19 July, 2020; originally announced July 2020.

Comments: 15 pages, 16 figures, 1 table, submitted to Signal Processing journal

Federated Learning with Compression: Unified Analysis and Sharp Guarantees

Authors: Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, Mehrdad Mahdavi

Abstract: In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions. Two notable trends to deal with the communication overhead of federated algorithms are gradient compression and local computation… ▽ More In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions. Two notable trends to deal with the communication overhead of federated algorithms are gradient compression and local computation with periodic communication. Despite many attempts, characterizing the relationship between these two approaches has proven elusive. We address this by proposing a set of algorithms with periodical compressed (quantized or sparsified) communication and analyze their convergence properties in both homogeneous and heterogeneous local data distribution settings. For the homogeneous setting, our analysis improves existing bounds by providing tighter convergence rates for both strongly convex and non-convex objective functions. To mitigate data heterogeneity, we introduce a local gradient tracking scheme and obtain sharp convergence rates that match the best-known communication complexities without compression for convex, strongly convex, and nonconvex settings. We complement our theoretical results and demonstrate the effectiveness of our proposed methods by several experiments on real-world datasets. △ Less

Submitted 20 November, 2020; v1 submitted 2 July, 2020; originally announced July 2020.

Comments: version 2. more experiments and comparisons

Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks

Authors: Weilin Cong, Rana Forsati, Mahmut Kandemir, Mehrdad Mahdavi

Abstract: Sampling methods (e.g., node-wise, layer-wise, or subgraph) has become an indispensable strategy to speed up training large-scale Graph Neural Networks (GNNs). However, existing sampling methods are mostly based on the graph structural information and ignore the dynamicity of optimization, which leads to high variance in estimating the stochastic gradients. The high variance issue can be very pron… ▽ More Sampling methods (e.g., node-wise, layer-wise, or subgraph) has become an indispensable strategy to speed up training large-scale Graph Neural Networks (GNNs). However, existing sampling methods are mostly based on the graph structural information and ignore the dynamicity of optimization, which leads to high variance in estimating the stochastic gradients. The high variance issue can be very pronounced in extremely large graphs, where it results in slow convergence and poor generalization. In this paper, we theoretically analyze the variance of sampling methods and show that, due to the composite structure of empirical risk, the variance of any sampling method can be decomposed into \textit{embedding approximation variance} in the forward stage and \textit{stochastic gradient variance} in the backward stage that necessities mitigating both types of variance to obtain faster convergence rate. We propose a decoupled variance reduction strategy that employs (approximate) gradient information to adaptively sample nodes with minimal variance, and explicitly reduces the variance introduced by embedding approximation. We show theoretically and empirically that the proposed method, even with smaller mini-batch sizes, enjoys a faster convergence rate and entails a better generalization compared to the existing methods. △ Less

Submitted 5 September, 2021; v1 submitted 24 June, 2020; originally announced June 2020.

The Effect of Coupling Memory and Block Length on Spatially Coupled Serially Concatenated Codes

Authors: Mojtaba Mahdavi, Muhammad Umar Farooq, Liang Liu, Ove Edfors, Viktor Öwall, Michael Lentmaier

Abstract: Spatially coupled serially concatenated codes (SC-SCCs) are a class of spatially coupled turbo-like codes, which have a close-to-capacity performance and low error floor. In this paper we investigate the impact of coupling memory, block length, decoding window size, and number of iterations on the performance, complexity, and latency of SC-SCCs. Several design tradeoffs are presented to see the re… ▽ More Spatially coupled serially concatenated codes (SC-SCCs) are a class of spatially coupled turbo-like codes, which have a close-to-capacity performance and low error floor. In this paper we investigate the impact of coupling memory, block length, decoding window size, and number of iterations on the performance, complexity, and latency of SC-SCCs. Several design tradeoffs are presented to see the relation between these parameters in a wide range. Also, our analysis provides design guidelines for SC-SCCs in different scenarios to make the code design independent of block length. As a result, block length and coupling memory can be exchanged flexibly without changing the latency and complexity. Also, we observe that the performance of SC-SCCs is improved with respect to the uncoupled ensembles for a fixed latency and complexity. △ Less

Submitted 25 July, 2021; v1 submitted 23 June, 2020; originally announced June 2020.

Comments: Presented at the IEEE 93rd Vehicular Technology Conference (VTC) 2021-Spring

Adaptive Personalized Federated Learning

Authors: Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi

Abstract: Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. We derive the… ▽ More Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. We derive the generalization bound of mixture of local and global models, and find the optimal mixing parameter. We also propose a communication-efficient optimization method to collaboratively learn the personalized models and analyze its convergence in both smooth strongly convex and nonconvex settings. The extensive experiments demonstrate the effectiveness of our personalization schema, as well as the correctness of established generalization theories. △ Less

Submitted 5 November, 2020; v1 submitted 30 March, 2020; originally announced March 2020.

Comments: [v3] Added convergence analysis for nonconvex losses and additional experiments along with new baselines [v2] A new generalization analysis is provided. Also, additional experiments are added

ScanSSD: Scanning Single Shot Detector for Mathematical Formulas in PDF Document Images

Authors: Parag Mali, Puneeth Kukkadapu, Mahshad Mahdavi, Richard Zanibbi

Abstract: We introduce the Scanning Single Shot Detector (ScanSSD) for locating math formulas offset from text and embedded in textlines. ScanSSD uses only visual features for detection: no formatting or typesetting information such as layout, font, or character labels are employed. Given a 600 dpi document page image, a Single Shot Detector (SSD) locates formulas at multiple scales using sliding windows, a… ▽ More We introduce the Scanning Single Shot Detector (ScanSSD) for locating math formulas offset from text and embedded in textlines. ScanSSD uses only visual features for detection: no formatting or typesetting information such as layout, font, or character labels are employed. Given a 600 dpi document page image, a Single Shot Detector (SSD) locates formulas at multiple scales using sliding windows, after which candidate detections are pooled to obtain page-level results. For our experiments we use the TFD-ICDAR2019v2 dataset, a modification of the GTDB scanned math article collection. ScanSSD detects characters in formulas with high accuracy, obtaining a 0.926 f-score, and detects formulas with high recall overall. Detection errors are largely minor, such as splitting formulas at large whitespace gaps (e.g., for variable constraints) and merging formulas on adjacent textlines. Formula detection f-scores of 0.796 (IOU $\geq0.5$) and 0.733 (IOU $\ge 0.75$) are obtained. Our data, evaluation tools, and code are publicly available. △ Less

Submitted 17 March, 2020; originally announced March 2020.

Comments: 8 pages, 7 figures

Manipulation and exchange of Light with Orbital Angular Momentum in Quantum Dot Molecules

Authors: Mahboubeh Mahdavi, Zahra Amini Sabegh, Mohammad Mohammadi, Hamid Reza Hamedi, Mohammad Mahmoudi

Abstract: We study the interaction of laser pulses carrying orbital angular momentum (OAM) with structural asymmetry quantum dot molecules characterized by four energy levels. We demonstrate how the inter-dot tunneling endows exchange of optical vortices between different frequencies. We consider a case where a weak probe beam has an optical vortex and thus has a zero intensity at the center. The presence o… ▽ More We study the interaction of laser pulses carrying orbital angular momentum (OAM) with structural asymmetry quantum dot molecules characterized by four energy levels. We demonstrate how the inter-dot tunneling endows exchange of optical vortices between different frequencies. We consider a case where a weak probe beam has an optical vortex and thus has a zero intensity at the center. The presence of tunneling coupling generates an additional weak laser beam with the same vorticity as that of the incident vortex beam. We analyze conditions for the vortex of the initial beam to be transferred efficiently to the generated beam. The propagation of Laguerre-Gaussian (LG) beams possessing OAM states characterized by both azimuthal and radial indices is then investigated for the case where the strong control beam is also an OAM mode. It is shown that the conservation of OAM states is always satisfied over the OAM exchange process. Yet, an abnormal case is observed in which the radial index induces some intensity patterns of the generated beam which differs from a pure LG beam of incident beams. Analytical solutions are provided to elucidate such effects induced by radial indices on propagation characteristics of OAM beams. When superimposing two initially present weak OAM modes, it is observed that the resulting optical vortices move about the beam axis as the light propagates, forming a sort of "constellation" around the center. The shift in axis of such a composite pulses is due to the effect of inter-dot tunneling which is controlled by an external electric voltage. The optical angular momenta may add a new degree of freedom in the study of solid systems suitable for quantum technologies. △ Less

Submitted 17 February, 2020; originally announced February 2020.

Journal ref: Phys. Rev. A 101, 063811 (2020)

Analysis of D2D Communication with RF Energy Harvesting and Interference Management

Authors: Nasrin Razmi, Mehdi Mahdavi, Mohammadali Mohammadi, Petar Popovski

Abstract: Device-to-device (D2D) underlaid cellular network, enabled with radio frequency energy harvesting (RFEH), and enhanced interference management schemes is a promising candidate to improve spectral and energy efficiency of next generation wireless networks. In this paper, we propose a time division duplexing (TDD)-based protocol, in which allows the devices to harvest energy from the downlink transm… ▽ More Device-to-device (D2D) underlaid cellular network, enabled with radio frequency energy harvesting (RFEH), and enhanced interference management schemes is a promising candidate to improve spectral and energy efficiency of next generation wireless networks. In this paper, we propose a time division duplexing (TDD)-based protocol, in which allows the devices to harvest energy from the downlink transmissions of the base station, while controlling the interference among D2D and cellular communication in the uplink. We propose two schemes for transmission coordination, based on fixed transmission probability (FTP) and adaptive transmission probability (ATP), respectively. In FTP, the D2D transmitters that have harvested enough energy can initiate data transmission with a fixed probability. Differently from this, in ATP a device utilizes its sensing capability to get improved coordination and interference control among the transmitting devices. We evaluate the network performance by presenting an accurate energy model and leveraging tools from stochastic geometry. The results on outage probability and D2D sum-rate reveal the importance of transmission coordination on network performance. These observations led to a solution for choosing the parameters of the ATP scheme that achieves an optimal tradeoff between the D2D outage probability and number of transmitting users. △ Less

Submitted 5 February, 2020; originally announced February 2020.

Pointwise Attention-Based Atrous Convolutional Neural Networks

Authors: Mobina Mahdavi, Fahimeh Fooladgar, Shohreh Kasaei

Abstract: With the rapid progress of deep convolutional neural networks, in almost all robotic applications, the availability of 3D point clouds improves the accuracy of 3D semantic segmentation methods. Rendering of these irregular, unstructured, and unordered 3D points to 2D images from multiple viewpoints imposes some issues such as loss of information due to 3D to 2D projection, discretizing artifacts,… ▽ More With the rapid progress of deep convolutional neural networks, in almost all robotic applications, the availability of 3D point clouds improves the accuracy of 3D semantic segmentation methods. Rendering of these irregular, unstructured, and unordered 3D points to 2D images from multiple viewpoints imposes some issues such as loss of information due to 3D to 2D projection, discretizing artifacts, and high computational costs. To efficiently deal with a large number of points and incorporate more context of each point, a pointwise attention-based atrous convolutional neural network architecture is proposed. It focuses on salient 3D feature points among all feature maps while considering outstanding contextual information via spatial channel-wise attention modules. The proposed model has been evaluated on the two most important 3D point cloud datasets for the 3D semantic segmentation task. It achieves a reasonable performance compared to state-of-the-art models in terms of accuracy, with a much smaller number of parameters. △ Less

Submitted 27 December, 2019; originally announced December 2019.

Comments: 7 pages, 6 figures. Author one and author two contributed equally

Efficient Fair Principal Component Analysis

Authors: Mohammad Mahdi Kamani, Farzin Haddadpour, Rana Forsati, Mehrdad Mahdavi

Abstract: It has been shown that dimension reduction methods such as PCA may be inherently prone to unfairness and treat data from different sensitive groups such as race, color, sex, etc., unfairly. In pursuit of fairness-enhancing dimensionality reduction, using the notion of Pareto optimality, we propose an adaptive first-order algorithm to learn a subspace that preserves fairness, while slightly comprom… ▽ More It has been shown that dimension reduction methods such as PCA may be inherently prone to unfairness and treat data from different sensitive groups such as race, color, sex, etc., unfairly. In pursuit of fairness-enhancing dimensionality reduction, using the notion of Pareto optimality, we propose an adaptive first-order algorithm to learn a subspace that preserves fairness, while slightly compromising the reconstruction loss. Theoretically, we provide sufficient conditions that the solution of the proposed algorithm belongs to the Pareto frontier for all sensitive groups; thereby, the optimal trade-off between overall reconstruction loss and fairness constraints is guaranteed. We also provide the convergence analysis of our algorithm and show its efficacy through empirical studies on different datasets, which demonstrates superior performance in comparison with state-of-the-art algorithms. The proposed fairness-aware PCA algorithm can be efficiently generalized to multiple group sensitive features and effectively reduce the unfairness decisions in downstream tasks such as classification. △ Less

Submitted 7 March, 2020; v1 submitted 12 November, 2019; originally announced November 2019.

On the Convergence of Local Descent Methods in Federated Learning

Authors: Farzin Haddadpour, Mehrdad Mahdavi

Abstract: In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., heterogeneous or non i.i.d. data samples). In this paper, we generalize the local stochastic and full gradient descent with periodic averaging-- originally designed for homogeneous dis… ▽ More In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., heterogeneous or non i.i.d. data samples). In this paper, we generalize the local stochastic and full gradient descent with periodic averaging-- originally designed for homogeneous distributed optimization, to solve nonconvex optimization problems in federated learning. Although scant research is available on the effectiveness of local SGD in reducing the number of communication rounds in homogeneous setting, its convergence and communication complexity in heterogeneous setting is mostly demonstrated empirically and lacks through theoretical understating. To bridge this gap, we demonstrate that by properly analyzing the effect of unbiased gradients and sampling schema in federated setting, under mild assumptions, the implicit variance reduction feature of local distributed methods generalize to heterogeneous data shards and exhibits the best known convergence rates of homogeneous setting both in general nonconvex and under {\pl}~ condition (generalization of strong-convexity). Our theoretical results complement the recent empirical studies that demonstrate the applicability of local GD/SGD to federated learning. We also specialize the proposed local method for networked distributed optimization. To the best of our knowledge, the obtained convergence rates are the sharpest known to date on the convergence of local decant methods with periodic averaging for solving nonconvex federated optimization in both centralized and networked distributed optimization. △ Less

Submitted 6 December, 2019; v1 submitted 31 October, 2019; originally announced October 2019.

Comments: 47 pages, "Updates from v1: A technical error in Lemma B3 is corrected"

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization

Authors: Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, Viveck R. Cadambe

Abstract: Communication overhead is one of the key challenges that hinders the scalability of distributed optimization algorithms. In this paper, we study local distributed SGD, where data is partitioned among computation nodes, and the computation nodes perform local updates with periodically exchanging the model among the workers to perform averaging. While local SGD is empirically shown to provide promis… ▽ More Communication overhead is one of the key challenges that hinders the scalability of distributed optimization algorithms. In this paper, we study local distributed SGD, where data is partitioned among computation nodes, and the computation nodes perform local updates with periodically exchanging the model among the workers to perform averaging. While local SGD is empirically shown to provide promising results, a theoretical understanding of its performance remains open. We strengthen convergence analysis for local SGD, and show that local SGD can be far less expensive and applied far more generally than current theory suggests. Specifically, we show that for loss functions that satisfy the Polyak-Łojasiewicz condition, $O((pT)^{1/3})$ rounds of communication suffice to achieve a linear speed up, that is, an error of $O(1/pT)$, where $T$ is the total number of model updates at each worker. This is in contrast with previous work which required higher number of communication rounds, as well as was limited to strongly convex loss functions, for a similar asymptotic performance. We also develop an adaptive synchronization scheme that provides a general condition for linear speed up. Finally, we validate the theory with experimental results, running over AWS EC2 clouds and an internal GPU cluster. △ Less

Submitted 14 May, 2020; v1 submitted 29 October, 2019; originally announced October 2019.

Comments: Paper accepted to NeurIPS 2019 - We fixed a flaw in the earlier version regarding the dependency on constants but this change does not affect the communication complexity

ED2: Two-stage Active Learning for Error Detection -- Technical Report

Authors: Felix Neutatz, Mohammad Mahdavi, Ziawasch Abedjan

Abstract: Traditional error detection approaches require user-defined parameters and rules. Thus, the user has to know both the error detection system and the data. However, we can also formulate error detection as a semi-supervised classification problem that only requires domain expertise. The challenges for such an approach are twofold: (1) to represent the data in a way that enables a classification mod… ▽ More Traditional error detection approaches require user-defined parameters and rules. Thus, the user has to know both the error detection system and the data. However, we can also formulate error detection as a semi-supervised classification problem that only requires domain expertise. The challenges for such an approach are twofold: (1) to represent the data in a way that enables a classification model to identify various kinds of data errors, and (2) to pick the most promising data values for learning. In this paper, we address these challenges with ED2, our new example-driven error detection method. First, we present a new two-dimensional multi-classifier sampling strategy for active learning. Second, we propose novel multi-column features. The combined application of these techniques provides fast convergence of the classification task with high detection accuracy. On several real-world datasets, ED2 requires, on average, less than 1% labels to outperform existing error detection approaches. This report extends the peer-reviewed paper "ED2: A Case for Active Learning in Error Detection". All source code related to this project is available on GitHub. △ Less

Submitted 17 August, 2019; originally announced August 2019.

Copper-catalyzed efficient synthesis of 5-arylindazolo[3,2-b]quinazolin-7(5H)-ones from 2-nitrobenzaldehydes

Authors: Zahra ArastehFard, Karim Akbari Dilmaghani, Mehdi Soheilizad, Mohammad Mahdavi

Abstract: A novel and practical copper-catalyzed approach is developed for the preparation of 5-arylindazolo[3,2-b]quinazolin-7(5H)-ones. The 2-amino-N'-arylbenzohydrazide, which easily prepared by reaction of isatoic anhydride with arylhydrazine, through a condensation/intramolecular cyclization reacted by 2-nitrobenzaldehydes in the present of CuI to afford corresponding 5-arylindazolo[3,2-b]quinazolin-7(… ▽ More A novel and practical copper-catalyzed approach is developed for the preparation of 5-arylindazolo[3,2-b]quinazolin-7(5H)-ones. The 2-amino-N'-arylbenzohydrazide, which easily prepared by reaction of isatoic anhydride with arylhydrazine, through a condensation/intramolecular cyclization reacted by 2-nitrobenzaldehydes in the present of CuI to afford corresponding 5-arylindazolo[3,2-b]quinazolin-7(5H)-ones in good yields. △ Less

Submitted 15 December, 2017; originally announced December 2017.

Comments: 5 pages, 3 schemes, 2 tables

Temperature Equilibration Rate of Quasi-Monoenergetic Deuteron Beam in a Fusion Plasmas

Authors: M. Mahdavi, R. Azadifar, T. Khoorokhi

Abstract: Thermal equilibrium rate can play an important role in the energy deposition of beam to the fuel in fast ignition due to high temperature difference between projectile ions and background plasma ions. In this study the temperature equilibration rate of a quasi-monoenergetic deuteron beam with an equimolar Deuterium-Tritium fusion plasma with a Maxwellian energy distribution is calculated by kineti… ▽ More Thermal equilibrium rate can play an important role in the energy deposition of beam to the fuel in fast ignition due to high temperature difference between projectile ions and background plasma ions. In this study the temperature equilibration rate of a quasi-monoenergetic deuteron beam with an equimolar Deuterium-Tritium fusion plasma with a Maxwellian energy distribution is calculated by kinetic theory equations. In this theory, binary collisions is described by the Boltzman equation and collective effects is described by the Lenard-Balescu equation. The obtained results show that at higher background temperatures, $T_{b}=100 keV$, the ions interactions effect in the temperature equilibration rate increases because the deuteron beam exchanges most of its energy with ions plasma △ Less

Submitted 13 December, 2017; originally announced December 2017.

The electromagnetic instabilities propagation in weak relativistic quantum plasmas

Authors: M. Mahdavi, H. Khanzadeh

Abstract: The electromagnetic instabilities excited by the temperature anisotropy have been always one of the interesting issues in real high-density physical systems, where the relativistic and quantum effects due to spin can be important. This paper discusses the case where plasma is not strongly coupled but is still in regimes where a classic plasma description is not fully adequate. The length scale of… ▽ More The electromagnetic instabilities excited by the temperature anisotropy have been always one of the interesting issues in real high-density physical systems, where the relativistic and quantum effects due to spin can be important. This paper discusses the case where plasma is not strongly coupled but is still in regimes where a classic plasma description is not fully adequate. The length scale of the plasma can be larger than the de-Broglie length so that the quantum effects relevance to the spin can be significant. In addition, the only relativistic effects are due to the electrons spin, for example the spin-orbit coupling effect (the weak (semi) relativistic effects). Obtained results imply that these effects can not be important in the ICF subjects while these can lead to significant results in the astrophysical subjects because of the strong magnetic fields. It is found that the weakly relativistic effects can cover the magnetic dipole force and the spin precession effects so that the growth rate of the instability can increase compared to the non relativistic spin polarized cases. Indeed, it is expected that the probability of electron capture in the background magnetic fields and the particle energy dissipation will be reduced so that there will be a high portion of free energy in the system. △ Less

Submitted 20 November, 2017; originally announced November 2017.

Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression

Authors: Samet Oymak, Mehrdad Mahdavi, Jiasi Chen

Abstract: For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred over linear models such as Lasso. This work proposes learning the feature nonlinearities by binning feature values and finding the best fit in each quantile using non-convex regulari… ▽ More For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred over linear models such as Lasso. This work proposes learning the feature nonlinearities by binning feature values and finding the best fit in each quantile using non-convex regularized linear regression. The algorithm first captures the dependence between neighboring quantiles by enforcing smoothness via piecewise-constant/linear approximation and then selects a sparse subset of good features. We prove that the proposed algorithm is statistically and computationally efficient. In particular, it achieves linear rate of convergence while requiring near-minimal number of samples. Evaluations on synthetic and real datasets demonstrate that algorithm is competitive with current state-of-the-art and accurately learns feature nonlinearities. Finally, we explore an interesting connection between the binning stage of our algorithm and sparse Johnson-Lindenstrauss matrices. △ Less

Submitted 19 May, 2017; originally announced May 2017.

Comments: 22 pages, 7 figures

Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Data

Authors: Jialei Wang, Jason D. Lee, Mehrdad Mahdavi, Mladen Kolar, Nathan Srebro

Abstract: Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessian sketch is an optimization process with preconditioning, and develop accelerate… ▽ More Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessian sketch is an optimization process with preconditioning, and develop accelerated iterative Hessian sketch via the searching the conjugate direction; we then establish primal-dual connections between the Hessian sketch and dual random projection, and apply the preconditioned conjugate gradient approach on the dual problem, which leads to the accelerated iterative dual random projection methods. Finally to tackle the challenges from both large sample size and high-dimensionality, we propose the primal-dual sketch, which iteratively sketches the primal and dual formulations. We show that using a logarithmic number of calls to solvers of small scale problem, primal-dual sketch is able to recover the optimum of the original problem up to arbitrary precision. The proposed algorithms are validated via extensive experiments on synthetic and real data sets which complements our theoretical results. △ Less

Submitted 10 October, 2016; originally announced October 2016.

QCD analysis of nucleon structure functions in deep-inelastic neutrino-nucleon scattering: Laplace transform and Jacobi polynomials approach

Authors: S. Mohammad Moosavi Nejad, Hamzeh Khanpour, S. Atashbar Tehrani, Mahdi Mahdavi

Abstract: We present a detailed QCD analysis of nucleon structure functions $xF_3 (x, Q^2)$, based on Laplace transforms and Jacobi polynomials approach. The analysis corresponds to the next-to-leading order and next-to-next-to-leading order approximation of perturbative QCD. The Laplace transform technique, as an exact analytical solution, is used for the solution of nonsinglet Dokshitzer-Gribov-Lipatov-Al… ▽ More We present a detailed QCD analysis of nucleon structure functions $xF_3 (x, Q^2)$, based on Laplace transforms and Jacobi polynomials approach. The analysis corresponds to the next-to-leading order and next-to-next-to-leading order approximation of perturbative QCD. The Laplace transform technique, as an exact analytical solution, is used for the solution of nonsinglet Dokshitzer-Gribov-Lipatov-Altarelli-Parisi evolution equations at low- and large-$x$ values. The extracted results are used as input to obtain the $x$ and Q$^2$ evolution of $xF_3(x, Q^2)$ structure functions using the Jacobi polynomials approach. In our work, the values of the typical QCD scale $Λ_{\overline{\rm MS}}^{(n_f)}$ and the strong coupling constant $α_s(M_Z^2)$ are determined for four quark flavors ($n_f=4$) as well. A careful estimation of the uncertainties shall be performed using the Hessian method for the valence-quark distributions, originating from the experimental errors. We compare our valence-quark parton distribution functions sets with those of other collaborations; in particular with the {\tt BBG}, {\tt CT14}, {\tt MMHT14} and {\tt NNPDF} sets, which are contemporary with the present analysis. The obtained results from the analysis are in good agreement with those from the literature. △ Less

Submitted 4 October, 2016; v1 submitted 17 September, 2016; originally announced September 2016.

Comments: 14 Pages, 8 Figures, 4 Tables

Journal ref: Phys. Rev. C 94, 045201 (2016)