-
AI Foundation Models for Weather and Climate: Applications, Design, and Implementation
Authors:
S. Karthik Mukkavilli,
Daniel Salles Civitarese,
Johannes Schmude,
Johannes Jakubik,
Anne Jones,
Nam Nguyen,
Christopher Phillips,
Sujit Roy,
Shraddha Singh,
Campbell Watson,
Raghu Ganti,
Hendrik Hamann,
Udaysankar Nair,
Rahul Ramachandran,
Kommy Weldemariam
Abstract:
Machine learning and deep learning methods have been widely explored in understanding the chaotic behavior of the atmosphere and furthering weather forecasting. There has been increasing interest from technology companies, government institutions, and meteorological agencies in building digital twins of the Earth. Recent approaches using transformers, physics-informed machine learning, and graph n…
▽ More
Machine learning and deep learning methods have been widely explored in understanding the chaotic behavior of the atmosphere and furthering weather forecasting. There has been increasing interest from technology companies, government institutions, and meteorological agencies in building digital twins of the Earth. Recent approaches using transformers, physics-informed machine learning, and graph neural networks have demonstrated state-of-the-art performance on relatively narrow spatiotemporal scales and specific tasks. With the recent success of generative artificial intelligence (AI) using pre-trained transformers for language modeling and vision with prompt engineering and fine-tuning, we are now moving towards generalizable AI. In particular, we are witnessing the rise of AI foundation models that can perform competitively on multiple domain-specific downstream tasks. Despite this progress, we are still in the nascent stages of a generalizable AI model for global Earth system models, regional climate models, and mesoscale weather models. Here, we review current state-of-the-art AI approaches, primarily from transformer and operator learning literature in the context of meteorology. We provide our perspective on criteria for success towards a family of foundation models for nowcasting and forecasting weather and climate predictions. We also discuss how such models can perform competitively on downstream tasks such as downscaling (super-resolution), identifying conditions conducive to the occurrence of wildfires, and predicting consequential meteorological phenomena across various spatiotemporal scales such as hurricanes and atmospheric rivers. In particular, we examine current AI methodologies and contend they have matured enough to design and implement a weather foundation model.
△ Less
Submitted 19 September, 2023; v1 submitted 19 September, 2023;
originally announced September 2023.
-
Estimation of Appearance and Occupancy Information in Birds Eye View from Surround Monocular Images
Authors:
Sarthak Sharma,
Unnikrishnan R. Nair,
Udit Singh Parihar,
Midhun Menon S,
Srikanth Vidapanakal
Abstract:
Autonomous driving requires efficient reasoning about the location and appearance of the different agents in the scene, which aids in downstream tasks such as object detection, object tracking, and path planning. The past few years have witnessed a surge in approaches that combine the different taskbased modules of the classic self-driving stack into an End-toEnd(E2E) trainable learning system. Th…
▽ More
Autonomous driving requires efficient reasoning about the location and appearance of the different agents in the scene, which aids in downstream tasks such as object detection, object tracking, and path planning. The past few years have witnessed a surge in approaches that combine the different taskbased modules of the classic self-driving stack into an End-toEnd(E2E) trainable learning system. These approaches replace perception, prediction, and sensor fusion modules with a single contiguous module with shared latent space embedding, from which one extracts a human-interpretable representation of the scene. One of the most popular representations is the Birds-eye View (BEV), which expresses the location of different traffic participants in the ego vehicle frame from a top-down view. However, a BEV does not capture the chromatic appearance information of the participants. To overcome this limitation, we propose a novel representation that captures various traffic participants appearance and occupancy information from an array of monocular cameras covering 360 deg field of view (FOV). We use a learned image embedding of all camera images to generate a BEV of the scene at any instant that captures both appearance and occupancy of the scene, which can aid in downstream tasks such as object tracking and executing language-based commands. We test the efficacy of our approach on synthetic dataset generated from CARLA. The code, data set, and results can be found at https://rebrand.ly/APP OCC-results.
△ Less
Submitted 8 November, 2022;
originally announced November 2022.
-
Test of Transitivity in Quantum Field theory using Rindler spacetime
Authors:
Sashideep Gutti,
Akhil U Nair,
Prasant Samantray
Abstract:
We consider a massless scalar field in Minkowski spacetime $\cal{M} $ in its vacuum state, and consider two Rindler wedges $R_1$ and $R_2$ in this space. $R_2$ is shifted to the right of $R_1$ by a distance $Δ$. We therefore have $R_2\subset R_1 \subset \cal{M}$ with the symbol $\subset$ implying a quantum subsystem. We find the reduced state in $R_2$ using two independent ways: a) by evaluation o…
▽ More
We consider a massless scalar field in Minkowski spacetime $\cal{M} $ in its vacuum state, and consider two Rindler wedges $R_1$ and $R_2$ in this space. $R_2$ is shifted to the right of $R_1$ by a distance $Δ$. We therefore have $R_2\subset R_1 \subset \cal{M}$ with the symbol $\subset$ implying a quantum subsystem. We find the reduced state in $R_2$ using two independent ways: a) by evaluation of the reduced state from vacuum state in $\cal{M}$ which yields a thermal density matrix, b) by first evaluating the reduced state in $R_1$ from $\cal{M} $ yielding a thermal state in $R_1$, and subsequently evaluate the reduced state in $R_2$ in that order of sequence. In this article we attempt to address the question whether both these independent ways yield the same reduced state in $R_2$. To that end, we devise a method which involves cleaving the Rindler wedge $R_1$ into two domains such that they form a thermofield double. One of the domains aligns itself along the wedge $R_2$ while the other is a diamond shaped construction between the boundaries of $R_1$ and $R_2$. We conclude that both these independent methods yield two different answers, and discuss the possible implications of our result in the context of quantum states outside a non-extremal black hole formed by collapsing matter.
△ Less
Submitted 25 October, 2022; v1 submitted 17 October, 2022;
originally announced October 2022.
-
Bridging Sim2Real Gap Using Image Gradients for the Task of End-to-End Autonomous Driving
Authors:
Unnikrishnan R Nair,
Sarthak Sharma,
Udit Singh Parihar,
Midhun S Menon,
Srikanth Vidapanakal
Abstract:
We present the first prize solution to NeurIPS 2021 - AWS Deepracer Challenge. In this competition, the task was to train a reinforcement learning agent (i.e. an autonomous car), that learns to drive by interacting with its environment, a simulated track, by taking an action in a given state to maximize the expected reward. This model was then tested on a real-world track with a miniature AWS Deep…
▽ More
We present the first prize solution to NeurIPS 2021 - AWS Deepracer Challenge. In this competition, the task was to train a reinforcement learning agent (i.e. an autonomous car), that learns to drive by interacting with its environment, a simulated track, by taking an action in a given state to maximize the expected reward. This model was then tested on a real-world track with a miniature AWS Deepracer car. Our goal is to train a model that can complete a lap as fast as possible without going off the track. The Deepracer challenge is a part of a series of embodied intelligence competitions in the field of autonomous vehicles, called The AI Driving Olympics (AI-DO). The overall objective of the AI-DO is to provide accessible mechanisms for benchmarking progress in autonomy applied to the task of autonomous driving. The tricky section of this challenge was the sim2real transfer of the learned skills. To reduce the domain gap in the observation space we did a canny edge detection in addition to crop** out of the unnecessary background information. We modeled the problem as a behavioral cloning task and used MLP-MIXER to optimize for runtime. We made sure our model was capable of handling control noise by careful filtration of the training data and that gave us a robust model capable of completing the track even when 50% of the commands were randomly changed. The overall runtime of the model was only 2-3ms on a modern CPU.
△ Less
Submitted 16 May, 2022;
originally announced May 2022.
-
NMR: Neural Manifold Representation for Autonomous Driving
Authors:
Unnikrishnan R. Nair,
Sarthak Sharma,
Midhun S. Menon,
Srikanth Vidapanakal
Abstract:
Autonomous driving requires efficient reasoning about the Spatio-temporal nature of the semantics of the scene. Recent approaches have successfully amalgamated the traditional modular architecture of an autonomous driving stack comprising perception, prediction, and planning in an end-to-end trainable system. Such a system calls for a shared latent space embedding with interpretable intermediate t…
▽ More
Autonomous driving requires efficient reasoning about the Spatio-temporal nature of the semantics of the scene. Recent approaches have successfully amalgamated the traditional modular architecture of an autonomous driving stack comprising perception, prediction, and planning in an end-to-end trainable system. Such a system calls for a shared latent space embedding with interpretable intermediate trainable projected representation. One such successfully deployed representation is the Bird's-Eye View(BEV) representation of the scene in ego-frame. However, a fundamental assumption for an undistorted BEV is the local coplanarity of the world around the ego-vehicle. This assumption is highly restrictive, as roads, in general, do have gradients. The resulting distortions make path planning inefficient and incorrect. To overcome this limitation, we propose Neural Manifold Representation (NMR), a representation for the task of autonomous driving that learns to infer semantics and predict way-points on a manifold over a finite horizon, centered on the ego-vehicle. We do this using an iterative attention mechanism applied on a latent high dimensional embedding of surround monocular images and partial ego-vehicle state. This representation helps generate motion and behavior plans consistent with and cognizant of the surface geometry. We propose a sampling algorithm based on edge-adaptive coverage loss of BEV occupancy grid and associated guidance flow field to generate the surface manifold while incurring minimal computational overhead. We aim to test the efficacy of our approach on CARLA and SYNTHIA-SF.
△ Less
Submitted 11 May, 2022;
originally announced May 2022.
-
Non Holonomic Collision Avoidance of Dynamic Obstacles under Non-Parametric Uncertainty: A Hilbert Space Approach
Authors:
Unni Krishnan R Nair,
Anish Gupta,
D. A. Sasi Kiran,
Ajay Shrihari,
Vanshil Shah,
Arun Kumar Singh,
K. Madhava Krishna
Abstract:
We consider the problem of an agent/robot with non-holonomic kinematics avoiding many dynamic obstacles. State and velocity noise of both the robot and obstacles as well as the robot's control noise are modelled as non-parametric distributions as often the Gaussian assumptions of noise models are violated in real-world scenarios. Under these assumptions, we formulate a robust MPC that samples robo…
▽ More
We consider the problem of an agent/robot with non-holonomic kinematics avoiding many dynamic obstacles. State and velocity noise of both the robot and obstacles as well as the robot's control noise are modelled as non-parametric distributions as often the Gaussian assumptions of noise models are violated in real-world scenarios. Under these assumptions, we formulate a robust MPC that samples robotic controls effectively in a manner that aligns the robot to the goal state while avoiding obstacles under the duress of such non-parametric noise. In particular, the MPC incorporates a distribution matching cost that effectively aligns the distribution of the current collision cone to a certain desired distribution whose samples are collision-free. This cost is posed as a distance function in the Hilbert Space, whose minimization typically results in the collision cone samples becoming collision-free. We compare and show tangible performance gain with methods that model the collision cone distribution by linearizing the Gaussian approximations of the original non-parametric state and obstacle distributions. We also show superior performance with methods that pose a chance constraint formulation of the Gaussian approximations of non-parametric noise without subjecting such approximations to further linearizations. The performance gain is shown both in terms of trajectory length and control costs that vindicates the efficacy of the proposed method. To the best of our knowledge, this is the first presentation of non-holonomic collision avoidance of moving obstacles in the presence of non-parametric state, velocity and actuator noise models.
△ Less
Submitted 1 January, 2022; v1 submitted 24 December, 2021;
originally announced December 2021.
-
Grounding Linguistic Commands to Navigable Regions
Authors:
Nivedita Rufus,
Kanishk Jain,
Unni Krishnan R Nair,
Vineet Gandhi,
K Madhava Krishna
Abstract:
Humans have a natural ability to effortlessly comprehend linguistic commands such as "park next to the yellow sedan" and instinctively know which region of the road the vehicle should navigate. Extending this ability to autonomous vehicles is the next step towards creating fully autonomous agents that respond and act according to human commands. To this end, we propose the novel task of Referring…
▽ More
Humans have a natural ability to effortlessly comprehend linguistic commands such as "park next to the yellow sedan" and instinctively know which region of the road the vehicle should navigate. Extending this ability to autonomous vehicles is the next step towards creating fully autonomous agents that respond and act according to human commands. To this end, we propose the novel task of Referring Navigable Regions (RNR), i.e., grounding regions of interest for navigation based on the linguistic command. RNR is different from Referring Image Segmentation (RIS), which focuses on grounding an object referred to by the natural language expression instead of grounding a navigable region. For example, for a command "park next to the yellow sedan," RIS will aim to segment the referred sedan, and RNR aims to segment the suggested parking region on the road. We introduce a new dataset, Talk2Car-RegSeg, which extends the existing Talk2car dataset with segmentation masks for the regions described by the linguistic commands. A separate test split with concise manoeuvre-oriented commands is provided to assess the practicality of our dataset. We benchmark the proposed dataset using a novel transformer-based architecture. We present extensive ablations and show superior performance over baselines on multiple evaluation metrics. A downstream path planner generating trajectories based on RNR outputs confirms the efficacy of the proposed framework.
△ Less
Submitted 24 December, 2021;
originally announced December 2021.
-
Cosine meets Softmax: A tough-to-beat baseline for visual grounding
Authors:
Nivedita Rufus,
Unni Krishnan R Nair,
K. Madhava Krishna,
Vineet Gandhi
Abstract:
In this paper, we present a simple baseline for visual grounding for autonomous driving which outperforms the state of the art methods, while retaining minimal design choices. Our framework minimizes the cross-entropy loss over the cosine distance between multiple image ROI features with a text embedding (representing the give sentence/phrase). We use pre-trained networks for obtaining the initial…
▽ More
In this paper, we present a simple baseline for visual grounding for autonomous driving which outperforms the state of the art methods, while retaining minimal design choices. Our framework minimizes the cross-entropy loss over the cosine distance between multiple image ROI features with a text embedding (representing the give sentence/phrase). We use pre-trained networks for obtaining the initial embeddings and learn a transformation layer on top of the text embedding. We perform experiments on the Talk2Car dataset and achieve 68.7% AP50 accuracy, improving upon the previous state of the art by 8.6%. Our investigation suggests reconsideration towards more approaches employing sophisticated attention mechanisms or multi-stage reasoning or complex metric learning loss functions by showing promise in simpler alternatives.
△ Less
Submitted 13 September, 2020;
originally announced September 2020.
-
SROM: Simple Real-time Odometry and Map** using LiDAR data for Autonomous Vehicles
Authors:
Nivedita Rufus,
Unni Krishnan R. Nair,
A. V. S. Sai Bhargav Kumar,
Vashist Madiraju,
K. Madhava Krishna
Abstract:
In this paper, we present SROM, a novel real-time Simultaneous Localization and Map** (SLAM) system for autonomous vehicles. The keynote of the paper showcases SROM's ability to maintain localization at low sampling rates or at high linear or angular velocities where most popular LiDAR based localization approaches get degraded fast. We also demonstrate SROM to be computationally efficient and c…
▽ More
In this paper, we present SROM, a novel real-time Simultaneous Localization and Map** (SLAM) system for autonomous vehicles. The keynote of the paper showcases SROM's ability to maintain localization at low sampling rates or at high linear or angular velocities where most popular LiDAR based localization approaches get degraded fast. We also demonstrate SROM to be computationally efficient and capable of handling high-speed maneuvers. It also achieves low drifts without the need for any other sensors like IMU and/or GPS. Our method has a two-layer structure wherein first, an approximate estimate of the rotation angle and translation parameters are calculated using a Phase Only Correlation (POC) method. Next, we use this estimate as an initialization for a point-to-plane ICP algorithm to obtain fine matching and registration. Another key feature of the proposed algorithm is the removal of dynamic objects before matching the scans. This improves the performance of our system as the dynamic objects can corrupt the matching scheme and derail localization. Our SLAM system can build reliable maps at the same time generating high-quality odometry. We exhaustively evaluated the proposed method in many challenging highways/country/urban sequences from the KITTI dataset and the results demonstrate better accuracy in comparisons to other state-of-the-art methods with reduced computational expense aiding in real-time realizations. We have also integrated our SROM system with our in-house autonomous vehicle and compared it with the state-of-the-art methods like LOAM and LeGO-LOAM.
△ Less
Submitted 7 May, 2020; v1 submitted 5 May, 2020;
originally announced May 2020.
-
SVM Enhanced Frenet Frame Planner For Safe Navigation Amidst Moving Agents
Authors:
Unni Krishnan R Nair,
Nivedita Rufus,
Vashist Madiraju,
K Madhava Krishna
Abstract:
This paper proposes an SVM Enhanced Trajectory Planner for dynamic scenes, typically those encountered in on road settings. Frenet frame based trajectory generation is popular in the context of autonomous driving both in research and industry. We incorporate a safety based maximal margin criteria using a SVM layer that generates control points that are maximally separated from all dynamic obstacle…
▽ More
This paper proposes an SVM Enhanced Trajectory Planner for dynamic scenes, typically those encountered in on road settings. Frenet frame based trajectory generation is popular in the context of autonomous driving both in research and industry. We incorporate a safety based maximal margin criteria using a SVM layer that generates control points that are maximally separated from all dynamic obstacles in the scene. A kinematically consistent trajectory generator then computes a path through these waypoints. We showcase through simulations as well as real world experiments on a self driving car that the SVM enhanced planner provides for a larger offset with dynamic obstacles than the regular Frenet frame based trajectory generation. Thereby, the authors argue that such a formulation is inherently suited for navigation amongst pedestrians. We assume the availability of an intent or trajectory prediction module that predicts the future trajectories of all dynamic actors in the scene.
△ Less
Submitted 11 September, 2020; v1 submitted 2 July, 2019;
originally announced July 2019.
-
Phylogenetic Analysis of Cell Types using Histone Modifications
Authors:
Nishanth Ulhas Nair,
Yu Lin,
Philipp Bucher,
Bernard M. E. Moret
Abstract:
In cell differentiation, a cell of a less specialized type becomes one of a more specialized type, even though all cells have the same genome. Transcription factors and epigenetic marks like histone modifications can play a significant role in the differentiation process. In this paper, we present a simple analysis of cell types and differentiation paths using phylogenetic inference based on ChIP-…
▽ More
In cell differentiation, a cell of a less specialized type becomes one of a more specialized type, even though all cells have the same genome. Transcription factors and epigenetic marks like histone modifications can play a significant role in the differentiation process. In this paper, we present a simple analysis of cell types and differentiation paths using phylogenetic inference based on ChIP-Seq histone modification data. We propose new data representation techniques and new distance measures for ChIP-Seq data and use these together with standard phylogenetic inference methods to build biologically meaningful trees that indicate how diverse types of cells are related. We demonstrate our approach on H3K4me3 and H3K27me3 data for 37 and 13 types of cells respectively, using the dataset to explore various issues surrounding replicate data, variability between cells of the same type, and robustness. The promising results we obtain point the way to a new approach to the study of cell differentiation.
△ Less
Submitted 30 July, 2013;
originally announced July 2013.
-
Elementary results on the binary quadratic form a^2+ab+b^2
Authors:
Umesh P. Nair
Abstract:
This paper examines with elementary proofs some interesting properties of numbers in the binary quadratic form $a^2+ab+b^2$, where $a$ and $b$ are non-negative integers. Key findings of this paper are (i) a prime number $p$ can be represented as $a^2+ab+b^2$ if and only if $p$ is of the form $6k+1$, with the only exception of 3, (ii) any positive integer can be represented as $a^2+ab+b^2$ if and…
▽ More
This paper examines with elementary proofs some interesting properties of numbers in the binary quadratic form $a^2+ab+b^2$, where $a$ and $b$ are non-negative integers. Key findings of this paper are (i) a prime number $p$ can be represented as $a^2+ab+b^2$ if and only if $p$ is of the form $6k+1$, with the only exception of 3, (ii) any positive integer can be represented as $a^2+ab+b^2$ if and only if its all prime factors that are not in the same form have even exponents in the standard factorization, and (iii) all the factors of an integer in the form $a^2+ab+b^2$, where $a$ and $b$ are positive and relatively prime to each other, are also of the same form. A general formula for the number of distinct representations of any positive integer in this form is conjectured. A comparison of the results with the properties of some other binary quadratic forms is given.
△ Less
Submitted 9 August, 2004;
originally announced August 2004.
-
Some Classes Of Distributions On The Non-Negative Lattice
Authors:
S. Satheesh,
N. Unnikrishnan Nair
Abstract:
A method for constructing distributions on the non negative integers as discrete analogue of continuous distributions on the non negative real is presented. A justification of the definition of discrete self decomposable laws is provided. Discrete analogue of distributions of the same type and the role of Bernoulli law in this context is discussed. Generalizations of some discrete laws and their…
▽ More
A method for constructing distributions on the non negative integers as discrete analogue of continuous distributions on the non negative real is presented. A justification of the definition of discrete self decomposable laws is provided. Discrete analogue of distributions of the same type and the role of Bernoulli law in this context is discussed. Generalizations of some discrete laws and their properties are given. The geometric compounding problem for discrete distributions is studied by introducing discrete semi Mittag Leffler laws.
△ Less
Submitted 20 November, 2003;
originally announced November 2003.
-
Stability of Random Sums
Authors:
S. Satheesh,
N. Unnikrishnan Nair,
E. Sandhya
Abstract:
When the distribution of a random (N) sum of independent copies of a r.v X is of the same type as that of X we say that X is N-sum stable. In this paper we consider a generalization of stability of geometric sums by studying distributions that are stable under summation w.r.t Harris law. We show that the notion of stability of random sums can be extended to include the case when X is discrete. F…
▽ More
When the distribution of a random (N) sum of independent copies of a r.v X is of the same type as that of X we say that X is N-sum stable. In this paper we consider a generalization of stability of geometric sums by studying distributions that are stable under summation w.r.t Harris law. We show that the notion of stability of random sums can be extended to include the case when X is discrete. Finally we propose a method to identify the probability law of N for which X is N-sum stable. See also Satheesh and Nair (2002), (Some classes of distributions on ther non-negative lattice, J. Ind. Statist. Assoc., 2002, 40, 41-58) for a study of discrete laws of the same type and stability of geometric sums of discrete laws.
△ Less
Submitted 20 November, 2003;
originally announced November 2003.
-
On the Stability of Geometric Extremes
Authors:
S. Satheesh,
N. Unnikrishnan Nair
Abstract:
Possible reasons for the uniqueness of the positive geometric law in the context of stability of random extremes are explored here culminating in a conjecture characterizing the geometric law. Our reasoning comes closer in justifying the geometric law in similar contexts discussed in Arnold et al. (1986) and Marshall & Olkin (1997) and also supplement their arguments.
Possible reasons for the uniqueness of the positive geometric law in the context of stability of random extremes are explored here culminating in a conjecture characterizing the geometric law. Our reasoning comes closer in justifying the geometric law in similar contexts discussed in Arnold et al. (1986) and Marshall & Olkin (1997) and also supplement their arguments.
△ Less
Submitted 14 April, 2005; v1 submitted 4 May, 2003;
originally announced May 2003.
-
A Note on Maximum and Minimum Stability of Certain Distributions
Authors:
S. Satheesh,
N. U. Nair
Abstract:
In the context of stability of the extremes of a random variable X with respect to a positive integer valued random variable N we discuss the cases (i) X is exponential (ii) non-geometric laws for N (iii) identifying N for the stability of a given X and (iv) extending the notion to a discrete random variable X.
In the context of stability of the extremes of a random variable X with respect to a positive integer valued random variable N we discuss the cases (i) X is exponential (ii) non-geometric laws for N (iii) identifying N for the stability of a given X and (iv) extending the notion to a discrete random variable X.
△ Less
Submitted 3 May, 2003;
originally announced May 2003.