-
NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs
Authors:
Mikhail Galkin,
Etienne Denis,
Jiapeng Wu,
William L. Hamilton
Abstract:
Conventional representation learning algorithms for knowledge graphs (KG) map each entity to a unique embedding vector. Such a shallow lookup results in a linear growth of memory consumption for storing the embedding matrix and incurs high computational costs when working with real-world KGs. Drawing parallels with subword tokenization commonly used in NLP, we explore the landscape of more paramet…
▽ More
Conventional representation learning algorithms for knowledge graphs (KG) map each entity to a unique embedding vector. Such a shallow lookup results in a linear growth of memory consumption for storing the embedding matrix and incurs high computational costs when working with real-world KGs. Drawing parallels with subword tokenization commonly used in NLP, we explore the landscape of more parameter-efficient node embedding strategies with possibly sublinear memory requirements. To this end, we propose NodePiece, an anchor-based approach to learn a fixed-size entity vocabulary. In NodePiece, a vocabulary of subword/sub-entity units is constructed from anchor nodes in a graph with known relation types. Given such a fixed-size vocabulary, it is possible to bootstrap an encoding and embedding for any entity, including those unseen during training. Experiments show that NodePiece performs competitively in node classification, link prediction, and relation prediction tasks while retaining less than 10% of explicit nodes in a graph as anchors and often having 10x fewer parameters. To this end, we show that a NodePiece-enabled model outperforms existing shallow models on a large OGB WikiKG 2 graph having 70x fewer parameters.
△ Less
Submitted 1 February, 2022; v1 submitted 22 June, 2021;
originally announced June 2021.
-
Empowering Urban Governance through Urban Science: Multi-scale Dynamics of Urban Systems Worldwide
Authors:
Juste Raimbault,
Eric Denis,
Denise Pumain
Abstract:
The current science of cities can provide a useful foundation for future urban policies, provided that these proposals have been validated by correct observations of the diversity of situations in the world. However, international comparisons of the evolution of cities often produce uncertain results because national territorial frameworks are not always in strict correspondence with the dynamics…
▽ More
The current science of cities can provide a useful foundation for future urban policies, provided that these proposals have been validated by correct observations of the diversity of situations in the world. However, international comparisons of the evolution of cities often produce uncertain results because national territorial frameworks are not always in strict correspondence with the dynamics of urban systems. We propose to provide various compositions of systems of cities to better take into account the dynamic networking of cities that go beyond regional and national territorial boundaries. Different models conceived for explaining city size and urban growth distributions enable to establish a correspondence between urban trajectories when observed at the level of cities and systems of cities. We test the validity and representativeness of several dynamic models of complex urban systems and their variations across regions of the world, at the macroscopic scale of systems of cities. The originality of the approach is in considering spatial interaction and evolutionary path dependence as major features in the general behavior of urban entities. The models studied include diverse and complementary processes, such as economic exchanges, diffusion of innovations and physical network flows. Complex systems' dynamics is in principle unpredictable, but contextualizing it regarding demographic, income and resource components may help in minimizing the forecasting errors.
△ Less
Submitted 20 May, 2020;
originally announced May 2020.
-
Inductive Relation Prediction by Subgraph Reasoning
Authors:
Komal K. Teru,
Etienne Denis,
William L. Hamilton
Abstract:
The dominant paradigm for relation prediction in knowledge graphs involves learning and operating on latent representations (i.e., embeddings) of entities and relations. However, these embedding-based methods do not explicitly capture the compositional logical rules underlying the knowledge graph, and they are limited to the transductive setting, where the full set of entities must be known during…
▽ More
The dominant paradigm for relation prediction in knowledge graphs involves learning and operating on latent representations (i.e., embeddings) of entities and relations. However, these embedding-based methods do not explicitly capture the compositional logical rules underlying the knowledge graph, and they are limited to the transductive setting, where the full set of entities must be known during training. Here, we propose a graph neural network based relation prediction framework, GraIL, that reasons over local subgraph structures and has a strong inductive bias to learn entity-independent relational semantics. Unlike embedding-based models, GraIL is naturally inductive and can generalize to unseen entities and graphs after training. We provide theoretical proof and strong empirical evidence that GraIL can represent a useful subset of first-order logic and show that GraIL outperforms existing rule-induction baselines in the inductive setting. We also demonstrate significant gains obtained by ensembling GraIL with various knowledge graph embedding methods in the transductive setting, highlighting the complementary inductive bias of our method.
△ Less
Submitted 11 February, 2020; v1 submitted 16 November, 2019;
originally announced November 2019.
