OmniFusion Technical Report
Authors:
Elizaveta Goncharova,
Anton Razzhigaev,
Matvey Mikhalchuk,
Maxim Kurkin,
Irina Abdullaeva,
Matvey Skripkin,
Ivan Oseledets,
Denis Dimitrov,
Andrey Kuznetsov
Abstract:
Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various…
▽ More
Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekee**, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion.
△ Less
Submitted 9 April, 2024;
originally announced April 2024.
On the nonlinear NMR and magnon BEC in antiferromagnetic materials with coupled electron-nuclear spin precession
Authors:
L. V. Abdurakhimov,
M. A. Borich,
Yu. M. Bunkov,
R. R. Gazizulin,
D. Konstantinov,
M. I. Kurkin,
A. P. Tankeyev
Abstract:
We present a new study of nonlinear NMR and Bose-Einstein Condensation (BEC) of nuclear spin waves in antiferromagnetic MnCO3 with coupled electron and nuclear spins. In particular, we show that the observed behaviour of NMR signals strongly contradicts the conventional description of paramagnetic ensembles of noninteracting spins based on the phenomenological Bloch equations. We present a new the…
▽ More
We present a new study of nonlinear NMR and Bose-Einstein Condensation (BEC) of nuclear spin waves in antiferromagnetic MnCO3 with coupled electron and nuclear spins. In particular, we show that the observed behaviour of NMR signals strongly contradicts the conventional description of paramagnetic ensembles of noninteracting spins based on the phenomenological Bloch equations. We present a new theoretical description of the coupled electron-nuclear spin precession, which takes into account an indirect relaxation of nuclear spins via the electron subsystem. We show that the magnitude of the nuclear magnetization is conserved for arbitrary large excitation powers, which is drastically different from the conventional heating scenario derived from the Bloch equations. This provides strong evidence that the coherent precession of macroscopic nuclear magnetization observed experimentally can be identified with BEC of nuclear spin waves with k=0.
△ Less
Submitted 16 January, 2018; v1 submitted 1 March, 2017;
originally announced March 2017.
