-
YaART: Yet Another ART Rendering Technology
Authors:
Sergey Kastryulin,
Artem Konev,
Alexander Shishenya,
Eugene Lyapustin,
Artem Khurshudov,
Alexander Tselousov,
Nikita Vinokurov,
Denis Kuznedelev,
Alexander Markovich,
Grigoriy Livshits,
Alexey Kirillov,
Anastasiia Tabisheva,
Liubov Chubarova,
Marina Kaminskaia,
Alexander Ustyuzhanin,
Artemii Shvetsov,
Daniil Shlenskii,
Valerii Startsev,
Dmitrii Kornilov,
Mikhail Romanov,
Artem Babenko,
Sergei Ovcharenko,
Valentin Khrulkov
Abstract:
In the rapidly progressing field of generative models, the development of efficient and high-fidelity text-to-image diffusion systems represents a significant frontier. This study introduces YaART, a novel production-grade text-to-image cascaded diffusion model aligned to human preferences using Reinforcement Learning from Human Feedback (RLHF). During the development of YaART, we especially focus…
▽ More
In the rapidly progressing field of generative models, the development of efficient and high-fidelity text-to-image diffusion systems represents a significant frontier. This study introduces YaART, a novel production-grade text-to-image cascaded diffusion model aligned to human preferences using Reinforcement Learning from Human Feedback (RLHF). During the development of YaART, we especially focus on the choices of the model and training dataset sizes, the aspects that were not systematically investigated for text-to-image cascaded diffusion models before. In particular, we comprehensively analyze how these choices affect both the efficiency of the training process and the quality of the generated images, which are highly important in practice. Furthermore, we demonstrate that models trained on smaller datasets of higher-quality images can successfully compete with those trained on larger datasets, establishing a more efficient scenario of diffusion models training. From the quality perspective, YaART is consistently preferred by users over many existing state-of-the-art models.
△ Less
Submitted 8 April, 2024;
originally announced April 2024.
-
Towards True Detail Restoration for Super-Resolution: A Benchmark and a Quality Metric
Authors:
Eugene Lyapustin,
Anastasia Kirillova,
Viacheslav Meshchaninov,
Evgeney Zimin,
Nikolai Karetin,
Dmitriy Vatolin
Abstract:
Super-resolution (SR) has become a widely researched topic in recent years. SR methods can improve overall image and video quality and create new possibilities for further content analysis. But the SR mainstream focuses primarily on increasing the naturalness of the resulting image despite potentially losing context accuracy. Such methods may produce an incorrect digit, character, face, or other s…
▽ More
Super-resolution (SR) has become a widely researched topic in recent years. SR methods can improve overall image and video quality and create new possibilities for further content analysis. But the SR mainstream focuses primarily on increasing the naturalness of the resulting image despite potentially losing context accuracy. Such methods may produce an incorrect digit, character, face, or other structural object even though they otherwise yield good visual quality. Incorrect detail restoration can cause errors when detecting and identifying objects both manually and automatically. To analyze the detail-restoration capabilities of image and video SR models, we developed a benchmark based on our own video dataset, which contains complex patterns that SR models generally fail to correctly restore. We assessed 32 recent SR models using our benchmark and compared their ability to preserve scene context. We also conducted a crowd-sourced comparison of restored details and developed an objective assessment metric that outperforms other quality metrics by correlation with subjective scores for this task. In conclusion, we provide a deep analysis of benchmark results that yields insights for future SR-based work.
△ Less
Submitted 16 March, 2022;
originally announced March 2022.
-
ERQA: Edge-Restoration Quality Assessment for Video Super-Resolution
Authors:
Anastasia Kirillova,
Eugene Lyapustin,
Anastasia Antsiferova,
Dmitry Vatolin
Abstract:
Despite the growing popularity of video super-resolution (VSR), there is still no good way to assess the quality of the restored details in upscaled frames. Some SR methods may produce the wrong digit or an entirely different face. Whether a method's results are trustworthy depends on how well it restores truthful details. Image super-resolution can use natural distributions to produce a high-reso…
▽ More
Despite the growing popularity of video super-resolution (VSR), there is still no good way to assess the quality of the restored details in upscaled frames. Some SR methods may produce the wrong digit or an entirely different face. Whether a method's results are trustworthy depends on how well it restores truthful details. Image super-resolution can use natural distributions to produce a high-resolution image that is only somewhat similar to the real one. VSR enables exploration of additional information in neighboring frames to restore details from the original scene. The ERQA metric, which we propose in this paper, aims to estimate a model's ability to restore real details using VSR. On the assumption that edges are significant for detail and character recognition, we chose edge fidelity as the foundation for this metric. Experimental validation of our work is based on the MSU Video Super-Resolution Benchmark, which includes the most difficult patterns for detail restoration and verifies the fidelity of details from the original frame. Code for the proposed metric is publicly available at https://github.com/msu-video-group/ERQA.
△ Less
Submitted 24 January, 2022; v1 submitted 19 October, 2021;
originally announced October 2021.
