-
Rethinking Processing Distortions: Disentangling the Impact of Speech Enhancement Errors on Speech Recognition Performance
Authors:
Tsubasa Ochiai,
Kazuma Iwamoto,
Marc Delcroix,
Rintaro Ikeshita,
Hiroshi Sato,
Shoko Araki,
Shigeru Katagiri
Abstract:
It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with a single-channel speech enhancement (SE) front-end. This is generally attributed to the processing distortions caused by the nonlinear processing of single-channel SE front-ends. However, the causes of such degraded ASR performance have not been fully investigated. How to design single-channel SE f…
▽ More
It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with a single-channel speech enhancement (SE) front-end. This is generally attributed to the processing distortions caused by the nonlinear processing of single-channel SE front-ends. However, the causes of such degraded ASR performance have not been fully investigated. How to design single-channel SE front-ends in a way that significantly improves ASR performance remains an open research question. In this study, we investigate a signal-level numerical metric that can explain the cause of degradation in ASR performance. To this end, we propose a novel analysis scheme based on the orthogonal projection-based decomposition of SE errors. This scheme manually modifies the ratio of the decomposed interference, noise, and artifact errors, and it enables us to directly evaluate the impact of each error type on ASR performance. Our analysis reveals the particularly detrimental effect of artifact errors on ASR performance compared to the other types of errors. This provides us with a more principled definition of processing distortions that cause the ASR performance degradation. Then, we study two practical approaches for reducing the impact of artifact errors. First, we prove that the simple observation adding (OA) post-processing (i.e., interpolating the enhanced and observed signals) can monotonically improve the signal-to-artifact ratio. Second, we propose a novel training objective, called artifact-boosted signal-to-distortion ratio (AB-SDR), which forces the model to estimate the enhanced signals with fewer artifact errors. Through experiments, we confirm that both the OA and AB-SDR approaches are effective in decreasing artifact errors caused by single-channel SE front-ends, allowing them to significantly improve ASR performance.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Natural-artificial hybrid swarm: Cyborg-insect group navigation in unknown obstructed soft terrain
Authors:
Yang Bai,
Phuoc Thanh Tran Ngoc,
Huu Duoc Nguyen,
Duc Long Le,
Quang Huy Ha,
Kazuki Kai,
Yu Xiang See To,
Yaosheng Deng,
Jie Song,
Naoki Wakamiya,
Hirotaka Sato,
Masaki Ogura
Abstract:
Navigating multi-robot systems in complex terrains has always been a challenging task. This is due to the inherent limitations of traditional robots in collision avoidance, adaptation to unknown environments, and sustained energy efficiency. In order to overcome these limitations, this research proposes a solution by integrating living insects with miniature electronic controllers to enable roboti…
▽ More
Navigating multi-robot systems in complex terrains has always been a challenging task. This is due to the inherent limitations of traditional robots in collision avoidance, adaptation to unknown environments, and sustained energy efficiency. In order to overcome these limitations, this research proposes a solution by integrating living insects with miniature electronic controllers to enable robotic-like programmable control, and proposing a novel control algorithm for swarming. Although these creatures, called cyborg insects, have the ability to instinctively avoid collisions with neighbors and obstacles while adapting to complex terrains, there is a lack of literature on the control of multi-cyborg systems. This research gap is due to the difficulty in coordinating the movements of a cyborg system under the presence of insects' inherent individual variability in their reactions to control input. In response to this issue, we propose a novel swarm navigation algorithm addressing these challenges. The effectiveness of the algorithm is demonstrated through an experimental validation in which a cyborg swarm was successfully navigated through an unknown sandy field with obstacles and hills. This research contributes to the domain of swarm robotics and showcases the potential of integrating biological organisms with robotics and control theory to create more intelligent autonomous systems with real-world applications.
△ Less
Submitted 27 March, 2024; v1 submitted 26 March, 2024;
originally announced March 2024.
-
Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters
Authors:
Kenichi Fujita,
Hiroshi Sato,
Takanori Ashihara,
Hiroki Kanagawa,
Marc Delcroix,
Takafumi Moriya,
Yusuke Ijima
Abstract:
The zero-shot text-to-speech (TTS) method, based on speaker embeddings extracted from reference speech using self-supervised learning (SSL) speech representations, can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method.…
▽ More
The zero-shot text-to-speech (TTS) method, based on speaker embeddings extracted from reference speech using self-supervised learning (SSL) speech representations, can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method. We incorporated adapters into the SSL model, which we fine-tuned with the TTS model using noisy reference speech. In addition, to further improve performance, we adopted a speech enhancement (SE) front-end. With these improvements, our proposed SSL-based zero-shot TTS achieved high-quality speech synthesis with noisy reference speech. Through the objective and subjective evaluations, we confirmed that the proposed method is highly robust to noise in reference speech, and effectively works in combination with SE.
△ Less
Submitted 10 January, 2024;
originally announced January 2024.
-
3D Programming of Patterned Heterogeneous Interface for 4D Smart Robotics
Authors:
Kewei Song,
Chunfeng Xiong,
Ze Zhang,
Kunlin Wu,
Weiyang Wan,
Yifan Wang,
Shinjiro Umezu,
Hirotaka Sato
Abstract:
Shape memory structures are playing an important role in many cutting-edge intelligent fields. However, the existing technologies can only realize 4D printing of a single polymer or metal, which limits practical applications. Here, we report a construction strategy for TSMP/M heterointerface, which uses Pd2+-containing shape memory polymer (AP-SMR) to induce electroless plating reaction and relies…
▽ More
Shape memory structures are playing an important role in many cutting-edge intelligent fields. However, the existing technologies can only realize 4D printing of a single polymer or metal, which limits practical applications. Here, we report a construction strategy for TSMP/M heterointerface, which uses Pd2+-containing shape memory polymer (AP-SMR) to induce electroless plating reaction and relies on molecular dynamics, which has both shape memory properties and metal activity and information processing power. Through multi-material DLP 3D printing technology, the interface can be 3D selectively programmed on functional substrate parts of arbitrary shapes to become 4D electronic smart devices (Robotics). Microscopically, this type of interface appears as a composite structure with a nanometer-micrometer interface height, which is composed of a pure substrate layer (smart materials), an intermediate layer (a composite structure in which metal particles are embedded in a polymer cross-linked network) and a pure metal layer. The structure programmed by TSMP/M heterointerface exhibits both SMA characteristics and metal properties, thus having more intelligent functions (electroactive, electrothermal deformation, electronically controlled denaturation) and higher performance (selectivity of shape memory structures can be realized control, remote control, inline control and low voltage control). This is expected to provide a more flexible manufacturing process as platform technology for designing, manufacturing and applying smart devices with new concepts, and promote the development of cutting-edge industries such as smart robots and smart electronics.
△ Less
Submitted 22 December, 2023;
originally announced December 2023.
-
How does end-to-end speech recognition training impact speech enhancement artifacts?
Authors:
Kazuma Iwamoto,
Tsubasa Ochiai,
Marc Delcroix,
Rintaro Ikeshita,
Hiroshi Sato,
Shoko Araki,
Shigeru Katagiri
Abstract:
Jointly training a speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end has been investigated as a way to mitigate the influence of \emph{processing distortion} generated by single-channel SE on ASR. In this paper, we investigate the effect of such joint training on the signal-level characteristics of the enhanced signals from the viewpoint of the decomposed noise a…
▽ More
Jointly training a speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end has been investigated as a way to mitigate the influence of \emph{processing distortion} generated by single-channel SE on ASR. In this paper, we investigate the effect of such joint training on the signal-level characteristics of the enhanced signals from the viewpoint of the decomposed noise and artifact errors. The experimental analyses provide two novel findings: 1) ASR-level training of the SE front-end reduces the artifact errors while increasing the noise errors, and 2) simply interpolating the enhanced and observed signals, which achieves a similar effect of reducing artifacts and increasing noise, improves ASR performance without jointly modifying the SE and ASR modules, even for a strong ASR back-end using a WavLM feature extractor. Our findings provide a better understanding of the effect of joint training and a novel insight for designing an ASR agnostic SE front-end.
△ Less
Submitted 20 November, 2023;
originally announced November 2023.
-
End-to-End Joint Target and Non-Target Speakers ASR
Authors:
Ryo Masumura,
Naoki Makishima,
Taiga Yamane,
Yoshihiko Yamazaki,
Saki Mizuno,
Mana Ihori,
Mihiro Uchida,
Keita Suzuki,
Hiroshi Sato,
Tomohiro Tanaka,
Akihiko Takashima,
Satoshi Suzuki,
Takafumi Moriya,
Nobukatsu Hojo,
Atsushi Ando
Abstract:
This paper proposes a novel automatic speech recognition (ASR) system that can transcribe individual speaker's speech while identifying whether they are target or non-target speakers from multi-talker overlapped speech. Target-speaker ASR systems are a promising way to only transcribe a target speaker's speech by enrolling the target speaker's information. However, in conversational ASR applicatio…
▽ More
This paper proposes a novel automatic speech recognition (ASR) system that can transcribe individual speaker's speech while identifying whether they are target or non-target speakers from multi-talker overlapped speech. Target-speaker ASR systems are a promising way to only transcribe a target speaker's speech by enrolling the target speaker's information. However, in conversational ASR applications, transcribing both the target speaker's speech and non-target speakers' ones is often required to understand interactive information. To naturally consider both target and non-target speakers in a single ASR model, our idea is to extend autoregressive modeling-based multi-talker ASR systems to utilize the enrollment speech of the target speaker. Our proposed ASR is performed by recursively generating both textual tokens and tokens that represent target or non-target speakers. Our experiments demonstrate the effectiveness of our proposed method.
△ Less
Submitted 4 June, 2023;
originally announced June 2023.
-
Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data
Authors:
Takafumi Moriya,
Hiroshi Sato,
Tsubasa Ochiai,
Marc Delcroix,
Takanori Ashihara,
Kohei Matsuura,
Tomohiro Tanaka,
Ryo Masumura,
Atsunori Ogawa,
Taichi Asami
Abstract:
Neural transducer (RNNT)-based target-speaker speech recognition (TS-RNNT) directly transcribes a target speaker's voice from a multi-talker mixture. It is a promising approach for streaming applications because it does not incur the extra computation costs of a target speech extraction frontend, which is a critical barrier to quick response. TS-RNNT is trained end-to-end given the input speech (i…
▽ More
Neural transducer (RNNT)-based target-speaker speech recognition (TS-RNNT) directly transcribes a target speaker's voice from a multi-talker mixture. It is a promising approach for streaming applications because it does not incur the extra computation costs of a target speech extraction frontend, which is a critical barrier to quick response. TS-RNNT is trained end-to-end given the input speech (i.e., mixtures and enrollment speech) and reference transcriptions. The training mixtures are generally simulated by mixing single-talker signals, but conventional TS-RNNT training does not utilize single-speaker signals. This paper proposes using knowledge distillation (KD) to exploit the parallel mixture/single-talker speech data. Our proposed KD scheme uses an RNNT system pretrained with the target single-talker speech input to generate pseudo labels for the TS-RNNT training. Experimental results show that TS-RNNT systems trained with the proposed KD scheme outperform a baseline TS-RNNT.
△ Less
Submitted 25 May, 2023;
originally announced May 2023.
-
Improving Scheduled Sampling for Neural Transducer-based ASR
Authors:
Takafumi Moriya,
Takanori Ashihara,
Hiroshi Sato,
Kohei Matsuura,
Tomohiro Tanaka,
Ryo Masumura
Abstract:
The recurrent neural network-transducer (RNNT) is a promising approach for automatic speech recognition (ASR) with the introduction of a prediction network that autoregressively considers linguistic aspects. To train the autoregressive part, the ground-truth tokens are used as substitutions for the previous output token, which leads to insufficient robustness to incorrect past tokens; a recognitio…
▽ More
The recurrent neural network-transducer (RNNT) is a promising approach for automatic speech recognition (ASR) with the introduction of a prediction network that autoregressively considers linguistic aspects. To train the autoregressive part, the ground-truth tokens are used as substitutions for the previous output token, which leads to insufficient robustness to incorrect past tokens; a recognition error in the decoding leads to further errors. Scheduled sampling (SS) is a technique to train autoregressive model robustly to past errors by randomly replacing some ground-truth tokens with actual outputs generated from a model. SS mitigates the gaps between training and decoding steps, known as exposure bias, and it is often used for attentional encoder-decoder training. However SS has not been fully examined for RNNT because of the difficulty in applying SS to RNNT due to the complicated RNNT output form. In this paper we propose SS approaches suited for RNNT. Our SS approaches sample the tokens generated from the distiribution of RNNT itself, i.e. internal language model or RNNT outputs. Experiments in three datasets confirm that RNNT trained with our SS approach achieves the best ASR performance. In particular, on a Japanese ASR task, our best system outperforms the previous state-of-the-art alternative.
△ Less
Submitted 25 May, 2023;
originally announced May 2023.
-
Downstream Task Agnostic Speech Enhancement with Self-Supervised Representation Loss
Authors:
Hiroshi Sato,
Ryo Masumura,
Tsubasa Ochiai,
Marc Delcroix,
Takafumi Moriya,
Takanori Ashihara,
Kentaro Shinayama,
Saki Mizuno,
Mana Ihori,
Tomohiro Tanaka,
Nobukatsu Hojo
Abstract:
Self-supervised learning (SSL) is the latest breakthrough in speech processing, especially for label-scarce downstream tasks by leveraging massive unlabeled audio data. The noise robustness of the SSL is one of the important challenges to expanding its application. We can use speech enhancement (SE) to tackle this issue. However, the mismatch between the SE model and SSL models potentially limits…
▽ More
Self-supervised learning (SSL) is the latest breakthrough in speech processing, especially for label-scarce downstream tasks by leveraging massive unlabeled audio data. The noise robustness of the SSL is one of the important challenges to expanding its application. We can use speech enhancement (SE) to tackle this issue. However, the mismatch between the SE model and SSL models potentially limits its effect. In this work, we propose a new SE training criterion that minimizes the distance between clean and enhanced signals in the feature representation of the SSL model to alleviate the mismatch. We expect that the loss in the SSL domain could guide SE training to preserve or enhance various levels of characteristics of the speech signals that may be required for high-level downstream tasks. Experiments show that our proposal improves the performance of an SE and SSL pipeline on five downstream tasks with noisy input while maintaining the SE performance.
△ Less
Submitted 24 May, 2023;
originally announced May 2023.
-
On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis
Authors:
Atsushi Ando,
Ryo Masumura,
Akihiko Takashima,
Satoshi Suzuki,
Naoki Makishima,
Keita Suzuki,
Takafumi Moriya,
Takanori Ashihara,
Hiroshi Sato
Abstract:
This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis~(MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded…
▽ More
This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis~(MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded by large-scale pre-trained encoders with conventional heuristic features. One each of the largest pre-trained encoders publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for visual, acoustic, and linguistic modalities, respectively. Experiments on two datasets reveal that methods with domain-specific pre-trained encoders attain better performance than those with conventional features in both unimodal and multimodal scenarios. We also find it better to use the outputs of the intermediate layers of the encoders than those of the output layer. The codes are available at https://github.com/ando-hub/MSA_Pretrain.
△ Less
Submitted 28 October, 2022;
originally announced October 2022.
-
Streaming Target-Speaker ASR with Neural Transducer
Authors:
Takafumi Moriya,
Hiroshi Sato,
Tsubasa Ochiai,
Marc Delcroix,
Takahiro Shinozaki
Abstract:
Although recent advances in deep learning technology have boosted automatic speech recognition (ASR) performance in the single-talker case, it remains difficult to recognize multi-talker speech in which many voices overlap. One conventional approach to tackle this problem is to use a cascade of a speech separation or target speech extraction front-end with an ASR back-end. However, the extra compu…
▽ More
Although recent advances in deep learning technology have boosted automatic speech recognition (ASR) performance in the single-talker case, it remains difficult to recognize multi-talker speech in which many voices overlap. One conventional approach to tackle this problem is to use a cascade of a speech separation or target speech extraction front-end with an ASR back-end. However, the extra computation costs of the front-end module are a critical barrier to quick response, especially for streaming ASR. In this paper, we propose a target-speaker ASR (TS-ASR) system that implicitly integrates the target speech extraction functionality within a streaming end-to-end (E2E) ASR system, i.e. recurrent neural network-transducer (RNNT). Our system uses a similar idea as adopted for target speech extraction, but implements it directly at the level of the encoder of RNNT. This allows TS-ASR to be realized without placing extra computation costs on the front-end. Note that this study presents two major differences between prior studies on E2E TS-ASR; we investigate streaming models and base our study on Conformer models, whereas prior studies used RNN-based systems and considered only offline processing. We confirm in experiments that our TS-ASR achieves comparable recognition performance with conventional cascade systems in the offline setting, while reducing computation costs and realizing streaming TS-ASR.
△ Less
Submitted 19 September, 2022; v1 submitted 9 September, 2022;
originally announced September 2022.
-
Strategies to Improve Robustness of Target Speech Extraction to Enrollment Variations
Authors:
Hiroshi Sato,
Tsubasa Ochiai,
Marc Delcroix,
Keisuke Kinoshita,
Takafumi Moriya,
Naoki Makishima,
Mana Ihori,
Tomohiro Tanaka,
Ryo Masumura
Abstract:
Target speech extraction is a technique to extract the target speaker's voice from mixture signals using a pre-recorded enrollment utterance that characterize the voice characteristics of the target speaker. One major difficulty of target speech extraction lies in handling variability in ``intra-speaker'' characteristics, i.e., characteristics mismatch between target speech and an enrollment utter…
▽ More
Target speech extraction is a technique to extract the target speaker's voice from mixture signals using a pre-recorded enrollment utterance that characterize the voice characteristics of the target speaker. One major difficulty of target speech extraction lies in handling variability in ``intra-speaker'' characteristics, i.e., characteristics mismatch between target speech and an enrollment utterance. While most conventional approaches focus on improving {\it average performance} given a set of enrollment utterances, here we propose to guarantee the {\it worst performance}, which we believe is of great practical importance. In this work, we propose an evaluation metric called worst-enrollment source-to-distortion ratio (SDR) to quantitatively measure the robustness towards enrollment variations. We also introduce a novel training scheme that aims at directly optimizing the worst-case performance by focusing on training with difficult enrollment cases where extraction does not perform well. In addition, we investigate the effectiveness of auxiliary speaker identification loss (SI-loss) as another way to improve robustness over enrollments. Experimental validation reveals the effectiveness of both worst-enrollment target training and SI-loss training to improve robustness against enrollment variations, by increasing speaker discriminability.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
Listen only to me! How well can target speech extraction handle false alarms?
Authors:
Marc Delcroix,
Keisuke Kinoshita,
Tsubasa Ochiai,
Katerina Zmolikova,
Hiroshi Sato,
Tomohiro Nakatani
Abstract:
Target speech extraction (TSE) extracts the speech of a target speaker in a mixture given auxiliary clues characterizing the speaker, such as an enrollment utterance. TSE addresses thus the challenging problem of simultaneously performing separation and speaker identification. There has been much progress in extraction performance following the recent development of neural networks for speech enha…
▽ More
Target speech extraction (TSE) extracts the speech of a target speaker in a mixture given auxiliary clues characterizing the speaker, such as an enrollment utterance. TSE addresses thus the challenging problem of simultaneously performing separation and speaker identification. There has been much progress in extraction performance following the recent development of neural networks for speech enhancement and separation. Most studies have focused on processing mixtures where the target speaker is actively speaking. However, the target speaker is sometimes silent in practice, i.e., inactive speaker (IS). A typical TSE system will tend to output a signal in IS cases, causing false alarms. It is a severe problem for the practical deployment of TSE systems. This paper aims at understanding better how well TSE systems can handle IS cases. We consider two approaches to deal with IS, (1) training a system to directly output zero signals or (2) detecting IS with an extra speaker verification module. We perform an extensive experimental comparison of these schemes in terms of extraction performance and IS detection using the LibriMix dataset and reveal their pros and cons.
△ Less
Submitted 14 July, 2022; v1 submitted 10 April, 2022;
originally announced April 2022.
-
How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR
Authors:
Kazuma Iwamoto,
Tsubasa Ochiai,
Marc Delcroix,
Rintaro Ikeshita,
Hiroshi Sato,
Shoko Araki,
Shigeru Katagiri
Abstract:
It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with single-channel speech enhancement (SE). In this paper, we investigate the causes of ASR performance degradation by decomposing the SE errors using orthogonal projection-based decomposition (OPD). OPD decomposes the SE errors into noise and artifact components. The artifact component is defined as t…
▽ More
It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with single-channel speech enhancement (SE). In this paper, we investigate the causes of ASR performance degradation by decomposing the SE errors using orthogonal projection-based decomposition (OPD). OPD decomposes the SE errors into noise and artifact components. The artifact component is defined as the SE error signal that cannot be represented as a linear combination of speech and noise sources. We propose manually scaling the error components to analyze their impact on ASR. We experimentally identify the artifact component as the main cause of performance degradation, and we find that mitigating the artifact can greatly improve ASR performance. Furthermore, we demonstrate that the simple observation adding (OA) technique (i.e., adding a scaled version of the observed signal to the enhanced speech) can monotonically increase the signal-to-artifact ratio under a mild condition. Accordingly, we experimentally confirm that OA improves ASR performance for both simulated and real recordings. The findings of this paper provide a better understanding of the influence of SE errors on ASR and open the door to future research on novel approaches for designing effective single-channel SE front-ends for ASR.
△ Less
Submitted 30 March, 2022; v1 submitted 17 January, 2022;
originally announced January 2022.
-
Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlap** Speech Recognition
Authors:
Hiroshi Sato,
Tsubasa Ochiai,
Marc Delcroix,
Keisuke Kinoshita,
Naoyuki Kamo,
Takafumi Moriya
Abstract:
The combination of a deep neural network (DNN) -based speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end is a widely used approach to implement overlap** speech recognition. However, the SE front-end generates processing artifacts that can degrade the ASR performance. We previously found that such performance degradation can occur even under fully overlap** co…
▽ More
The combination of a deep neural network (DNN) -based speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end is a widely used approach to implement overlap** speech recognition. However, the SE front-end generates processing artifacts that can degrade the ASR performance. We previously found that such performance degradation can occur even under fully overlap** conditions, depending on the signal-to-interference ratio (SIR) and signal-to-noise ratio (SNR). To mitigate the degradation, we introduced a rule-based method to switch the ASR input between the enhanced and observed signals, which showed promising results. However, the rule's optimality was unclear because it was heuristically designed and based only on SIR and SNR values. In this work, we propose a DNN-based switching method that directly estimates whether ASR will perform better on the enhanced or observed signals. We also introduce soft-switching that computes a weighted sum of the enhanced and observed signals for ASR input, with weights given by the switching model's output posteriors. The proposed learning-based switching showed performance comparable to that of rule-based oracle switching. The soft-switching further improved the ASR performance and achieved a relative character error rate reduction of up to 23 % as compared with the conventional method.
△ Less
Submitted 11 January, 2022;
originally announced January 2022.
-
Long-Term Optimal Delivery Planning for Replacing the Liquefied Petroleum Gas Cylinder
Authors:
Akihiro Yoshida,
Haruki Sato,
Shiori Uchiumi,
Nariaki Tateiwa,
Daisuke Kataoka,
Akira Tanaka,
Nozomi Hata,
Yousuke Yatsushiro,
Ayano Ide,
Hiroki Ishikura,
Shingo Egi,
Miyu Fujii,
Hiroki Kai,
Katsuki Fujisawa
Abstract:
In the daily operation of liquefied petroleum gas service, gas providers visit customers and replace cylinders if the gas is about to run out. For a long time, frequent visits to customers were required because they could not determine the amount of remaining gas without a staff visit and observation. To solve this problem, smart meters are started to be employed to acquire gas consumption more fr…
▽ More
In the daily operation of liquefied petroleum gas service, gas providers visit customers and replace cylinders if the gas is about to run out. For a long time, frequent visits to customers were required because they could not determine the amount of remaining gas without a staff visit and observation. To solve this problem, smart meters are started to be employed to acquire gas consumption more frequently without visiting customers. In this study, we construct a system to optimize plans for cylinder replacement, and evaluate it with a large-scale field test. We propose an algorithm to create a replacement plan with three steps: estimating the replacement date, acquiring the customer list for replacement, and determining the delivery route. A more accurate estimation of the replacement date can be acquired with a smart meter, which is used for making a customer list for replacement. The formulation for making a customer list enables the gas provider to replace cylinders some days before the date when the gas would run out. It can suppress the concentration of replacements on certain days. Large-scale verification experiments were performed with more than 1,000 customers in Chiba prefecture in Japan. In the field test, the gas provider incorporated the system into its replacement operations. Moreover, the replacement plans developed by the proposed system were compared with that by the gas provider. Our system reduced the number of gas cylinders with gas shortage, the number of visits without replacement due to plenty of gas remaining, and the working duration per customer, which shows that our system benefits both gas providers and customers.
△ Less
Submitted 20 June, 2022; v1 submitted 22 December, 2021;
originally announced December 2021.
-
New metal-plastic hybrid additive manufacturing strategy: Fabrication of arbitrary metal-patterns on external and even internal surfaces of 3D plastic structures
Authors:
Kewei Song,
Yue Cui,
Tiannan Tao,
Xiangyi Meng,
Michinari Sone,
Masahiro Yoshino,
Shinjiro Umezu,
Hirotaka Sato
Abstract:
Constructing precise micro-nano metal patterns on complex three-dimensional (3D) plastic parts allows the fabrication of functional devices for advanced applications. However, this patterning is currently expensive and requires complex processes with long manufacturing lead time. The present work demonstrates a process for the fabrication of micro-nano 3D metal-plastic composite structures with ar…
▽ More
Constructing precise micro-nano metal patterns on complex three-dimensional (3D) plastic parts allows the fabrication of functional devices for advanced applications. However, this patterning is currently expensive and requires complex processes with long manufacturing lead time. The present work demonstrates a process for the fabrication of micro-nano 3D metal-plastic composite structures with arbitrarily complex shapes. In this approach, a light-cured resin is modified to prepare an active precursor capable of allowing subsequent electroless plating (ELP). A multi-material digital light processing 3D printer was newly developed to enable the fabrication of parts containing regions made of either standard resin or active precursor resin nested within each other. Selective 3D ELP processing of such parts provided various metal-plastic composite parts having complicated hollow micro-nano structures with specific topological relationships on a size scale as small as 40 um. Using this technique, 3D metal topologies that cannot be manufactured by traditional methods are possible, and metal patterns can be produced inside plastic parts as a means of further miniaturizing electronic devices. The proposed method can also generate metal coatings exhibiting improved adhesion of metal to plastic substrate. Based on this technique, several sensors composed of different functional nonmetallic materials and specific metal patterns were designed and fabricated. The present results demonstrate the viability of the proposed method and suggest potential applications in the fields of smart 3D micro-nano electronics, 3D wearable devices, micro/nano-sensors, and health care.
△ Less
Submitted 21 December, 2021;
originally announced December 2021.
-
Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlap** Speech Recognition
Authors:
Hiroshi Sato,
Tsubasa Ochiai,
Marc Delcroix,
Keisuke Kinoshita,
Takafumi Moriya,
Naoyuki Kamo
Abstract:
Although recent advances in deep learning technology improved automatic speech recognition (ASR), it remains difficult to recognize speech when it overlaps other people's voices. Speech separation or extraction is often used as a front-end to ASR to handle such overlap** speech. However, deep neural network-based speech enhancement can generate `processing artifacts' as a side effect of the enha…
▽ More
Although recent advances in deep learning technology improved automatic speech recognition (ASR), it remains difficult to recognize speech when it overlaps other people's voices. Speech separation or extraction is often used as a front-end to ASR to handle such overlap** speech. However, deep neural network-based speech enhancement can generate `processing artifacts' as a side effect of the enhancement, which degrades ASR performance. For example, it is well known that single-channel noise reduction for non-speech noise (non-overlap** speech) often does not improve ASR. Likewise, the processing artifacts may also be detrimental to ASR in some conditions when processing overlap** speech with a separation/extraction method, although it is usually believed that separation/extraction improves ASR. In order to answer the question `Do we always have to separate/extract speech from mixtures?', we analyze ASR performance on observed and enhanced speech at various noise and interference conditions, and show that speech enhancement degrades ASR under some conditions even for overlap** speech. Based on these findings, we propose a simple switching algorithm between observed and enhanced speech based on the estimated signal-to-interference ratio and signal-to-noise ratio. We demonstrated experimentally that such a simple switching mechanism can improve recognition performance when processing artifacts are detrimental to ASR.
△ Less
Submitted 2 June, 2021;
originally announced June 2021.
-
Multimodal Attention Fusion for Target Speaker Extraction
Authors:
Hiroshi Sato,
Tsubasa Ochiai,
Keisuke Kinoshita,
Marc Delcroix,
Tomohiro Nakatani,
Shoko Araki
Abstract:
Target speaker extraction, which aims at extracting a target speaker's voice from a mixture of voices using audio, visual or locational clues, has received much interest. Recently an audio-visual target speaker extraction has been proposed that extracts target speech by using complementary audio and visual clues. Although audio-visual target speaker extraction offers a more stable performance than…
▽ More
Target speaker extraction, which aims at extracting a target speaker's voice from a mixture of voices using audio, visual or locational clues, has received much interest. Recently an audio-visual target speaker extraction has been proposed that extracts target speech by using complementary audio and visual clues. Although audio-visual target speaker extraction offers a more stable performance than single modality methods for simulated data, its adaptation towards realistic situations has not been fully explored as well as evaluations on real recorded mixtures. One of the major issues to handle realistic situations is how to make the system robust to clue corruption because in real recordings both clues may not be equally reliable, e.g. visual clues may be affected by occlusions. In this work, we propose a novel attention mechanism for multi-modal fusion and its training methods that enable to effectively capture the reliability of the clues and weight the more reliable ones. Our proposals improve signal to distortion ratio (SDR) by 1.0 dB over conventional fusion mechanisms on simulated data. Moreover, we also record an audio-visual dataset of simultaneous speech with realistic visual clue corruption and show that audio-visual target speaker extraction with our proposals successfully work on real data.
△ Less
Submitted 2 February, 2021;
originally announced February 2021.
