Bypassing Optimization Complexity through Transfer Learning & Deep Neural Nets for Speech Intelligibility Improvement (2021)

Abstract: This extended abstract highlights the research ventures and findings in the domain of speech intelligibility improvement. Till this point, an effort has been to simulate the Lombard effect, which is the deliberate human attempt to make a speech more intelligible when speaking in the presence of interfering background noise. To that end, an attempt has been made to shift the formants away from the noisy regions in spectrum both sub-optimally and optimally. The sub-optimal shifting methods were based upon Kalman filtering and EM approach. The optimal shifting involved the use of optimization to maximize an objective intelligibility index after shifting the formants. A transfer learning framework was also set up to bring down the computational complexity.

## motivation of research while much of the research focus has been on improvement in quality of speech signals, certain applications call for proper intelligibility of the speech rather than how pleasing it is to the listener. thus, the prime motivation of the current research is to ensure that information is not lost to noise and is communicated in a robust manner, especially at very low snr levels. ## key issues; identified and addressed the most common causative factor of loss in speech intelligibility is the presence of background noise. when noise occupies the same regions of the spectrum as speech, the intelligibility falls drastically. one solution is to shift the formants away from the noisy regions in the spectrum. the result would be that the information content in those formants would also be shifted away from the noise and would thus cease to be afflicted by it. the details of formant shifting are given in @xcite . ## major contributions following have been the major contributions so far: ## methodologies the foundation of the current work lies in shifting the formants away from noisy regions in speech. to that end, the mathematical representation of this shifting through a gaussian delta function can be represented as shown: the subscripts 's' & 't' denote the source and target transformation functions respectively, and the subscripts 'sn' & 'tn' denote the gaussian approximation of the magnitude spectra of the two noises being compared. f tavg and f savg are the average formant frequencies of the target and source languages respectively. the modification of mean is contributed to by both language and noise transfer. the modification in standard deviation is controlled by noise transfer alone. ## experiments and results the pipeline of the experiments conducted started with the generation of a trapezoidal delta function through clpso for a certain combination of language, noise type and snr level. this led to a certain improvement in short time objective intelligibility (stoi) @xcite for that specific combination. thereafter transfer was done across languages and the results were compared with that obtained through direct training through clpso. next, the clpso was used to obtain the gaussian delta function for a combination as discussed before. thereafter, transfer was done across noises, followed by a combination of language and noise. these results can be exemplified by one of the cases of training and transfer, as shown in ## future plans and road-map for thesis the work done up to this point has provided certain insightful results. these results will be used as prior information to train a neural network to develop a fully autonomous system that is scalable in terms of applications. thus, the thesis is projected to be consisting of two parts. the former dealing with intelligibility improvement of speech using basic machine learning and optimization, while the latter handling these issues through the employment of deep neural networks.