Patent application EP16172863A (EP), filed 2016-06-03
Publication number: EP3252769A1
Title: ADDING BACKGROUND SOUND TO SPEECH-CONTAINING AUDIO DATA
Abstract: An editing method (40) facilitates the task of adding background sound to speech-containing audio data so as to augment the listening experience. The editing method (40) is executed by a processor in a computing device and comprises obtaining (41) characterization data that characterizes time segments in the audio data by at least one of topic and sentiment; deriving (43), for a respective time segment in the audio data and based on the characterization data, a desired property of a background sound to be added to the audio data in the respective time segment, and providing (44) the desired property for the respective time segment so as to enable the audio data to be combined, within the respective time segment, with background sound having the desired property. The background sound may be selected and added automatically or by manual user intervention.
IPC codes: G06F  17/    27            A I
G06F  17/    30            A I
G09B   5/    06            A I
G10L  25/    48            A I
G11B  27/   031            A I
Claims: number: 1
text: A method of augmenting speech-containing audio data (A) on a computing device (1, 1') comprising a processor (3, 3'), said method being executed by the processor (3, 3') in the computing device (1, 1') and comprising:
number: 2
text: The method of claim 1, wherein the desired property is derived to match the topic and/or the sentiment of the respective time segment.
number: 3
text: The method of claim 1 or 2, wherein the topic represents substance or theme of spoken language within the respective time segment in the audio data (A).
number: 4
text: The method of any one of the preceding claims, wherein the sentiment represents emotion of spoken language within the respective time segment in the audio data (A) and comprises one or more of: a positive sentiment, a negative sentiment, a neutral sentiment, anger, joy, sadness, surprise, accusation, fear, anxiety, unpleasantness, disgust, relief, desire, love, hate, calm, excitement and suspense.
number: 5
text: The method of any one of the preceding claims, wherein the desired property is derived at least partly based on the sentiment of the respective time segment and is indicative of one or more of: a playback volume of the background sound (B), a tempo of the background sound (B), a tonality of the background sound (B), and a mood of the background sound (B).
number: 6
text: The method of any one of the preceding claims, wherein the characterization data ([D]) further characterizes the time segments in the audio data (A) by literary category (CID), and wherein the desired property is further derived to match the literary category (CID) of the respective time segment.
number: 7
text: The method of any one of the preceding claims, wherein the step of obtaining the characteristic data ([D]) comprises processing the audio data (A) by natural language processing.
number: 8
text: The method of any one of the preceding claims, wherein the step of obtaining the characteristic data comprises: generating text (A_T) corresponding to the audio data (A) by operating a speech recognition algorithm on the audio data (A), associating the text (A_T) with time points in the audio data (A), operating a topic segmentation algorithm and/or a sentiment analysis algorithm on the text (A_T) to determine topics and/or sentiments in the text (A_T), each topic and/or sentiment being determined for a respective text segment in the text (A_T), said method further comprising generating the characterization data ([D]) by determining time segments in the audio data (A) corresponding to the text segments and generating an association between the time segments and topic identifiers (TID) representing the topics and/or sentiment identifiers (SID) representing the sentiments.
number: 9
text: The method of any one of the preceding claims, wherein the step of obtaining characterization data ([D]) comprises processing the audio data (A) for extraction of audio features, analyzing the audio features for determination of a sentiment for at least one time segment the audio data (A), and generating an association between the at least one time segment and a sentiment identifier (SID) representing the sentiment.
number: 10
text: The method of any one of the preceding claims, wherein the audio data (A) is associated with image data to be played in synchronization with the audio data (A), and wherein the step of obtaining characterization data ([D]) comprises analyzing the image data for determination of topics and/or sentiments in the image data, and generating an association between time segments in the audio data (A) and topic identifiers (TID) representing the topics and/or sentiment identifiers (SID) representing the sentiments.
number: 11
text: The method of any one of the preceding claims, wherein the step of providing the desired property comprises: displaying, on a display (6), an option for allocating background sound (B) to the respective time segment together with an indication of the desired property for the respective time segment, receiving a user selection of one or more sound files containing background sound (B) for the respective time segment, and generating augmented audio data (A*) by combining the audio data (A) within the respective time segment with the background sound (B) in the one or more sound files.;12. The method of any one of claims 1-10, further comprising: automatically obtaining background sound (B) having the desired property of the respective time segment, and generating augmented audio data (A*) by combining the audio data (A) within the respective time segment with the background sound (B).
number: 13
text: The method of claim 12, wherein the step of automatically obtaining the background sound (B) comprises at least one of: obtaining one or more sound files from a library of background sound (B) and algorithmically generating the background sound (B).
number: 14
text: A computer-readable medium comprising computer instructions which, when executed by a processor (3), cause the processor (3) to perform the method of any preceding claim.
number: 15
text: A computing device for augmenting speech-containing audio data (A), said computing device being configured to:
Office action: search_report, dated 2017-12-06
Rejection reasons: novelty_epc_54
inventive_step_epc_56
Prior art cited: ref_id: - ARORA ET AL.: "Automatic Speech Recognition: A Review", INTERNATIONAL JOURNAL OF COMPUTER APPLICATIONS, vol. 60, no. 9, 2012, pages 34 - 44
ref_type: npl
severity: background
ref_id: - PORIA ET AL.: "Fusing audio, visual and textual clues for sentiment analysis from multimodal content", NEUROCOMPUTING, vol. 174, 2016, pages 50 - 59
ref_type: npl
severity: background
ref_id: - WEDHAT ET AL.: "Sentiment analysis algorithms and applications: A survey", AIN SHAMS ENGINEERING JOURNAL, vol. 5, 2014, pages 1093 - 1113
ref_type: npl
severity: background
ref_id: - BAMMAN; SMITH: "Contextualized Sarcasm Detection on Twitter", NINTH INTERNATIONAL AAAI CONFERENCE ON WEB AND SOCIAL MEDIA, 2015
ref_type: npl
severity: background
ref_id: EP1666967A1
ref_type: patent
severity: background
ref_id: US2015255066A1
ref_type: patent
severity: background
ref_id: EP1666967
ref_type: patent
severity: novelty_destroying
ref_id: US2015255066
ref_type: patent
severity: obviousness
Final disposition: pending (2017-12-06)