Patent application EP19175238A (EP), filed 2019-05-17
Publication number: EP3739570A1
Title: ATTENTION-BASED NEURAL SEQUENCE TO SEQUENCE MAPPING APPLIED TO SPEECH SYNTHESIS AND VOCAL TRANSLATION
Abstract: [0001]    A computer implemented method of sequence to sequence data processing, comprising:
inputting a first input comprising a first input data sequence into a model, the model outputting a first output data sequence, a first part of the model generating an intermediate state comprising information relating to an alignment relationship between the first input data sequence and the first output data sequence, the intermediate state being used in the model to generate the first output data sequence;
storing the intermediate state;
modifying the model to replace the first part with the stored intermediate state;
inputting a second input comprising a second input data sequence into the modified model, the modified model outputting a second output data sequence using the intermediate state.
IPC codes: G10L  13/    02            A I
G06F  40/    40            A I
G10L  15/    26            A I
Claims: number: 1
text: A computer implemented method of sequence to sequence data processing, comprising:
number: 2
text: The method according to claim 1, wherein the second input data sequence has the same length as the first input data sequence.
number: 3
text: The method according to claim 1 or 2 the first part of the model comprises a first trained algorithm, and wherein the first trained algorithm is configured to generate a first set of parameters representing an alignment relationship between the first input data sequence and the first output data sequence, and wherein modifying the model comprises replacing the first trained algorithm with the stored first set of parameters.
number: 4
text: The method according to claim 2 or 3, wherein the first trained algorithm is an attention mechanism.
number: 5
text: The method according to any preceding claim, wherein the input data sequence comprises a sequence of text related units and the output data sequence comprises a sequence representing speech frames.
number: 6
text: The method according to claim 5, wherein the sequence of text related units is extracted from an input text signal and wherein an output speech signal is generated from the sequence representing speech frames.
number: 7
text: The method according to claim 5 or 6, wherein the first input further comprises a first characteristic vector and the second input further comprises a second characteristic vector.
number: 8
text: The method according to claim 7, wherein the first characteristic vector comprises a set of features corresponding to a first speaker and the second characteristic vector comprises a set of features corresponding to a second speaker.
number: 9
text: The method according to any preceding claim, wherein an input to the first part of the model for a current time step comprises information generated from a previous time step.
number: 10
text: The method according to claim 3, wherein an input to the first part of the model for a current time step comprises information generated from a previous time step, wherein the model further comprises a first neural network, and wherein the first trained algorithm is configured to:
number: 11
text: The method according to any of claims 2 to 10, further comprising:
number: 12
text: The method according to claim 11, wherein identifying that the second input is a candidate for improving alignment comprises detecting a fault in the alignment relationship between the second input data sequence and the third output data sequence.
number: 13
text: The method according to claim 12, wherein detecting a fault comprises:
number: 14
text: A carrier medium comprising computer readable code configured to cause a computer to perform a method according to any preceding claim.
number: 15
text: A system for sequence to sequence data processing, comprising:
Office action: search_report, dated 2020-11-18
Rejection reasons: novelty_epc_54
inventive_step_epc_56
Prior art cited: ref_id: US2018336880A1
ref_type: patent
severity: background
ref_id: [XYI]     - YUXUAN WANG ET AL: "Tacotron: Towards End-to-End Speech Synthesis", INTERSPEECH 2017, 2017, ISCA, pages 4006 - 4010, XP055633013, DOI: 10.21437/Interspeech.2017-1452
ref_type: npl
severity: background
ref_id: &#91;XYI]     - JOSE SOTELO ET AL: "Workshop track -ICLR 2017 CHAR2WAV: END-TO-END SPEECH SYNTHESIS", 10 March 2017 (2017-03-10), XP055481275, Retrieved from the Internet <URL:https://pdfs.semanticscholar.org/9203/d6c076bffe87336f2ea91f5851436c02dbe6.pdf> [retrieved on 20180605&#93;
ref_type: npl
severity: background
ref_id: - THANG LUONG ET AL: "Effective Approaches to Attention-based Neural Machine Translation", PROCEEDINGS OF THE 2015 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING, 17 September 2015 (2015-09-17), Stroudsburg, PA, USA, pages 1412 - 1421, XP055435303, DOI: 10.18653/v1/D15-1166
ref_type: npl
severity: background
ref_id: - RJ SKERRY-RYAN ET AL: "Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 24 March 2018 (2018-03-24), XP080862501
ref_type: npl
severity: background
ref_id: US2018336880
ref_type: patent
severity: obviousness
ref_id: - YUXUAN WANG ET AL, "Tacotron: Towards End-to-End Speech Synthesis", INTERSPEECH 2017, ISCA, (2017), doi:10.21437/Interspeech.2017-1452, pages 4006 - 4010, XP055633013 [X] 1,3-6,9,10,14,15 * the whole document * [Y] 7,8 [I] 2,11-13
ref_type: npl
severity: novelty_destroying
ref_id: - Jose Sotelo ET AL, "Workshop track -ICLR 2017 CHAR2WAV: END-TO-END SPEECH SYNTHESIS", (20170310), URL: https://pdfs.semanticscholar.org/9203/d6c076bffe87336f2ea91f5851436c02dbe6.pdf, (20180605), XP055481275 [X] 1,3-6,9,10,14,15 * section 3.1 * [Y] 7,8 [I] 2,11-13
ref_type: npl
severity: novelty_destroying
ref_id: - THANG LUONG ET AL, "Effective Approaches to Attention-based Neural Machine Translation", PROCEEDINGS OF THE 2015 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING, Stroudsburg, PA, USA, (20150917), doi:10.18653/v1/D15-1166, pages 1412 - 1421, XP055435303 [X] 1,3,4,9,10,14,15 * section 1-3; figures 2-4 *
ref_type: npl
severity: novelty_destroying
ref_id: - RJ SKERRY-RYAN ET AL, "Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (20180324), XP080862501 [Y] 7,8 * section 3; figures 1,2 *
ref_type: npl
severity: obviousness
Final disposition: pending (2020-11-18)