Patent application EP20181464A (EP), filed 2020-06-22
Publication number: EP3929814A1
Title: MAKING TIME-SERIES PREDICTIONS USING A TRAINED DECODER MODEL
Abstract: The invention relates to a computer-implemented prediction method (900) of making time-series predictions for controlling and/or monitoring a computer-controlled system, e.g., a semi-autonomous vehicle. The method uses a time series of one or more observed states. A state comprises values of measurable quantities of multiple interacting objects. Based on the observed states, values of time-invariant latent features for the multiple objects are determined, for example, according to an encoder model. A decoder model is then used to predict at least one next state. This involves applying a trained graph model to obtain a first prediction contribution based on an object's interactions with other objects, and applying a trained function to obtain a second prediction contribution based just on information about the object itself. Based on the predicted next state, output data is generated for use in controlling and/or monitoring the computer-controlled system.
IPC codes: B60W  50/    00            A I
G05D   1/    00            A I
G06N   3/    04            A I
G06N   3/    08            A I
G06V  10/   776            A I
G06V  10/    84            A I
Claims: number: 1
text: A computer-implemented prediction method (900) of making time-series predictions of multiple interacting objects for controlling and/or monitoring a computer-controlled system, wherein a respective object is part of the computer-controlled system or its environment, the method comprising:
number: 2
text: The prediction method (900) of claim 1, wherein the values of the one or more time-invariant latent features for an object are determined according to a trained encoder model, said determining comprising applying a trained graph model of the encoder model based on pairwise contributions of the object with multiple further objects, a pairwise contribution of the object with a further object being determined from the values of the one or more measurable quantities of the object and the further object comprised in the one or more observed states.
number: 3
text: The prediction method (600) of claim 2, wherein a time-invariant latent feature is a discrete feature having a finite set of possible values, said determining the time-invariant latent feature comprising discretizing an output of the trained encoder model to the finite set.
number: 4
text: The prediction method (900) of any preceding claim, wherein the one or more measurable quantities comprise one or more of a position, a velocity, and an acceleration of the object.
number: 5
text: The prediction method (900) of claim 4, wherein the computer-controlled system is an autonomous or semi-autonomous vehicle, the multiple objects comprising multiple traffic participants in an environment of the vehicle.
number: 6
text: A computer-implemented training method (1000) of training a decoder model for making time-series predictions of multiple interacting objects relating to a computer-controlled system, wherein a respective object is part of the computer-controlled system or its environment, the method comprising:
number: 7
text: The method (1000) of claim 6, further comprising training an encoder model for determining, from one or more observed states of the multiple interacting objects, the values of the one or more time-invariant latent features for an object, said determining comprising applying a trained graph model of the encoder model based on pairwise contributions of an object with multiple further objects, a pairwise contribution of the object with a further object being determined from the values of the one or more measurable quantities of the object and the further object comprised in the one or more observed states.
number: 8
text: The method (1000) of claim 7, wherein training the decoder model comprises jointly learning, as trainable parameters, values of one or more time-invariant latent features for an object of the observed scene; the method further comprising using the learned values of the one or more time-invariant latent features to perform supervised training of the encoder model.
number: 9
text: The method (1000) of claim 8, wherein the training data comprises values of one or more time-invariant latent features for a subset of objects and/or time-invariant latent features and/or observed scenes.
number: 10
text: A computer-implemented analysis method (1100) of analysing observations of multiple interacting objects for controlling and/or monitoring a computer-controlled system, wherein a respective object is part of the computer-controlled system or its environment, the method comprising:
number: 11
text: The method (1100) of claim 10, wherein generating the output data comprises applying a further machine leaning model based on the values of said time-invariant latent features.
number: 12
text: A prediction system (200) for making time-series predictions of multiple interacting objects for controlling and/or monitoring a computer-controlled system, wherein a respective object is part of the computer-controlled system or its environment, the system comprising:
number: 13
text: A training system (100) for training a decoder model (041) for making time-series predictions of multiple interacting objects relating to a computer-controlled system, wherein a respective object is part of the computer-controlled system or its environment, the system comprising:
number: 14
text: An analysis system (200) for analysing observations of multiple interacting objects for controlling and/or monitoring a computer-controlled system, wherein a respective object is part of the computer-controlled system or its environment, the system comprising:
number: 15
text: A computer-readable medium (1200) comprising transitory or non-transitory data (1210) representing one or more of:
Office action: search_report, dated 2021-12-29
Rejection reasons: novelty_epc_54
Prior art cited: ref_id: - T. KIPF ET AL., NEURAL RELATIONAL INFERENCE FOR INTERACTING SYSTEMS, Retrieved from the Internet <URL:https://arxiv.org/abs/1802.04687>
ref_type: npl
severity: background
ref_id: - F. SCARSELLI ET AL.: "The graph neural network model", IEEE TRANSACTIONS ON NEURAL NETWORKS, 2009
ref_type: npl
severity: background
ref_id: - P. BATTAGLIA, INTERACTION NETWORKS FOR LEARNING ABOUT OBJECTS, RELATIONS AND PHYSICS, Retrieved from the Internet <URL:https://arxiv.org/abs/1612.00222>
ref_type: npl
severity: background
ref_id: - WANG MINGKUN ET AL: "Unsupervised Pedestrian Trajectory Prediction with Graph Neural Networks", 2019 IEEE 31ST INTERNATIONAL CONFERENCE ON TOOLS WITH ARTIFICIAL INTELLIGENCE (ICTAI), IEEE, 4 November 2019 (2019-11-04), pages 832 - 839, XP033713808, DOI: 10.1109/ICTAI.2019.00119
ref_type: npl
severity: background
ref_id: - DONSUK LEE ET AL: "Joint Interaction and Trajectory Prediction for Autonomous Driving using Graph Neural Networks", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 17 December 2019 (2019-12-17), XP081560106
ref_type: npl
severity: background
ref_id: - BLAIOTTA CLAUDIA: "Learning Generative Socially Aware Models of Pedestrian Motion", IEEE ROBOTICS AND AUTOMATION LETTERS, IEEE, vol. 4, no. 4, 1 October 2019 (2019-10-01), pages 3433 - 3440, XP011736646, DOI: 10.1109/LRA.2019.2928202
ref_type: npl
severity: background
ref_id: - WANG MINGKUN ET AL, "Unsupervised Pedestrian Trajectory Prediction with Graph Neural Networks", 2019 IEEE 31ST INTERNATIONAL CONFERENCE ON TOOLS WITH ARTIFICIAL INTELLIGENCE (ICTAI), IEEE, (20191104), doi:10.1109/ICTAI.2019.00119, pages 832 - 839, XP033713808 [X] 1-2,4-7,10-15 * figure 3 * * section III * [I] 3,8-9
ref_type: npl
severity: novelty_destroying
ref_id: - DONSUK LEE ET AL, "Joint Interaction and Trajectory Prediction for Autonomous Driving using Graph Neural Networks", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (20191217), XP081560106 [X] 1,4-6,12-13,15 * abstract * * figure 2 * * section 1 * * section 3 * * section 2 * * section 4 * [I] 2-3,7-11,14
ref_type: npl
severity: novelty_destroying
ref_id: - BLAIOTTA CLAUDIA, "Learning Generative Socially Aware Models of Pedestrian Motion", IEEE ROBOTICS AND AUTOMATION LETTERS, IEEE, (20191001), vol. 4, no. 4, doi:10.1109/LRA.2019.2928202, pages 3433 - 3440, XP011736646 [A] 1-15 * the whole document *
ref_type: npl
severity: background
Final disposition: pending (2021-12-29)