Patent application EP20315288A (EP), filed 2020-05-29
Publication number: EP3916556A1
Title: METHOD AND SYSTEM FOR DETECTING ANOMALIES IN A DATA PIPELINE
Abstract: A method and system for detecting anomalies in a data pipeline are disclosed. Data tables are extracted from data pipelines. A best time column is calculated for each data table. Table lines are aggregated so as to establish counts of data lines per a time interval, and corresponding sequences of points are produced that are being analysed through regression methods. An anomaly is raised when points have counts of data lines that lie outside confidence intervals around expected values for such counts.
IPC codes: G06F  11/    07            A I
G06F  17/    18            A I
G06F  11/    34            A I
G06N  20/    00            A I
Claims: number: 1
text: A method for detecting anomalies in a data pipeline, wherein a Reader (202) extracts data tables from a Source Dataset (201) at a node of the data pipeline, the method comprising the steps of:
number: 2
text: The method of claim 1, wherein the calculating by an Anomaly Detector (311) a confidence interval for a first selection of time intervals in the third sequence comprises:
number: 3
text: The method of claim 2, wherein the calculating by the Anomaly Detector (311) a second error comprises:
number: 4
text: The method of any of the claims 1 to 3, wherein the calculating by the Breakpoint Finder (304) a first regression for the second sequence of M points comprises:
number: 5
text: The method of any of the claims 1 to 4, further comprising calculating by the Breakpoint Finder (304) an updated best breakpoint for the updated first data table, comprising:
number: 6
text: The method of any of the claims 1 to 5 further comprising performing by the Table Analyser (301), after the retaining the best time column for the first data table:
number: 7
text: A system for detecting anomalies in a data pipeline, comprising:
number: 8
text: The system of claim 7, wherein the Anomaly Detector (311) is further configured to calculate a confidence interval for a first selection of time intervals in the third sequence by:
number: 9
text: The system of claim 8, wherein the Anomaly Detector (311) is further configured to calculate the second error by:
number: 10
text: The system of any of the claims 7 to 9, wherein the Breakpoint Finder (304) is further configured to calculate a first regression for the second sequence of M points by:
number: 11
text: The system of any of the claims 7 to 10, wherein the Breakpoint Finder (304) is further configured to calculate an updated best breakpoint for the updated first data table by:
number: 12
text: The system of any of the claims 7 to 11 wherein the Table Analyser (301) is further configured to perform after the retaining the best time column for the first data table:
Office action: search_report, dated 2021-12-01
Prior art cited: ref_id: EP3258426A1
ref_type: patent
severity: background
ref_id: - SEJUN KIM ET AL: "Fast Detection of Abnormal Data in IIoT with Segmented Linear Regression", PROCEEDINGS ON THE INTERNATIONAL CONFERENCE ON INTERNET COMPUTING (ICOMP), 1 January 2019 (2019-01-01), Athens, pages 88 - 94, XP055746426
ref_type: npl
severity: background
ref_id: - ANONYMOUS: "Excel Tutorial on Linear Regression", 28 May 2002 (2002-05-28), XP055746938, Retrieved from the Internet <URL:http://science.clemson.edu/physics/labs/tutorials/excel/regression.html> [retrieved on 20201104]
ref_type: npl
severity: background
ref_id: EP3258426
ref_type: patent
severity: background
ref_id: - SEJUN KIM ET AL, "Fast Detection of Abnormal Data in IIoT with Segmented Linear Regression", PROCEEDINGS ON THE INTERNATIONAL CONFERENCE ON INTERNET COMPUTING (ICOMP), Athens, (20190101), pages 88 - 94, XP055746426 [A] 1-12 * abstract * * page 90, right column, last paragraph *
ref_type: npl
severity: background
ref_id: - Anonymous, "Excel Tutorial on Linear Regression", (20020528), URL: http://science.clemson.edu/physics/labs/tutorials/excel/regression.html, (20201104), XP055746938 [A] 1-12 * page 1, 2 *
ref_type: npl
severity: background
Final disposition: pending (2021-12-01)