Patent application EP21173831A (EP), filed 2021-05-14
Publication number: EP3910479A1
Title: A METHOD AND A SYSTEM FOR TESTING MACHINE LEARNING AND DEEP LEARNING MODELS FOR ROBUSTNESS, AND DURABILITY AGAINST ADVERSARIAL BIAS AND PRIVACY ATTACKS
Abstract: [0001]    A system for testing Machine Learning (ML) and deep learning models for robustness, and durability against adversarial bias and privacy attacks, comprising a Project Repository for storing metadata of ongoing projects each of which having a defined project policy, and created ML models and data sources being associated with the ongoing projects; a Secure Data Repository, for storing training and testing datasets and models used in each project for evaluating the robustness of the each project; a Data/Model Profiler for creating a profile, based on the settings and configurations of the datasets and the models; a Test Recommendation Engine for recommending the relevant and most indicative attacks/tests for each examined model and for creating indicative and effective test suites; a Test/Attack Ontology module for storing all attacks/tests with their metadata and mapping the attacks/tests to their corresponding settings and configurations; an Attack Repository for storing the implemented tests/attacks. An ML model is tested against each one of the robustness categories (privacy, bias and adversarial learning); a Test Execution Environment for Initializing a test suite, running multiple tests and prioritizing tests in the test suite; a Project/Test Analytics module for analyzing the test suite results and monitoring changes in performance over time; a Defenses Repository for storing implemented defense methods implemented for each robustness category.
IPC codes: G06F  11/    36            A I
G06F  21/    57            A I
G06F  21/    62            A I
G06N  20/    00            A I
G06N   5/    02            A I
Claims: number: 1
text: A system for testing Machine Learning (ML) and deep learning models for robustness, and durability against adversarial bias and privacy attacks, comprising:
number: 2
text: A system according to claim 1, wherein the defined project policy specifies the acceptance criteria for bias, privacy and adversarial learning and defines the minimum robustness score that is required for a model to be accepted and certified.
number: 3
text: A system according to claim 1, wherein a project is completed after its corresponding the ML model is certified to comply with all the constrains of its corresponding policy.
number: 4
text: A system according to claim 1, wherein states of a project are selected from the group consisting of:
number: 5
text: A system according to claim 1, wherein a training dataset is used to induce a ML model and for evaluating the performance of the ML model.
number: 6
text: A system according to claim 1, wherein each attack/test is evaluated relatively to a data source being a training or testing dataset and the evaluation outcome corresponds to the robustness of the model on said data source.
number: 7
text: A system according to claim 1, wherein the Secure Data Repository further comprises a Model Repository for storing model versions that reflect changes in an ML model;
number: 8
text: A system according to claim 1, wherein relevant tests to be executed on an examined model are selected from the group of:
number: 9
text: A system according to claim 1, implemented over a Fronted Management Server being adapted to run the system modules and provide an API access for external command-line interface (CLI) and a frontend User Interface (UI) service that allows performing one or more system operations.
number: 10
text: A system according to claim 1, wherein the system operations include one or more of the following:
Office action: search_report, dated 2021-11-17
Rejection reasons: inventive_step_epc_56
Prior art cited: ref_id: - FREDRIKSON ET AL., FREDRIKSON, JHA, & RISTENPART, MODEL INVERSION ATTACKS THAT EXPLOIT CONFIDENCE INFORMATION AND BASIC COUNTERMEASURES, 15 October 2018 (2018-10-15)
ref_type: npl
severity: background
ref_id: - ATENIESE, G.FELICI, G.MANCINI, L. V.SPOGNARDI, A.VILLANI, A.VITALI, D.: "Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers", ARXIV PREPRINT ARXIV:1306.4447., 19 June 2013 (2013-06-19)
ref_type: npl
severity: background
ref_id: - BARRENO, M.NELSON, B.JOSEPH, A. D.TYGAR, J. D.: "The security of machine learning", MACHINE LEARNING, vol. 8, no. 12, 2010, pages 121 - 148
ref_type: npl
severity: background
ref_id: - CHOLLET, F., KERAS, 2015
ref_type: npl
severity: background
ref_id: - COMMISION, E., EU DATA PROTECTION RULES. (EUROPEAN COMMISION, 2018, Retrieved from the Internet <URL:https://ec.europa.eu/commission/priorities/justice-and-fundamental-rights/data-protection/2018-reform-eu-data-protection-rules/eu-data-protection-rules-en>
ref_type: npl
severity: background
ref_id: - FREDRIKSON, M.JHA, S.RISTENPART, T.: "Model inversion attacks that exploit confidence information and basic countermeasures", PROCEEDINGS OF THE 22ND ACM SIGSAC CONFERENCE ON COMPUTER AND COMMUNICATIONS SECURITY, 2015, pages 1322 - 1333, XP055468313, DOI: 10.1145/2810103.2813677
ref_type: npl
severity: background
ref_id: - FREDRIKSON, M.LANTZ, E.JHA, S.LIN, S.PAGE, D.RISTENPART, T.: "Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing", 23RD {USENIX} SECURITY SYMPOSIUM ({USENIX} SECURITY 14, pages 17 - 32
ref_type: npl
severity: background
ref_id: - MEHRABI, N.MORSTATTER, F.SAXENA, N.LERMAN, K.GALSTYAN, A.: "A Survey on Bias and Fairness in Machine Learning", ARXIV PREPRINT ARXIV:1908.09635, 23 August 2019 (2019-08-23)
ref_type: npl
severity: background
ref_id: - PAPERNOT, N.MCDANIEL, P.SINHA, A.WELLMAN , M. P.: "In 2018 IEEE European Symposium on Security and Privacy (EuroS&P", 24 April 2018, IEEE, article "SoK: Security and privacy in machine learning", pages: 399 - 414
ref_type: npl
severity: background
ref_id: - PFLEEGER, S. L.P., P. C.: "Analyzing Computer Security: A Threat/Vulnerability/Countermeasure Approach", PRENTICE HALL PROFESSIONAL, 2012
ref_type: npl
severity: background
ref_id: - SHOKRI, R.STRONATI, M.SONG, C.SHMATIKOV, V.: "Membership inference attacks against machine learning models", 2017 IEEE SYMPOSIUM ON SECURITY AND PRIVACY (SP, 2017, pages 3 - 18, XP033108084, DOI: 10.1109/SP.2017.41
ref_type: npl
severity: background
ref_id: - SZEGEDY, C.ZAREMBA, W.SUTSKEVER, I.BRUNA, J.ERHAN, D.GOODFELLOW, I.FERGUS, R., INTRIGUING PROPERTIES OF NEURAL NETWORKS. ARXIV PREPRINT ARXIV:1312.6199, 21 December 2013 (2013-12-21)
ref_type: npl
severity: background
ref_id: - TRAMER, F.ZHANG, F.JUELS, A.REITER, M. K.RISTENPART, T.: "Stealing machine learning models via prediction apis", 25TH {USENIX} SECURITY SYMPOSIUM ({USENIX} SECURITY 16, 2016, pages 601 - 618
ref_type: npl
severity: background
ref_id: - VAROQUAUX, G.BUITINCK, L.LOUPPE, G.GRISEL, O.PEDREGOSA, F.MUELLER, A.: "Scikit-learn: Machine learning without learning the machinery", GETMOBILE: MOBILE COMPUTING AND COMMUNICATIONS, vol. 19, no. 1, 2015, pages 29 - 33
ref_type: npl
severity: background
ref_id: - VEALE, M.BINNS, R.EDWARDS, L.: "Algorithms that remember: model inversion attacks and data protection law", PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY A: MATHEMATICAL, PHYSICAL AND ENGINEERING SCIENCES, vol. 376, no. 2133, 15 October 2018 (2018-10-15), pages 20180083
ref_type: npl
severity: background
ref_id: US2016132787A1
ref_type: patent
severity: background
ref_id: US2020082097A1
ref_type: patent
severity: background
ref_id: - SATO D. ET AL: "Continuous delivery for machine learning", 19 September 2019 (2019-09-19), XP055839738, Retrieved from the Internet <URL:https://web.archive.org/web/20200503222841/https://martinfowler.com/articles/cd4ml.html> [retrieved on 20210909]
ref_type: npl
severity: background
ref_id: - VINAY SRIDHAR SRIRAM SUBRAMANIAN SWAMINATHAN SUNDARARAMAN DULCARDO ARTEAGA DREW ROSELLI PARALLELM NISHA TALAGALA: "Model Governance: Reducing the Anarchy of Production ML", 11 July 2018 (2018-07-11), pages 362 - 368, XP061025639, Retrieved from the Internet <URL:https://www.usenix.org/sites/default/files/atc18_full_proceedings_interior.pdf> [retrieved on 20180711]
ref_type: npl
severity: background
ref_id: US2016132787
ref_type: patent
severity: obviousness
ref_id: US2020082097
ref_type: patent
severity: obviousness
ref_id: - Sato D. ET AL, "Continuous delivery for machine learning", (20190919), URL: https://web.archive.org/web/20200503222841/https://martinfowler.com/articles/cd4ml.html, (20210909), XP055839738 [A] 1-10 * page  2  - page  30 *
ref_type: npl
severity: background
ref_id: - VINAY SRIDHAR SRIRAM SUBRAMANIAN SWAMINATHAN SUNDARARAMAN DULCARDO ARTEAGA DREW ROSELLI PARALLELM NISHA TALAGALA, "Model Governance: Reducing the Anarchy of Production ML", (20180711), pages 362 - 368, USENIX, USENIX, THE ADVANCED COMPUTING SYSTEMS ASSOCIATION, URL: https://www.usenix.org/sites/default/files/atc18_full_proceedings_interior.pdf, (20180711), XP061025639 [A] 1-10 * figure 1 *
ref_type: npl
severity: background
Final disposition: pending (2021-11-17)