jerpint commited on
Commit
17b7b0e
β€’
0 Parent(s):

Deploy app

Browse files
LICENSE.md ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2023 MILA - Institut quebecois d'intelligence artificielle
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: AIR πŸ’¬
3
+ emoji: 🌎
4
+ colorFrom: pink
5
+ colorTo: green
6
+ sdk: gradio
7
+ sdk_version: 4.36.1
8
+ app_file: src/buster/gradio_app.py
9
+ python: 3.11
10
+ pinned: false
11
+ ---
data/documents_metadata.csv ADDED
@@ -0,0 +1,431 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Source,Title,Year,Country,Link
2
+ LimeSurvey v2,Strategy for Augmenting Intelligence Using Machines AIM,2019,United States,https://www.dni.gov/files/ODNI/documents/AIM-Strategy.pdf
3
+ LimeSurvey v2,TRIBUNALE ORDINARIO di BOLOGNA: Sezione Lavoro,2022,Italy,https://www.bollettinoadapt.it/wp-content/uploads/2021/01/Ordinanza-Bologna.pdf
4
+ LimeSurvey v2,BOLETÍN OFICIAL DEL ESTADO,2023,Spain,https://www.boe.es/boe/dias/2023/06/03/pdfs/BOE-A-2023-13313.pdf
5
+ LimeSurvey v2,US Air Force AI Annex to the Department of Defense AI Strategy,2019,United States,https://www.af.mil/Portals/1/documents/5/USAF-AI-Annex-to-DoD-AI-Strategy.pdf
6
+ LimeSurvey v2,Australias Artificial Intelligence Action Plan,2021,Australia,https://wp.oecd.ai/app/uploads/2021/12/Australia_AI_Action_Plan_2021.pdf
7
+ LimeSurvey v2,Office of National Higher Education Science Research and Innovation Policy Council,2020,Thailand,https://waa.inter.nstda.or.th/stks/pub/2020/20200717-conference-ethics.pdf
8
+ LimeSurvey v2,National Strategy for Critical and Emerging Technologies,2020,United States,https://trumpwhitehouse.archives.gov/wp-content/uploads/2020/10/National-Strategy-for-CET.pdf
9
+ LimeSurvey v2,White House Summit on AI in Government,2019,United States,https://trumpwhitehouse.archives.gov/wp-content/uploads/2019/09/Summary-of-White-House-Summit-on-AI-in-Government-September-2019.pdf
10
+ LimeSurvey v2,White House Summit on AI for American Industry,2022,United States,https://trumpwhitehouse.archives.gov/wp-content/uploads/2018/05/Summary-Report-of-White-House-AI-Summit.pdf
11
+ LimeSurvey v2,Knowledge and Innovation Agenda,2021,Netherlands,https://topsectoragrifood.nl/wp-content/uploads/2018/04/Kennis-en-innovatieagenda-EN.pdf
12
+ LimeSurvey v2,Spanish RDI Strategy In Artificial Intelligence,2021,Spain,https://www.ciencia.gob.es/dam/jcr:5af98ba2-166c-4e63-9380-4f3f68db198e/Estrategia_Inteligencia_Artificial_IDI.pdf
13
+ LimeSurvey v2,Data Protection Ombudsman Fines Kymen Vesi Oy,2022,Finland,https://tietosuoja.fi/documents/6927448/22406974/Ty%C3%B6ntekij%C3%B6iden+sijaintitietojen+k%C3%A4sittely+ja+vaikutustenarviointi.pdf/2d04e545-d427-8a0d-3f4d-967de7b428ac/Ty%C3%B6ntekij%C3%B6iden+sijaintitietojen+k%C3%A4sittely+ja+vaikutustenarviointi.pdf
14
+ LimeSurvey v2,2020 Federal Data Strategy Action Plan,2022,United States,https://strategy.data.gov/assets/docs/2020-federal-data-strategy-action-plan.pdf
15
+ LimeSurvey v2,ARTIFICIAL INTELLIGENCE FOR SOCIAL GOOD IN LATIN AMERICA AND THE CARIBBEAN: The Regional Landscape and 12 Country Snapshots,2021,Argentina,https://publications.iadb.org/publications/english/document/Artificial-Intelligence-for-Social-Good-in-Latin-America-and-the-Caribbean-The-Regional-Landscape-and-12-Country-Snapshots.pdf
16
+ LimeSurvey v2,Global Partnership on AI,2023,Serbia,https://prosveta.gov.rs/wp-content/uploads/2021/12/akcioni-plan-strategija-vestacke-inteligencije.pdf
17
+ LimeSurvey v2,Public tender for the technical work to be performed within the programme,2023,Spain,https://portal.mineco.gob.es/RecursosNoticia/mineco/prensa/noticias/2022/20221213_plan_algoritmos_verdes.pdf
18
+ LimeSurvey v2,Mauritius AI Council,2022,Mauritius,https://ncb.govmu.org/ncb/strategicplans/MauritiusAIStrategy2018.pdf
19
+ LimeSurvey v2,POLÍTICA NACIONAL DE INTELIGENCIA ARTIFICIAL,2019,Chile,https://minciencia.gob.cl/uploads/filer_public/bc/38/bc389daf-4514-4306-867c-760ae7686e2c/documento_politica_ia_digital_.pdf
20
+ LimeSurvey v2,Egypts National AI Strategy,2021,Egypt,https://mcit.gov.eg/Upcont/Documents/Publications_672021000_Egypt-National-AI-Strategy-English.pdf
21
+ LimeSurvey v2,Action Plan,2022,Turkey,https://inhak.adalet.gov.tr/Resimler/SayfaDokuman/1262021081047Action_Plan_On_Human_Rights.pdf
22
+ LimeSurvey v2,Kenyas Digital Economy Strategy,2021,Kenya,https://ict.go.ke/wp-content/uploads/2020/08/10TH-JULY-FINAL-COPY-DIGITAL-ECONOMY-STRATEGY-DRAFT-ONE.pdf
23
+ LimeSurvey v2,Federal Data Strategy,2022,United States,https://strategy.data.gov/assets/docs/2020-federal-data-strategy-framework.pdf
24
+ LimeSurvey v2,National Initiative for Secured Intelligent Systems,2022,Israel,https://icrc.tau.ac.il/sites/cyberstudies-english.tau.ac.il/files/media_server/cyber%20center/The%20National%20Initiative_eng%202021_digital.pdf
25
+ LimeSurvey v2,National AI Initiative Act of 2020,2021,United States,https://www.congress.gov/116/crpt/hrpt617/CRPT-116hrpt617.pdf#page=1210
26
+ LimeSurvey v2,A legal framework for artificial intelligence,2022,Switzerland,https://www.dsi.uzh.ch/dam/jcr:3a0cb402-c3b3-4360-9332-f800895fdc58/dsi-strategy-lab-21-de.pdf
27
+ LimeSurvey v2,Governing Innovation,2021,Japan,https://www.meti.go.jp/press/2020/07/20200713001/20200713001-2.pdf
28
+ LimeSurvey v2,Governance Innovation ver2 A Guide to Designing and Implementing Agile Governance,2021,Japan,https://www.meti.go.jp/english/press/2021/pdf/0219_004a.pdf
29
+ LimeSurvey v2,Strategy for the Development of Artificial Intelligence in the Republic of Serbia for the period 2020-2025,2022,Serbia,https://www.media.srbija.gov.rs/medsrp/dokumenti/strategy_artificial_intelligence.pdf
30
+ LimeSurvey v2,Artificial Intelligence Strategy of the German Federal Government,2019,Germany,https://www.ki-strategie-deutschland.de/files/downloads/Fortschreibung_KI-Strategie_engl.pdf
31
+ LimeSurvey v2,Artificial Intelligence Strategic Programme 2022 2024,2021,Italy,https://www.mise.gov.it/images/stories/documenti/Strategia-Nazionale-Intelligenza-Artificiale-Bozza-Consultazione.pdf
32
+ LimeSurvey v2,Portugals National Strategy for Artificial Intelligence,2019,Portugal,https://www.incode2030.gov.pt/en/wp-content/uploads/2022/01/julho_incode_brochura.pdf
33
+ LimeSurvey v2,Plan for Federal Engagement in Developing Technical Standards and Related Tools,2019,United States,https://www.nist.gov/system/files/documents/2019/08/10/ai_standards_fedengagement_plan_9aug2019.pdf
34
+ LimeSurvey v2,Addition of Software Specially Designed To Automate the Analysis of Geospatial Imagery to the Export Control Classification Number 0Y521 Series,2021,United States,https://www.govinfo.gov/content/pkg/FR-2020-01-06/pdf/2019-27649.pdf
35
+ LimeSurvey v2,National Institute of Standards and Technology Principles for Explainable AI,2021,United States,https://www.nist.gov/system/files/documents/2020/08/17/NIST%20Explainable%20AI%20Draft%20NISTIR8312%20%281%29.pdf
36
+ LimeSurvey v2,AI and Society,2022,United States,https://www.nsf.gov/pubs/2019/nsf19018/nsf19018.pdf
37
+ LimeSurvey v2,National Defense Authorization Act for Fiscal Year 2021,2022,United States,https://www.govinfo.gov/content/pkg/BILLS-116hr6395enr/pdf/BILLS-116hr6395enr.pdf
38
+ LimeSurvey v2,NITRD NAIIO Supplement to the Presidents FY2022 Budget,2021,United States,https://www.nitrd.gov/pubs/FY2022-NITRD-NAIIO-Supplement.pdf
39
+ LimeSurvey v2,National AI RD Strategic Plan 2019 Update,2019,United States,https://www.nitrd.gov/pubs/National-AI-RD-Strategy-2019.pdf
40
+ LimeSurvey v2,National AI RD Strategic Plan,2022,United States,https://www.nitrd.gov/pubs/national_ai_rd_strategic_plan.pdf
41
+ LimeSurvey v2,PROJECTS OF SCIENTIFIC RESEARCH AND TECHNOLOGICAL DEVELOPMENT IN DATA SCIENCE AND ARTIFICIAL INTELLIGENCE IN PUBLIC ADMINISTRATION,2023,Portugal,https://www.fct.pt/wp-content/uploads/2022/06/Brochura_ResearchinDataScienceandAIappliedtoPA.pdf
42
+ LimeSurvey v2,National Landcover mapping project,2021,Ireland,https://www.npws.ie/sites/default/files/general/NLCHM%20Newsletter%20May%202017.pdf
43
+ LimeSurvey v2,Fairness Ethics Accountability and Transparency,2022,United States,https://www.nsf.gov/pubs/2019/nsf19016/nsf19016.pdf
44
+ LimeSurvey v2,Assessment of current initiatives of the European Commission on better regulation,2022,European Union,https://www.europarl.europa.eu/RegData/etudes/IDAN/2022/734766/IPOL_IDA(2022)734766_EN.pdf
45
+ LimeSurvey v2,Ministry of Digital Economy and Society,2020,Thailand,https://www.etda.or.th/getattachment/9d370f25-f37a-4b7c-b661-48d2d730651d/Digital-Thailand-AI-Ethics-Principle-and-Guideline.pdf.aspx?lang=th-TH
46
+ LimeSurvey v2,Federal 5 Year STEM Education Strategic Plan,2019,United States,https://www.energy.gov/sites/default/files/2019/05/f62/STEM-Education-Strategic-Plan-2018.pdf
47
+ LimeSurvey v2,NITRD Supplement to the Presidents FY2020 Budget,2022,United States,https://www.nitrd.gov/pubs/FY2020-NITRD-Supplement.pdf
48
+ LimeSurvey v2,Emerging Technologies Handboo,2021,Colombia,https://gobiernodigital.mintic.gov.co/692/articles-160829_Guia_Tecnologias_Emergentes.pdf
49
+ LimeSurvey v2,FRVT Demographic Effects,2021,United States,https://nvlpubs.nist.gov/nistpubs/ir/2019/NIST.IR.8280.pdf
50
+ LimeSurvey v2,"Artificial intelligence: Human rights, social justice and development",2021,Argentina,https://giswatch.org/sites/default/files/gisw2019_artificial_intelligence.pdf
51
+ LimeSurvey v2,Public Consultation for the National AI Strategy for Ireland,2021,Ireland,https://enterprise.gov.ie/en/Consultations/Consultations-files/AI-Strategy-Public-Consultation-Report.pdf
52
+ LimeSurvey v2,12th National Economic and Social Development Plan,2019,Thailand,https://www.oneplanetnetwork.org/sites/default/files/thailand_national_economic_and_social_development_plan_nesdp.pdf
53
+ LimeSurvey v2,NATIONAL ENTREPRENEURSHIP POLICY CONPES 4011,2021,Colombia,https://colaboracion.dnp.gov.co/CDT/Conpes/EconΓ³micos/4011.pdf
54
+ LimeSurvey v2,POLÍTICA NACIONAL PARA LA TRANSFORMACIΓ“N DIGITAL E INTELIGENCIA ARTIFICIAL,2021,Colombia,https://colaboracion.dnp.gov.co/CDT/Conpes/EconΓ³micos/3975.pdf
55
+ LimeSurvey v2,Pan Canadian Artificial Intelligence Strategy,2019,Canada,https://cifar.ca/wp-content/uploads/2020/11/AICan-2020-CIFAR-Pan-Canadian-AI-Strategy-Impact-Report.pdf
56
+ LimeSurvey v2,National Center for Innovation and AI,2021,Peru,https://cdn.www.gob.pe/uploads/document/file/1909267/National%20Artificial%20Intelligence%20Strategy%20-%20Peru.pdf
57
+ LimeSurvey v2,National Artificial Ingelligence strategy 2021-2025,2022,Turkey,https://cbddo.gov.tr/SharedFolderServer/Genel/File/TRNationalAIStrategy2021-2025.pdf
58
+ LimeSurvey v2,AI in Financial Services,2021,United Kingdom,https://www.turing.ac.uk/sites/default/files/2021-06/ati_ai_in_financial_services_lores.pdf
59
+ LimeSurvey v2,Good Practice Recommendations to Integrate Ethics in the Development of AI Solutions in Healthcare,2022,France,https://esante.gouv.fr/sites/default/files/media_entity/documents/ethic_by_design_guide_vf.pdf
60
+ LimeSurvey v2,AI Ecosystem Survey,2021,United Kingdom,https://www.turing.ac.uk/sites/default/files/2021-09/ai-strategy-survey_results_020921.pdf
61
+ LimeSurvey v2,US Patent and Trademark Office Report on Public Views on AI and Intellectual Property Policy,2020,United States,https://www.uspto.gov/sites/default/files/documents/USPTO_AI-Report_2020-10-07.pdf
62
+ LimeSurvey v2,Guidance for Regulation of AI Applications,2020,United States,https://www.whitehouse.gov/wp-content/uploads/2020/11/M-21-06.pdf
63
+ LimeSurvey v2,PhD Scholarship,2022,Turkey,https://www.yok.gov.tr/Documents/Yayinlar/Yayinlarimiz/2020/100-2000-yok-doktora-projesi-2020.pdf
64
+ LimeSurvey v2,UAE National Strategy for AI 2031,2019,United Arab Emirates,https://ai.gov.ae/wp-content/uploads/2021/07/UAE-National-Strategy-for-Artificial-Intelligence-2031.pdf
65
+ LimeSurvey v2,Human-Centred Artificial Intelligence for Human Resources: A Toolkit for Human Resources Professionals,2022,Turkey,https://www3.weforum.org/docs/WEF_Human_Centred_Artificial_Intelligence_for_Human_Resources_2021.pdf
66
+ LimeSurvey v2,Decree No 2007 3003 of November 27 2007 setting the operating procedures of the national body for the protection of personal data,2022,Tunisia,http://www.inpdp.tn/ressources/decret_3003.pdf
67
+ LimeSurvey v2,Amendments to the Polish Labour Code,2022,Poland,http://ilo.org/dyn/natlex/docs/ELECTRONIC/45181/91758/F1623906595/The-Labour-Code%20consolidated%201997.pdf
68
+ LimeSurvey v2,Satellite Platform for Ireland SPEir,2021,Ireland,http://eoscience.esa.int/landtraining2018/files/posters/hanafin.pdf
69
+ LimeSurvey v2,Common Regulatory Capacity for AI,2022,European Union,https://www.turing.ac.uk/sites/default/files/2022-07/common_regulatory_capacity_for_ai_the_alan_turing_institute.pdf
70
+ LimeSurvey v2,Digital Switzerland Strategy,2019,Switzerland,https://ethicsandtechnology.org/wp-content/uploads/2019/12/bericht_idag_ki_d.pdf
71
+ LimeSurvey v2,AGILE GOVERNANCE UPDATE How Governments Businesses and Civil Society Can Create a Better World by Reimagining Governance,2022,Japan,https://www.meti.go.jp/shingikai/mono_info_service/governance_model_kento/pdf/20220808_2.pdf
72
+ LimeSurvey v2,Policy on the Compliance Assistance Sandbox,2022,United States,https://files.consumerfinance.gov/f/documents/cfpb_final-policy-on-cas.pdf
73
+ LimeSurvey v2,Policy on No Action Letters,2019,United States,https://files.consumerfinance.gov/f/documents/cfpb_final-policy-on-no-action-letters.pdf
74
+ LimeSurvey v2,Policy to Encourage Trial Disclosure Programs,2022,United States,https://files.consumerfinance.gov/f/documents/cfpb_final-policy-to-encourage-tdp.pdf
75
+ OECD iLibrary,Opportunities and drawbacks of using artificial intelligence for training,2021,OECD,https://www.oecd-ilibrary.org/deliver/22729bd6-en.pdf?itemId=%2Fcontent%2Fpaper%2F22729bd6-en&mimeType=pdf
76
+ OECD iLibrary,"Who develops AI-related innovations, goods and services? : A firm-level analysis",2021,OECD,https://www.oecd-ilibrary.org/deliver/3e4aedd4-en.pdf?itemId=%2Fcontent%2Fpaper%2F3e4aedd4-en&mimeType=pdf
77
+ OECD iLibrary,Nowcasting aggregate services trade,2021,OECD,https://www.oecd-ilibrary.org/deliver/0ad7d27c-en.pdf?itemId=%2Fcontent%2Fpaper%2F0ad7d27c-en&mimeType=pdf
78
+ OECD iLibrary,OECD Employment Outlook 2023 : Artificial Intelligence and the Labour Market,2023,OECD,https://www.oecd-ilibrary.org/deliver/08785bba-en.pdf?itemId=%2Fcontent%2Fpublication%2F08785bba-en&mimeType=pdf
79
+ OECD iLibrary,Know thy AI: Assessing the risks of artificial intelligence,2022,OECD,https://www.oecd-ilibrary.org/deliver/067f4cff-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F067f4cff-en&mimeType=pdf
80
+ OECD iLibrary,"Positive, High-achieving Students? : What Schools and Teachers Can Do",2021,OECD,https://www.oecd-ilibrary.org/deliver/3b9551db-en.pdf?itemId=%2Fcontent%2Fpublication%2F3b9551db-en&mimeType=pdf
81
+ OECD iLibrary,"PENELOPE 2018: A code system for Monte Carlo simulation of electron and photon transport : Workshop Proceedings, Barcelona, Spain, 28 January – 1 February 2019",2019,OECD,https://www.oecd-ilibrary.org/deliver/32da5043-en.pdf?itemId=%2Fcontent%2Fpublication%2F32da5043-en&mimeType=pdf
82
+ OECD iLibrary,Measuring the AI content of government-funded R&D projects : A proof of concept for the OECD Fundstat initiative,2021,OECD,https://www.oecd-ilibrary.org/deliver/7b43b038-en.pdf?itemId=%2Fcontent%2Fpaper%2F7b43b038-en&mimeType=pdf
83
+ OECD iLibrary,Is digital media literacy the answer to our disinformation woes?,2022,OECD,https://www.oecd-ilibrary.org/deliver/326b63bf-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F326b63bf-en&mimeType=pdf
84
+ OECD iLibrary,The human capital behind AI : Jobs and skills demand from online job postings,2021,OECD,https://www.oecd-ilibrary.org/deliver/2e278150-en.pdf?itemId=%2Fcontent%2Fpaper%2F2e278150-en&mimeType=pdf
85
+ OECD iLibrary,Measuring the environmental impacts of artificial intelligence compute and applications : The AI footprint,2022,OECD,https://www.oecd-ilibrary.org/deliver/7babf571-en.pdf?itemId=%2Fcontent%2Fpaper%2F7babf571-en&mimeType=pdf
86
+ OECD iLibrary,Improving reproducibility of artificial intelligence research to increase trust and productivity,2023,OECD,https://www.oecd-ilibrary.org/improving-reproducibility-of-artificial-intelligence-research-to-increase-trust-and-productivity_3f57323a-en.pdf?itemId=%2Fcontent%2Fcomponent%2F3f57323a-en&mimeType=pdf
87
+ OECD iLibrary,The impact of Artificial Intelligence on the labour market : What do we know so far?,2021,OECD,https://www.oecd-ilibrary.org/deliver/7c895724-en.pdf?itemId=%2Fcontent%2Fpaper%2F7c895724-en&mimeType=pdf
88
+ OECD iLibrary,Nowcasting trade in value added indicators,2023,OECD,https://www.oecd-ilibrary.org/deliver/00f8aff7-en.pdf?itemId=%2Fcontent%2Fpaper%2F00f8aff7-en&mimeType=pdf
89
+ OECD iLibrary,"Artificial intelligence companies, goods and services : A trademark-based analysis",2021,OECD,https://www.oecd-ilibrary.org/deliver/2db2d7f4-en.pdf?itemId=%2Fcontent%2Fpaper%2F2db2d7f4-en&mimeType=pdf
90
+ OECD iLibrary,Putting AI to the test : How does the performance of GPT and 15-year-old students in PISA compare?,2023,OECD,https://www.oecd-ilibrary.org/deliver/2c297e0b-en.pdf?itemId=%2Fcontent%2Fpaper%2F2c297e0b-en&mimeType=pdf
91
+ OECD iLibrary,"Artificial intelligence, its diffusion and uses in manufacturing",2021,OECD,https://www.oecd-ilibrary.org/deliver/249e2003-en.pdf?itemId=%2Fcontent%2Fpaper%2F249e2003-en&mimeType=pdf
92
+ OECD iLibrary,"Why are some U.S. cities successful, while others are not? Empirical evidence from machine learning",2020,OECD,https://www.oecd-ilibrary.org/deliver/7f77c2e7-en.pdf?itemId=%2Fcontent%2Fpaper%2F7f77c2e7-en&mimeType=pdf
93
+ OECD iLibrary,Open science - Enabling discovery in the digital age,2021,OECD,https://www.oecd-ilibrary.org/deliver/81a9dcf0-en.pdf?itemId=%2Fcontent%2Fpaper%2F81a9dcf0-en&mimeType=pdf
94
+ OECD iLibrary,Using Artificial Intelligence in the workplace : What are the main ethical risks?,2022,OECD,https://www.oecd-ilibrary.org/deliver/840a2d9f-en.pdf?itemId=%2Fcontent%2Fpaper%2F840a2d9f-en&mimeType=pdf
95
+ OECD iLibrary,Enhancing Access to and Sharing of Data : Reconciling Risks and Benefits for Data Re-use across Societies,2019,OECD,https://www.oecd-ilibrary.org/deliver/276aaca8-en.pdf?itemId=%2Fcontent%2Fpublication%2F276aaca8-en&mimeType=pdf
96
+ OECD iLibrary,Artificial intelligence and labour market matching,2023,OECD,https://www.oecd-ilibrary.org/deliver/2b440821-en.pdf?itemId=%2Fcontent%2Fpaper%2F2b440821-en&mimeType=pdf
97
+ OECD iLibrary,A blueprint for building national compute capacity for artificial intelligence,2023,OECD,https://www.oecd-ilibrary.org/deliver/876367e3-en.pdf?itemId=%2Fcontent%2Fpaper%2F876367e3-en&mimeType=pdf
98
+ OECD iLibrary,New Directions for Data-Driven Transport Safety,2019,OECD,https://www.oecd-ilibrary.org/deliver/2b2f71bc-en.pdf?itemId=%2Fcontent%2Fpaper%2F2b2f71bc-en&mimeType=pdf
99
+ OECD iLibrary,What role will artificial intelligence (AI) play in the classroom?,2020,OECD,https://www.oecd-ilibrary.org/deliver/2980fe96-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F2980fe96-en&mimeType=pdf
100
+ OECD iLibrary,Tools for trustworthy AI : A framework to compare implementation tools for trustworthy AI systems,2021,OECD,https://www.oecd-ilibrary.org/deliver/008232ec-en.pdf?itemId=%2Fcontent%2Fpaper%2F008232ec-en&mimeType=pdf
101
+ OECD iLibrary,Artificial Intelligence in Proactive Road Infrastructure Safety Management : Summary and Conclusions,2021,OECD,https://www.oecd-ilibrary.org/deliver/04509d3f-en.pdf?itemId=%2Fcontent%2Fpublication%2F04509d3f-en&mimeType=pdf
102
+ OECD iLibrary,Demand for AI skills in jobs : Evidence from online job postings,2021,OECD,https://www.oecd-ilibrary.org/deliver/3ed32d94-en.pdf?itemId=%2Fcontent%2Fpaper%2F3ed32d94-en&mimeType=pdf
103
+ OECD iLibrary,Is Education Losing the Race with Technology? : AI's Progress in Maths and Reading,2023,OECD,https://www.oecd-ilibrary.org/deliver/73105f99-en.pdf?itemId=%2Fcontent%2Fpublication%2F73105f99-en&mimeType=pdf
104
+ OECD iLibrary,AI measurement in ICT usage surveys : A review,2021,OECD,https://www.oecd-ilibrary.org/deliver/72cce754-en.pdf?itemId=%2Fcontent%2Fpaper%2F72cce754-en&mimeType=pdf
105
+ OECD iLibrary,The impact of AI on the workplace: Evidence from OECD case studies of AI implementation,2023,OECD,https://www.oecd-ilibrary.org/deliver/2247ce58-en.pdf?itemId=%2Fcontent%2Fpaper%2F2247ce58-en&mimeType=pdf
106
+ OECD iLibrary,Identifying artificial intelligence actors using online data,2023,OECD,https://www.oecd-ilibrary.org/deliver/1f5307e7-en.pdf?itemId=%2Fcontent%2Fpaper%2F1f5307e7-en&mimeType=pdf
107
+ OECD iLibrary,Adaptive Trees: a new approach to economic forecasting,2020,OECD,https://www.oecd-ilibrary.org/deliver/5569a0aa-en.pdf?itemId=%2Fcontent%2Fpaper%2F5569a0aa-en&mimeType=pdf
108
+ OECD iLibrary,Developing Minds in the Digital Age : Towards a Science of Learning for 21st Century Education,2019,OECD,https://www.oecd-ilibrary.org/deliver/562a8659-en.pdf?itemId=%2Fcontent%2Fpublication%2F562a8659-en&mimeType=pdf
109
+ OECD iLibrary,"OECD Digital Education Outlook 2021 : Pushing the Frontiers with Artificial Intelligence, Blockchain and Robots",2021,OECD,https://www.oecd-ilibrary.org/deliver/589b283f-en.pdf?itemId=%2Fcontent%2Fpublication%2F589b283f-en&mimeType=pdf
110
+ OECD iLibrary,How will technology and artificial intelligence (AI) affect education?,2019,OECD,https://www.oecd-ilibrary.org/deliver/58d1112a-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F58d1112a-en&mimeType=pdf
111
+ OECD iLibrary,Defining and classifying AI in the workplace,2023,OECD,https://www.oecd-ilibrary.org/deliver/59e89d7f-en.pdf?itemId=%2Fcontent%2Fpaper%2F59e89d7f-en&mimeType=pdf
112
+ OECD iLibrary,Linking Aid to the Sustainable Development Goals – a machine learning approach,2019,OECD,https://www.oecd-ilibrary.org/deliver/4bdaeb8c-en.pdf?itemId=%2Fcontent%2Fpaper%2F4bdaeb8c-en&mimeType=pdf
113
+ OECD iLibrary,Laying the foundations for artificial intelligence in health,2021,OECD,https://www.oecd-ilibrary.org/deliver/3f62817d-en.pdf?itemId=%2Fcontent%2Fpaper%2F3f62817d-en&mimeType=pdf
114
+ OECD iLibrary,"AI and the Future of Skills, Volume 1 : Capabilities and Assessments",2021,OECD,https://www.oecd-ilibrary.org/deliver/5ee71f34-en.pdf?itemId=%2Fcontent%2Fpublication%2F5ee71f34-en&mimeType=pdf
115
+ OECD iLibrary,Regulatory sandboxes in artificial intelligence,2023,OECD,https://www.oecd-ilibrary.org/deliver/8f80a0e6-en.pdf?itemId=%2Fcontent%2Fpaper%2F8f80a0e6-en&mimeType=pdf
116
+ OECD iLibrary,The Strategic and Responsible Use of Artificial Intelligence in the Public Sector of Latin America and the Caribbean,2022,OECD,https://www.oecd-ilibrary.org/deliver/1f334543-en.pdf?itemId=%2Fcontent%2Fpublication%2F1f334543-en&mimeType=pdf
117
+ OECD iLibrary,Identifying and measuring developments in artificial intelligence : Making the impossible possible,2020,OECD,https://www.oecd-ilibrary.org/deliver/5f65ff7e-en.pdf?itemId=%2Fcontent%2Fpaper%2F5f65ff7e-en&mimeType=pdf
118
+ OECD iLibrary,Progress on implementing and using electronic health record systems : Developments in OECD countries as of 2021,2023,OECD,https://www.oecd-ilibrary.org/deliver/4f4ce846-en.pdf?itemId=%2Fcontent%2Fpaper%2F4f4ce846-en&mimeType=pdf
119
+ OECD iLibrary,State of implementation of the OECD AI Principles : Insights from national AI policies,2021,OECD,https://www.oecd-ilibrary.org/deliver/1cd40c44-en.pdf?itemId=%2Fcontent%2Fpaper%2F1cd40c44-en&mimeType=pdf
120
+ OECD iLibrary,Advancing accountability in AI : Governing and managing risks throughout the lifecycle for trustworthy AI,2023,OECD,https://www.oecd-ilibrary.org/deliver/2448f04b-en.pdf?itemId=%2Fcontent%2Fpaper%2F2448f04b-en&mimeType=pdf
121
+ OECD iLibrary,The digital innovation policy landscape in 2019,2019,OECD,https://www.oecd-ilibrary.org/deliver/6171f649-en.pdf?itemId=%2Fcontent%2Fpaper%2F6171f649-en&mimeType=pdf
122
+ OECD iLibrary,Review of national policy initiatives in support of digital and AI-driven innovation,2019,OECD,https://www.oecd-ilibrary.org/deliver/15491174-en.pdf?itemId=%2Fcontent%2Fpaper%2F15491174-en&mimeType=pdf
123
+ OECD iLibrary,The Effects of AI on the Working Lives of Women,2022,OECD,https://www.oecd-ilibrary.org/deliver/14e9b92c-en.pdf?itemId=%2Fcontent%2Fpublication%2F14e9b92c-en&mimeType=pdf
124
+ OECD iLibrary,The digitalisation of agriculture : A literature review and emerging policy issues,2022,OECD,https://www.oecd-ilibrary.org/deliver/285cc27d-en.pdf?itemId=%2Fcontent%2Fpaper%2F285cc27d-en&mimeType=pdf
125
+ OECD iLibrary,Labour and Skills Demand in Alberta : Insights Using Big Data Intelligence,2023,OECD,https://www.oecd-ilibrary.org/deliver/659ce346-en.pdf?itemId=%2Fcontent%2Fpublication%2F659ce346-en&mimeType=pdf
126
+ OECD iLibrary,The Artificial Intelligence Act: Addressing the divergence between the public and private sectors,2022,OECD,https://www.oecd-ilibrary.org/deliver/146fa80b-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F146fa80b-en&mimeType=pdf
127
+ OECD iLibrary,Tracking activity in real time with Google Trends,2020,OECD,https://www.oecd-ilibrary.org/deliver/6b9c7518-en.pdf?itemId=%2Fcontent%2Fpaper%2F6b9c7518-en&mimeType=pdf
128
+ OECD iLibrary,"AI language models : Technological, socio-economic and policy considerations",2023,OECD,https://www.oecd-ilibrary.org/deliver/13d38f92-en.pdf?itemId=%2Fcontent%2Fpaper%2F13d38f92-en&mimeType=pdf
129
+ OECD iLibrary,Artificial Intelligence and international trade : Some preliminary implications,2022,OECD,https://www.oecd-ilibrary.org/deliver/13212d3e-en.pdf?itemId=%2Fcontent%2Fpaper%2F13212d3e-en&mimeType=pdf
130
+ OECD iLibrary,"G20 Compendium on the Use of Digital Tools for Public Service Continuity : Report for the G20 Digital Economy Tak Force, Trieste, Italy, August 2021",2021,OECD,https://www.oecd-ilibrary.org/deliver/6f800fd5-en.pdf?itemId=%2Fcontent%2Fpublication%2F6f800fd5-en&mimeType=pdf
131
+ OECD iLibrary,"A portrait of AI adopters across countries : Firm characteristics, assets’ complementarities and productivity",2023,OECD,https://www.oecd-ilibrary.org/deliver/0fb79bb9-en.pdf?itemId=%2Fcontent%2Fpaper%2F0fb79bb9-en&mimeType=pdf
132
+ OECD iLibrary,"Hello, World : Artificial intelligence and its use in the public sector",2019,OECD,https://www.oecd-ilibrary.org/deliver/726fd39d-en.pdf?itemId=%2Fcontent%2Fpaper%2F726fd39d-en&mimeType=pdf
133
+ OECD iLibrary,Identifying and characterising AI adopters : A novel approach based on big data,2022,OECD,https://www.oecd-ilibrary.org/deliver/154981d7-en.pdf?itemId=%2Fcontent%2Fpaper%2F154981d7-en&mimeType=pdf
134
+ OECD iLibrary,What skills and abilities can automation technologies replicate and what does it mean for workers? : New evidence,2022,OECD,https://www.oecd-ilibrary.org/deliver/646aad77-en.pdf?itemId=%2Fcontent%2Fpaper%2F646aad77-en&mimeType=pdf
135
+ OECD iLibrary,Countering Public Grant Fraud in Spain : Machine Learning for Assessing Risks and Targeting Control Activities,2021,OECD,https://www.oecd-ilibrary.org/deliver/0ea22484-en.pdf?itemId=%2Fcontent%2Fpublication%2F0ea22484-en&mimeType=pdf
136
+ OECD iLibrary,AI scoring for international large-scale assessments using a deep learning model and multilingual data,2023,OECD,https://www.oecd-ilibrary.org/deliver/9918e1fb-en.pdf?itemId=%2Fcontent%2Fpaper%2F9918e1fb-en&mimeType=pdf
137
+ OECD iLibrary,Enabling environment for FinTech innovation in the Czech Republic and EU,2022,OECD,https://www.oecd-ilibrary.org/enabling-environment-for-fintech-innovation-in-the-czech-republic-and-eu_47b4db37-en.pdf?itemId=%2Fcontent%2Fcomponent%2F47b4db37-en&mimeType=pdf
138
+ OECD iLibrary,Elicit: Language models as research tools,2023,OECD,https://www.oecd-ilibrary.org/elicit-language-models-as-research-tools_174aee8f-en.pdf?itemId=%2Fcontent%2Fcomponent%2F174aee8f-en&mimeType=pdf
139
+ OECD iLibrary,"Efforts to develop a responsible, trustworthy and human-centric approach",2022,OECD,https://www.oecd-ilibrary.org/efforts-to-develop-a-responsible-trustworthy-and-human-centric-approach_96d49021-en.pdf?itemId=%2Fcontent%2Fcomponent%2F96d49021-en&mimeType=pdf
140
+ OECD iLibrary,Editorial,2021,OECD,https://www.oecd-ilibrary.org/editorial_b0dd0724-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fb0dd0724-en&mimeType=pdf
141
+ OECD iLibrary,Editorial: Beyond the hype on AI – early signs of divides in the labour market,2023,OECD,https://www.oecd-ilibrary.org/editorial-beyond-the-hype-on-ai-early-signs-of-divides-in-the-labour-market_11489d95-en.pdf?itemId=%2Fcontent%2Fcomponent%2F11489d95-en&mimeType=pdf
142
+ OECD iLibrary,Editorial: A transition agenda for a Future that Works for all,2019,OECD,https://www.oecd-ilibrary.org/editorial-a-transition-agenda-for-a-future-that-works-for-all_86970c5c-en.pdf?itemId=%2Fcontent%2Fcomponent%2F86970c5c-en&mimeType=pdf
143
+ OECD iLibrary,Ensuring trustworthy artificial intelligence in the workplace: Countries’ policy action,2023,OECD,https://www.oecd-ilibrary.org/ensuring-trustworthy-artificial-intelligence-in-the-workplace-countries-policy-action_04b3d08d-en.pdf?itemId=%2Fcontent%2Fcomponent%2F04b3d08d-en&mimeType=pdf
144
+ OECD iLibrary,Early warning systems and indicators of dropping out of upper secondary school: the emerging role of digital technologies,2021,OECD,https://www.oecd-ilibrary.org/early-warning-systems-and-indicators-of-dropping-out-of-upper-secondary-school-the-emerging-role-of-digital-technologies_c8e57e15-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc8e57e15-en&mimeType=pdf
145
+ OECD iLibrary,Digitalisation and productivity: A story of complementarities,2019,OECD,https://www.oecd-ilibrary.org/digitalisation-and-productivity-a-story-of-complementarities_5713bd7d-en.pdf?itemId=%2Fcontent%2Fcomponent%2F5713bd7d-en&mimeType=pdf
146
+ OECD iLibrary,Digital innovation,2020,OECD,https://www.oecd-ilibrary.org/digital-innovation_8e8f2750-en.pdf?itemId=%2Fcontent%2Fcomponent%2F8e8f2750-en&mimeType=pdf
147
+ OECD iLibrary,Digital health,2021,OECD,https://www.oecd-ilibrary.org/digital-health_08cffda7-en.pdf?itemId=%2Fcontent%2Fcomponent%2F08cffda7-en&mimeType=pdf
148
+ OECD iLibrary,Digital government strategies and institutional frameworks,2019,OECD,https://www.oecd-ilibrary.org/digital-government-strategies-and-institutional-frameworks_63029fcb-en.pdf?itemId=%2Fcontent%2Fcomponent%2F63029fcb-en&mimeType=pdf
149
+ OECD iLibrary,Democratising artificial intelligence to accelerate scientific discovery,2023,OECD,https://www.oecd-ilibrary.org/democratising-artificial-intelligence-to-accelerate-scientific-discovery_be9632d7-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fbe9632d7-en&mimeType=pdf
150
+ OECD iLibrary,Initial policy considerations for generative artificial intelligence,2023,OECD,https://www.oecd-ilibrary.org/deliver/fae2d1e6-en.pdf?itemId=%2Fcontent%2Fpaper%2Ffae2d1e6-en&mimeType=pdf
151
+ OECD iLibrary,Digitalisation of health information,2023,OECD,https://www.oecd-ilibrary.org/digitalisation-of-health-information_f44731cf-en.pdf?itemId=%2Fcontent%2Fcomponent%2Ff44731cf-en&mimeType=pdf
152
+ OECD iLibrary,Venture capital investments in artificial intelligence : Analysing trends in VC in AI companies from 2012 through 2020,2021,OECD,https://www.oecd-ilibrary.org/deliver/f97beae7-en.pdf?itemId=%2Fcontent%2Fpaper%2Ff97beae7-en&mimeType=pdf
153
+ OECD iLibrary,Essai nΒ° 471: Essai de mutation rΓ©verse sur des bactΓ©ries,2020,OECD,https://www.oecd-ilibrary.org/essai-n-471-essai-de-mutation-reverse-sur-des-bacteries_5lmqcr2k7md4.pdf?itemId=%2Fcontent%2Fpublication%2F9789264071254-fr&mimeType=pdf
154
+ OECD iLibrary,Evolution of human skills versus AI capabilities,2023,OECD,https://www.oecd-ilibrary.org/evolution-of-human-skills-versus-ai-capabilities_d077ad2f-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd077ad2f-en&mimeType=pdf
155
+ OECD iLibrary,Human rights due diligence through responsible AI,2021,OECD,https://www.oecd-ilibrary.org/human-rights-due-diligence-through-responsible-ai_31e7edcc-en.pdf?itemId=%2Fcontent%2Fcomponent%2F31e7edcc-en&mimeType=pdf
156
+ OECD iLibrary,How can artificial intelligence help scientists? A (non-exhaustive) overview,2023,OECD,https://www.oecd-ilibrary.org/how-can-artificial-intelligence-help-scientists-a-non-exhaustive-overview_a8e6c3b6-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fa8e6c3b6-en&mimeType=pdf
157
+ OECD iLibrary,"How are science, technology and innovation going digital? The statistical evidence",2020,OECD,https://www.oecd-ilibrary.org/how-are-science-technology-and-innovation-going-digital-the-statistical-evidence_1cfd272a-en.pdf?itemId=%2Fcontent%2Fcomponent%2F1cfd272a-en&mimeType=pdf
158
+ OECD iLibrary,High-performance computing leadership to enable advances in artificial intelligence and a thriving compute ecosystem,2023,OECD,https://www.oecd-ilibrary.org/high-performance-computing-leadership-to-enable-advances-in-artificial-intelligence-and-a-thriving-compute-ecosystem_0e1b4c2f-en.pdf?itemId=%2Fcontent%2Fcomponent%2F0e1b4c2f-en&mimeType=pdf
159
+ OECD iLibrary,Guidance and regulatory frameworks for digital education,2023,OECD,https://www.oecd-ilibrary.org/guidance-and-regulatory-frameworks-for-digital-education_a0a23dc6-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fa0a23dc6-en&mimeType=pdf
160
+ OECD iLibrary,Government investment spending,2019,OECD,https://www.oecd-ilibrary.org/government-investment-spending_b4536bd6-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fb4536bd6-en&mimeType=pdf
161
+ OECD iLibrary,Evaluating revolutions in artificial intelligence from a human perspective,2021,OECD,https://www.oecd-ilibrary.org/evaluating-revolutions-in-artificial-intelligence-from-a-human-perspective_004710fe-en.pdf?itemId=%2Fcontent%2Fcomponent%2F004710fe-en&mimeType=pdf
162
+ OECD iLibrary,Global mega-trends and the future of education,2019,OECD,https://www.oecd-ilibrary.org/global-mega-trends-and-the-future-of-education_5j3x77mdk442.pdf?itemId=%2Fcontent%2Fcomponent%2Ftrends_edu-2019-3-en&mimeType=pdf
163
+ OECD iLibrary,From knowledge discovery to knowledge creation: How can literature-based discovery accelerate progress in science?,2023,OECD,https://www.oecd-ilibrary.org/from-knowledge-discovery-to-knowledge-creation-how-can-literature-based-discovery-accelerate-progress-in-science_de74d7a9-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fde74d7a9-en&mimeType=pdf
164
+ OECD iLibrary,Fraud in public grants: Piloting a data-driven risk model in Spain,2021,OECD,https://www.oecd-ilibrary.org/fraud-in-public-grants-piloting-a-data-driven-risk-model-in-spain_55696a8b-en.pdf?itemId=%2Fcontent%2Fcomponent%2F55696a8b-en&mimeType=pdf
165
+ OECD iLibrary,Foreword,2022,OECD,https://www.oecd-ilibrary.org/foreword_1d5c6062-en.pdf?itemId=%2Fcontent%2Fcomponent%2F1d5c6062-en&mimeType=pdf
166
+ OECD iLibrary,"First-class humans, not second-class robots – Andreas Schleicher on learning and the future of work",2019,OECD,https://www.oecd-ilibrary.org/first-class-humans-not-second-class-robots-andreas-schleicher-on-learning-and-the-future-of-work_87958706-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F87958706-en&mimeType=pdf
167
+ OECD iLibrary,Experts’ assessments of AI capabilities in literacy and numeracy,2023,OECD,https://www.oecd-ilibrary.org/experts-assessments-of-ai-capabilities-in-literacy-and-numeracy_134fa8aa-en.pdf?itemId=%2Fcontent%2Fcomponent%2F134fa8aa-en&mimeType=pdf
168
+ OECD iLibrary,Executive summary,2019,OECD,https://www.oecd-ilibrary.org/executive-summary_98aef837-en.pdf?itemId=%2Fcontent%2Fcomponent%2F98aef837-en&mimeType=pdf
169
+ OECD iLibrary,Frontiers of smart education technology: Opportunities and challenges,2021,OECD,https://www.oecd-ilibrary.org/frontiers-of-smart-education-technology-opportunities-and-challenges_d3153fcd-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd3153fcd-en&mimeType=pdf
170
+ OECD iLibrary,"Data-Driven, Information-Enabled Regulatory Delivery",2021,OECD,https://www.oecd-ilibrary.org/deliver/8f99ec8c-en.pdf?itemId=%2Fcontent%2Fpublication%2F8f99ec8c-en&mimeType=pdf
171
+ OECD iLibrary,Harnessing the power of AI and emerging technologies : Background paper for the CDEP Ministerial meeting,2022,OECD,https://www.oecd-ilibrary.org/deliver/f94df8ec-en.pdf?itemId=%2Fcontent%2Fpaper%2Ff94df8ec-en&mimeType=pdf
172
+ OECD iLibrary,Artificial Intelligence in Society,2019,OECD,https://www.oecd-ilibrary.org/deliver/eedfee77-en.pdf?itemId=%2Fcontent%2Fpublication%2Feedfee77-en&mimeType=pdf
173
+ OECD iLibrary,"Shaping Digital Education : Enabling Factors for Quality, Equity and Efficiency",2023,OECD,https://www.oecd-ilibrary.org/deliver/bac4dc9f-en.pdf?itemId=%2Fcontent%2Fpublication%2Fbac4dc9f-en&mimeType=pdf
174
+ OECD iLibrary,OECD Business and Finance Outlook 2021 : AI in Business and Finance,2021,OECD,https://www.oecd-ilibrary.org/deliver/ba682899-en.pdf?itemId=%2Fcontent%2Fpublication%2Fba682899-en&mimeType=pdf
175
+ OECD iLibrary,"The Digitalisation of Science, Technology and Innovation : Key Developments and Policies",2020,OECD,https://www.oecd-ilibrary.org/deliver/b9e4a2c0-en.pdf?itemId=%2Fcontent%2Fpublication%2Fb9e4a2c0-en&mimeType=pdf
176
+ OECD iLibrary,Disinformation and its discontents,2022,OECD,https://www.oecd-ilibrary.org/deliver/b8856732-en.pdf?itemId=%2Fcontent%2Fmultimedia%2Fb8856732-en&mimeType=pdf
177
+ OECD iLibrary,Statistical profiling in public employment services : An international comparison,2019,OECD,https://www.oecd-ilibrary.org/deliver/b5e5f16e-en.pdf?itemId=%2Fcontent%2Fpaper%2Fb5e5f16e-en&mimeType=pdf
178
+ OECD iLibrary,Using artificial intelligence to help combat COVID-19,2020,OECD,https://www.oecd-ilibrary.org/deliver/ae4c5c21-en.pdf?itemId=%2Fcontent%2Fpaper%2Fae4c5c21-en&mimeType=pdf
179
+ OECD iLibrary,"The supply, demand and characteristics of the AI workforce across OECD countries",2023,OECD,https://www.oecd-ilibrary.org/deliver/bb17314a-en.pdf?itemId=%2Fcontent%2Fpaper%2Fbb17314a-en&mimeType=pdf
180
+ OECD iLibrary,Speaking the same language : A machine learning approach to classify skills in Burning Glass Technologies data,2021,OECD,https://www.oecd-ilibrary.org/deliver/adb03746-en.pdf?itemId=%2Fcontent%2Fpaper%2Fadb03746-en&mimeType=pdf
181
+ OECD iLibrary,Six questions about the demand for artificial intelligence skills in labour markets,2023,OECD,https://www.oecd-ilibrary.org/deliver/ac1bebf0-en.pdf?itemId=%2Fcontent%2Fpaper%2Fac1bebf0-en&mimeType=pdf
182
+ OECD iLibrary,Perspectives de l’emploi de l’OCDE 2023 : Intelligence artificielle et marchΓ© du travail,2023,OECD,https://www.oecd-ilibrary.org/deliver/aae5dba0-fr.pdf?itemId=%2Fcontent%2Fpublication%2Faae5dba0-fr&mimeType=pdf
183
+ OECD iLibrary,Improving effectiveness of Lithuania’s innovation policy,2021,OECD,https://www.oecd-ilibrary.org/deliver/a8fec2ee-en.pdf?itemId=%2Fcontent%2Fpaper%2Fa8fec2ee-en&mimeType=pdf
184
+ OECD iLibrary,"Artificial Intelligence in Science : Challenges, Opportunities and the Future of Research",2023,OECD,https://www.oecd-ilibrary.org/deliver/a8d820bd-en.pdf?itemId=%2Fcontent%2Fpublication%2Fa8d820bd-en&mimeType=pdf
185
+ OECD iLibrary,Trustworthy artificial intelligence (AI) in education : Promises and challenges,2020,OECD,https://www.oecd-ilibrary.org/deliver/a6c90fa9-en.pdf?itemId=%2Fcontent%2Fpaper%2Fa6c90fa9-en&mimeType=pdf
186
+ OECD iLibrary,Digital Innovation : Seizing Policy Opportunities,2019,OECD,https://www.oecd-ilibrary.org/deliver/a298dc87-en.pdf?itemId=%2Fcontent%2Fpublication%2Fa298dc87-en&mimeType=pdf
187
+ OECD iLibrary,Effective Government Information Websites : Toolkit for Implementation,2023,OECD,https://www.oecd-ilibrary.org/deliver/ac325b03-en.pdf?itemId=%2Fcontent%2Fpublication%2Fac325b03-en&mimeType=pdf
188
+ OECD iLibrary,Shaping the transition : Artificial intelligence and social dialogue,2022,OECD,https://www.oecd-ilibrary.org/deliver/f097c48a-en.pdf?itemId=%2Fcontent%2Fpaper%2Ff097c48a-en&mimeType=pdf
189
+ OECD iLibrary,Data-driven innovation in clinical pharmaceutical research,2023,OECD,https://www.oecd-ilibrary.org/data-driven-innovation-in-clinical-pharmaceutical-research_8763a389-en.pdf?itemId=%2Fcontent%2Fcomponent%2F8763a389-en&mimeType=pdf
190
+ OECD iLibrary,AI is poised to revolutionise healthcare. Building trust will be key,2022,OECD,https://www.oecd-ilibrary.org/deliver/bc7f47fa-en.pdf?itemId=%2Fcontent%2Fmultimedia%2Fbc7f47fa-en&mimeType=pdf
191
+ OECD iLibrary,Governing Transport in the Algorithmic Age,2019,OECD,https://www.oecd-ilibrary.org/deliver/eec0b9aa-en.pdf?itemId=%2Fcontent%2Fpaper%2Feec0b9aa-en&mimeType=pdf
192
+ OECD iLibrary,The impact of AI on the workplace: Main findings from the OECD AI surveys of employers and workers,2023,OECD,https://www.oecd-ilibrary.org/deliver/ea0a0fe1-en.pdf?itemId=%2Fcontent%2Fpaper%2Fea0a0fe1-en&mimeType=pdf
193
+ OECD iLibrary,Not lost in translation : The implications of machine translation technologies for language professionals and for broader society,2023,OECD,https://www.oecd-ilibrary.org/deliver/e1d1d170-en.pdf?itemId=%2Fcontent%2Fpaper%2Fe1d1d170-en&mimeType=pdf
194
+ OECD iLibrary,The value of data in digital-based business models: Measurement and economic policy implications,2022,OECD,https://www.oecd-ilibrary.org/deliver/d960a10c-en.pdf?itemId=%2Fcontent%2Fpaper%2Fd960a10c-en&mimeType=pdf
195
+ OECD iLibrary,Scoping the OECD AI principles : Deliberations of the Expert Group on Artificial Intelligence at the OECD (AIGO),2019,OECD,https://www.oecd-ilibrary.org/deliver/d62f618a-en.pdf?itemId=%2Fcontent%2Fpaper%2Fd62f618a-en&mimeType=pdf
196
+ OECD iLibrary,The EU-OECD definition of a functional urban area,2019,OECD,https://www.oecd-ilibrary.org/deliver/d58cb34d-en.pdf?itemId=%2Fcontent%2Fpaper%2Fd58cb34d-en&mimeType=pdf
197
+ OECD iLibrary,Enhancing the security of communication infrastructure,2023,OECD,https://www.oecd-ilibrary.org/deliver/bb608fe5-en.pdf?itemId=%2Fcontent%2Fpaper%2Fbb608fe5-en&mimeType=pdf
198
+ OECD iLibrary,Embracing a One Health Framework to Fight Antimicrobial Resistance,2023,OECD,https://www.oecd-ilibrary.org/deliver/ce44c755-en.pdf?itemId=%2Fcontent%2Fpublication%2Fce44c755-en&mimeType=pdf
199
+ OECD iLibrary,OECD Framework for the Classification of AI systems,2022,OECD,https://www.oecd-ilibrary.org/deliver/cb6d9eca-en.pdf?itemId=%2Fcontent%2Fpaper%2Fcb6d9eca-en&mimeType=pdf
200
+ OECD iLibrary,Governing data and AI for all: Which model for a sustainable and just data governance?,2022,OECD,https://www.oecd-ilibrary.org/deliver/c3a0a473-en.pdf?itemId=%2Fcontent%2Fmultimedia%2Fc3a0a473-en&mimeType=pdf
201
+ OECD iLibrary,Artificial intelligence and employment : New cross-country evidence,2021,OECD,https://www.oecd-ilibrary.org/deliver/c2c1d276-en.pdf?itemId=%2Fcontent%2Fpaper%2Fc2c1d276-en&mimeType=pdf
202
+ OECD iLibrary,An overview of national AI strategies and policies,2021,OECD,https://www.oecd-ilibrary.org/deliver/c05140d9-en.pdf?itemId=%2Fcontent%2Fpaper%2Fc05140d9-en&mimeType=pdf
203
+ OECD iLibrary,G7 Hiroshima Process on Generative Artificial Intelligence (AI) : Towards a G7 Common Understanding on Generative AI,2023,OECD,https://www.oecd-ilibrary.org/deliver/bf3c0c60-en.pdf?itemId=%2Fcontent%2Fpublication%2Fbf3c0c60-en&mimeType=pdf
204
+ OECD iLibrary,The Digital Transformation of SMEs,2021,OECD,https://www.oecd-ilibrary.org/deliver/bdb9256a-en.pdf?itemId=%2Fcontent%2Fpublication%2Fbdb9256a-en&mimeType=pdf
205
+ OECD iLibrary,Development Co-operation Report 2021 : Shaping a Just Digital Transformation,2021,OECD,https://www.oecd-ilibrary.org/deliver/ce08832f-en.pdf?itemId=%2Fcontent%2Fpublication%2Fce08832f-en&mimeType=pdf
206
+ OECD iLibrary,Data driven approaches towards risk-based regulatory delivery,2021,OECD,https://www.oecd-ilibrary.org/data-driven-approaches-towards-risk-based-regulatory-delivery_eff10ad4-en.pdf?itemId=%2Fcontent%2Fcomponent%2Feff10ad4-en&mimeType=pdf
207
+ OECD iLibrary,Applying machine learning techniques to inspections,2021,OECD,https://www.oecd-ilibrary.org/applying-machine-learning-techniques-to-inspections_c052cabf-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc052cabf-en&mimeType=pdf
208
+ OECD iLibrary,Cross-cultural validity and comparability in assessments of complex constructs,2023,OECD,https://www.oecd-ilibrary.org/cross-cultural-validity-and-comparability-in-assessments-of-complex-constructs_91f3d034-en.pdf?itemId=%2Fcontent%2Fcomponent%2F91f3d034-en&mimeType=pdf
209
+ OECD iLibrary,Shifting global gravity,2019,OECD,https://www.oecd-ilibrary.org/shifting-global-gravity_5j3x77mdjbbp.pdf?itemId=%2Fcontent%2Fcomponent%2Ftrends_edu-2019-4-en&mimeType=pdf
210
+ OECD iLibrary,Setting the stage: Approaches to assessing AI’s impact,2023,OECD,https://www.oecd-ilibrary.org/setting-the-stage-approaches-to-assessing-ai-s-impact_8ee0810f-en.pdf?itemId=%2Fcontent%2Fcomponent%2F8ee0810f-en&mimeType=pdf
211
+ OECD iLibrary,Seizing the productive potential of digital change,2020,OECD,https://www.oecd-ilibrary.org/seizing-the-productive-potential-of-digital-change_af0dffc7-en.pdf?itemId=%2Fcontent%2Fcomponent%2Faf0dffc7-en&mimeType=pdf
212
+ OECD iLibrary,Scaling up ambitious learning practices,2019,OECD,https://www.oecd-ilibrary.org/scaling-up-ambitious-learning-practices_97c31bfe-en.pdf?itemId=%2Fcontent%2Fcomponent%2F97c31bfe-en&mimeType=pdf
213
+ OECD iLibrary,Robot scientists: From Adam to Eve to Genesis,2023,OECD,https://www.oecd-ilibrary.org/robot-scientists-from-adam-to-eve-to-genesis_d89b39b6-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd89b39b6-en&mimeType=pdf
214
+ OECD iLibrary,Risk-based control in Spain: A foundation for improved analytics,2021,OECD,https://www.oecd-ilibrary.org/risk-based-control-in-spain-a-foundation-for-improved-analytics_f3c1b67c-en.pdf?itemId=%2Fcontent%2Fcomponent%2Ff3c1b67c-en&mimeType=pdf
215
+ OECD iLibrary,Questions to guide assessment of artificial intelligence systems against human performance,2021,OECD,https://www.oecd-ilibrary.org/questions-to-guide-assessment-of-artificial-intelligence-systems-against-human-performance_344b0fa3-en.pdf?itemId=%2Fcontent%2Fcomponent%2F344b0fa3-en&mimeType=pdf
216
+ OECD iLibrary,Putting in place key enablers for AI in the public sector,2022,OECD,https://www.oecd-ilibrary.org/putting-in-place-key-enablers-for-ai-in-the-public-sector_e88632a9-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fe88632a9-en&mimeType=pdf
217
+ OECD iLibrary,Public policy considerations,2019,OECD,https://www.oecd-ilibrary.org/public-policy-considerations_969ff07f-en.pdf?itemId=%2Fcontent%2Fcomponent%2F969ff07f-en&mimeType=pdf
218
+ OECD iLibrary,Previous work,2019,OECD,https://www.oecd-ilibrary.org/previous-work_633d28e7-en.pdf?itemId=%2Fcontent%2Fcomponent%2F633d28e7-en&mimeType=pdf
219
+ OECD iLibrary,Preface,2019,OECD,https://www.oecd-ilibrary.org/preface_89af43d8-en.pdf?itemId=%2Fcontent%2Fcomponent%2F89af43d8-en&mimeType=pdf
220
+ OECD iLibrary,Personalisation of learning: Towards hybrid human-AI learning technologies,2021,OECD,https://www.oecd-ilibrary.org/personalisation-of-learning-towards-hybrid-human-ai-learning-technologies_2cc25e37-en.pdf?itemId=%2Fcontent%2Fcomponent%2F2cc25e37-en&mimeType=pdf
221
+ OECD iLibrary,Performance of the SDG identification algorithm,2021,OECD,https://www.oecd-ilibrary.org/performance-of-the-sdg-identification-algorithm_d7af6eac-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd7af6eac-en&mimeType=pdf
222
+ OECD iLibrary,Overview: What the TALIS‑PISA link insights imply for policy and future research,2021,OECD,https://www.oecd-ilibrary.org/overview-what-the-talis-pisa-link-insights-imply-for-policy-and-future-research_9bc425ba-en.pdf?itemId=%2Fcontent%2Fcomponent%2F9bc425ba-en&mimeType=pdf
223
+ OECD iLibrary,On evaluating artificial intelligence systems: Competitions and benchmarks,2021,OECD,https://www.oecd-ilibrary.org/on-evaluating-artificial-intelligence-systems-competitions-and-benchmarks_d755c6d6-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd755c6d6-en&mimeType=pdf
224
+ OECD iLibrary,New approaches to understanding the impact of computers on work and education,2021,OECD,https://www.oecd-ilibrary.org/new-approaches-to-understanding-the-impact-of-computers-on-work-and-education_65774a32-en.pdf?itemId=%2Fcontent%2Fcomponent%2F65774a32-en&mimeType=pdf
225
+ OECD iLibrary,National policies for Artificial Intelligence: What about diffusion?,2021,OECD,https://www.oecd-ilibrary.org/national-policies-for-artificial-intelligence-what-about-diffusion_cc3a9728-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fcc3a9728-en&mimeType=pdf
226
+ OECD iLibrary,Methodology for assessing AI capabilities using the Survey of Adult Skills (PIAAC),2023,OECD,https://www.oecd-ilibrary.org/methodology-for-assessing-ai-capabilities-using-the-survey-of-adult-skills-piaac_b70b1373-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fb70b1373-en&mimeType=pdf
227
+ OECD iLibrary,Methodological guidelines to define functional areas,2020,OECD,https://www.oecd-ilibrary.org/methodological-guidelines-to-define-functional-areas_782b6659-en.pdf?itemId=%2Fcontent%2Fcomponent%2F782b6659-en&mimeType=pdf
228
+ OECD iLibrary,Data and indicator gaps on pressures and responses,2019,OECD,https://www.oecd-ilibrary.org/data-and-indicator-gaps-on-pressures-and-responses_bc88d5bf-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fbc88d5bf-en&mimeType=pdf
229
+ OECD iLibrary,Media freedoms and digital rights in Finland,2021,OECD,https://www.oecd-ilibrary.org/media-freedoms-and-digital-rights-in-finland_919641b1-en.pdf?itemId=%2Fcontent%2Fcomponent%2F919641b1-en&mimeType=pdf
230
+ OECD iLibrary,Media freedoms and civic space in the digital age in Romania,2023,OECD,https://www.oecd-ilibrary.org/media-freedoms-and-civic-space-in-the-digital-age-in-romania_cbc4ab42-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fcbc4ab42-en&mimeType=pdf
231
+ OECD iLibrary,Measuring the share of the private sector related to the SDGs,2021,OECD,https://www.oecd-ilibrary.org/measuring-the-share-of-the-private-sector-related-to-the-sdgs_4de00ac5-en.pdf?itemId=%2Fcontent%2Fcomponent%2F4de00ac5-en&mimeType=pdf
232
+ OECD iLibrary,Managing access to AI advances to safeguard countries’ essential security interests,2021,OECD,https://www.oecd-ilibrary.org/managing-access-to-ai-advances-to-safeguard-countries-essential-security-interests_2e819dc8-en.pdf?itemId=%2Fcontent%2Fcomponent%2F2e819dc8-en&mimeType=pdf
233
+ OECD iLibrary,Main findings and recommendations,2019,OECD,https://www.oecd-ilibrary.org/main-findings-and-recommendations_78f937cc-en.pdf?itemId=%2Fcontent%2Fcomponent%2F78f937cc-en&mimeType=pdf
234
+ OECD iLibrary,Skill needs and policies in the age of artificial intelligence,2023,OECD,https://www.oecd-ilibrary.org/skill-needs-and-policies-in-the-age-of-artificial-intelligence_638df49a-en.pdf?itemId=%2Fcontent%2Fcomponent%2F638df49a-en&mimeType=pdf
235
+ OECD iLibrary,Skills assessments in education,2021,OECD,https://www.oecd-ilibrary.org/skills-assessments-in-education_68191ce9-en.pdf?itemId=%2Fcontent%2Fcomponent%2F68191ce9-en&mimeType=pdf
236
+ OECD iLibrary,Social dialogue and collective bargaining in the age of artificial intelligence,2023,OECD,https://www.oecd-ilibrary.org/social-dialogue-and-collective-bargaining-in-the-age-of-artificial-intelligence_c35af387-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc35af387-en&mimeType=pdf
237
+ OECD iLibrary,Social Robots as educators,2021,OECD,https://www.oecd-ilibrary.org/social-robots-as-educators_1c3b1d56-en.pdf?itemId=%2Fcontent%2Fcomponent%2F1c3b1d56-en&mimeType=pdf
238
+ OECD iLibrary,Zusammenfassung,2020,OECD,https://www.oecd-ilibrary.org/zusammenfassung_adae5bf3-de.pdf?itemId=%2Fcontent%2Fcomponent%2Fadae5bf3-de&mimeType=pdf
239
+ OECD iLibrary,"Why everyone in Finland’s teaching themselves AI. Teemu Roos, U of Helsinki, tells us.",2019,OECD,https://www.oecd-ilibrary.org/why-everyone-in-finland-s-teaching-themselves-ai-teemu-roos-u-of-helsinki-tells-us_d1f98cfe-en.pdf?itemId=%2Fcontent%2Fmultimedia%2Fd1f98cfe-en&mimeType=pdf
240
+ OECD iLibrary,What do teachers and schools do that matters most for students' social and emotional development?,2021,OECD,https://www.oecd-ilibrary.org/what-do-teachers-and-schools-do-that-matters-most-for-students-social-and-emotional-development_d44d0f58-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd44d0f58-en&mimeType=pdf
241
+ OECD iLibrary,What can governments do to be ready for the future of work? With OECD’s Stefano Scarpetta,2019,OECD,https://www.oecd-ilibrary.org/what-can-governments-do-to-be-ready-for-the-future-of-work-with-oecd-s-stefano-scarpetta_2957db51-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F2957db51-en&mimeType=pdf
242
+ OECD iLibrary,What can artificial intelligence do for physics?,2023,OECD,https://www.oecd-ilibrary.org/what-can-artificial-intelligence-do-for-physics_724b14a6-en.pdf?itemId=%2Fcontent%2Fcomponent%2F724b14a6-en&mimeType=pdf
243
+ OECD iLibrary,β€œVirtual Assistant” for VAT,2019,OECD,https://www.oecd-ilibrary.org/virtual-assistant-for-vat_faa89db0-en.pdf?itemId=%2Fcontent%2Fcomponent%2Ffaa89db0-en&mimeType=pdf
244
+ OECD iLibrary,Using machine learning to verify scientific claims,2023,OECD,https://www.oecd-ilibrary.org/using-machine-learning-to-verify-scientific-claims_ab9d235c-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fab9d235c-en&mimeType=pdf
245
+ OECD iLibrary,Using human skills taxonomies and tests as measures of artificial intelligence,2021,OECD,https://www.oecd-ilibrary.org/using-human-skills-taxonomies-and-tests-as-measures-of-artificial-intelligence_e182dd0d-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fe182dd0d-en&mimeType=pdf
246
+ OECD iLibrary,Understanding digital transformation,2019,OECD,https://www.oecd-ilibrary.org/understanding-digital-transformation_58ee7fe5-en.pdf?itemId=%2Fcontent%2Fcomponent%2F58ee7fe5-en&mimeType=pdf
247
+ OECD iLibrary,Trends Shaping Education 2019,2019,OECD,https://www.oecd-ilibrary.org/trends-shaping-education-2019_5j8jll3f016g.pdf?itemId=%2Fcontent%2Fpublication%2Ftrends_edu-2019-en&mimeType=pdf
248
+ OECD iLibrary,Trends and policy frameworks for AI in finance,2021,OECD,https://www.oecd-ilibrary.org/trends-and-policy-frameworks-for-ai-in-finance_cbc9d1af-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fcbc9d1af-en&mimeType=pdf
249
+ OECD iLibrary,Trend 4: New ways of engaging citizens and residents,2023,OECD,https://www.oecd-ilibrary.org/trend-4-new-ways-of-engaging-citizens-and-residents_d5b3aed9-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd5b3aed9-en&mimeType=pdf
250
+ OECD iLibrary,"Machine reading: Successes, challenges and implications for science",2023,OECD,https://www.oecd-ilibrary.org/machine-reading-successes-challenges-and-implications-for-science_368178a6-en.pdf?itemId=%2Fcontent%2Fcomponent%2F368178a6-en&mimeType=pdf
251
+ OECD iLibrary,Trend 2: New approaches to care,2023,OECD,https://www.oecd-ilibrary.org/trend-2-new-approaches-to-care_9be6313e-en.pdf?itemId=%2Fcontent%2Fcomponent%2F9be6313e-en&mimeType=pdf
252
+ OECD iLibrary,The use of SupTech to enhance market supervision and integrity,2021,OECD,https://www.oecd-ilibrary.org/the-use-of-suptech-to-enhance-market-supervision-and-integrity_d478df4c-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd478df4c-en&mimeType=pdf
253
+ OECD iLibrary,The technical landscape,2019,OECD,https://www.oecd-ilibrary.org/the-technical-landscape_8b303b6f-en.pdf?itemId=%2Fcontent%2Fcomponent%2F8b303b6f-en&mimeType=pdf
254
+ OECD iLibrary,The skill profiles and the competences of workers in digital occupations,2022,OECD,https://www.oecd-ilibrary.org/the-skill-profiles-and-the-competences-of-workers-in-digital-occupations_6bafcb87-en.pdf?itemId=%2Fcontent%2Fcomponent%2F6bafcb87-en&mimeType=pdf
255
+ OECD iLibrary,The rule of law: A fragile tool for the development of emerging nuclear technologies,2023,OECD,https://www.oecd-ilibrary.org/the-rule-of-law-a-fragile-tool-for-the-development-of-emerging-nuclear-technologies_3bd72ca3-en.pdf?itemId=%2Fcontent%2Fpaper%2F3bd72ca3-en&mimeType=pdf
256
+ OECD iLibrary,The importance of knowledge bases for artificial intelligence in science,2023,OECD,https://www.oecd-ilibrary.org/the-importance-of-knowledge-bases-for-artificial-intelligence-in-science_3aba9f4b-en.pdf?itemId=%2Fcontent%2Fcomponent%2F3aba9f4b-en&mimeType=pdf
257
+ OECD iLibrary,The encroachment of artificial intelligence: Timing and prospects,2021,OECD,https://www.oecd-ilibrary.org/the-encroachment-of-artificial-intelligence-timing-and-prospects_e39255bb-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fe39255bb-en&mimeType=pdf
258
+ OECD iLibrary,The economic landscape,2019,OECD,https://www.oecd-ilibrary.org/the-economic-landscape_3abc27f1-en.pdf?itemId=%2Fcontent%2Fcomponent%2F3abc27f1-en&mimeType=pdf
259
+ OECD iLibrary,The case for a digital government in Mexico,2020,OECD,https://www.oecd-ilibrary.org/the-case-for-a-digital-government-in-mexico_6a7384e3-en.pdf?itemId=%2Fcontent%2Fcomponent%2F6a7384e3-en&mimeType=pdf
260
+ OECD iLibrary,The air we breathe : Rich Fuller of Pure Earth,2019,OECD,https://www.oecd-ilibrary.org/the-air-we-breathe-rich-fuller-of-pure-earth_9a00aff1-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F9a00aff1-en&mimeType=pdf
261
+ OECD iLibrary,Test No. 471: Bacterial Reverse Mutation Test,2020,OECD,https://www.oecd-ilibrary.org/test-no-471-bacterial-reverse-mutation-test_5lmqcr2k7mg0.pdf?itemId=%2Fcontent%2Fpublication%2F9789264071247-en&mimeType=pdf
262
+ OECD iLibrary,Technische Grundlagen,2020,OECD,https://www.oecd-ilibrary.org/technische-grundlagen_64ae49f7-de.pdf?itemId=%2Fcontent%2Fcomponent%2F64ae49f7-de&mimeType=pdf
263
+ OECD iLibrary,Tasks and tests for assessing artificial intelligence and robotics in comparison with humans,2021,OECD,https://www.oecd-ilibrary.org/tasks-and-tests-for-assessing-artificial-intelligence-and-robotics-in-comparison-with-humans_265f8d24-en.pdf?itemId=%2Fcontent%2Fcomponent%2F265f8d24-en&mimeType=pdf
264
+ OECD iLibrary,Trend 1: New forms of accountability for a new era of government,2023,OECD,https://www.oecd-ilibrary.org/trend-1-new-forms-of-accountability-for-a-new-era-of-government_1b12de43-en.pdf?itemId=%2Fcontent%2Fcomponent%2F1b12de43-en&mimeType=pdf
265
+ OECD iLibrary,Leveraging artificial intelligence for proactive delivery of public policies and services,2023,OECD,https://www.oecd-ilibrary.org/leveraging-artificial-intelligence-for-proactive-delivery-of-public-policies-and-services_5702d250-en.pdf?itemId=%2Fcontent%2Fcomponent%2F5702d250-en&mimeType=pdf
266
+ OECD iLibrary,Methodological algorithms,2020,OECD,https://www.oecd-ilibrary.org/methodological-algorithms_caf29d59-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fcaf29d59-en&mimeType=pdf
267
+ OECD iLibrary,LAC Artificial Intelligence strategies,2022,OECD,https://www.oecd-ilibrary.org/lac-artificial-intelligence-strategies_636827ae-en.pdf?itemId=%2Fcontent%2Fcomponent%2F636827ae-en&mimeType=pdf
268
+ OECD iLibrary,Artificial intelligence for science and engineering: A priority for public investment in research and development,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-for-science-and-engineering-a-priority-for-public-investment-in-research-and-development_7b7b1bce-en.pdf?itemId=%2Fcontent%2Fcomponent%2F7b7b1bce-en&mimeType=pdf
269
+ OECD iLibrary,Artificial intelligence for science in Africa,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-for-science-in-africa_399730cc-en.pdf?itemId=%2Fcontent%2Fcomponent%2F399730cc-en&mimeType=pdf
270
+ OECD iLibrary,Artificial intelligence in education: Bringing it all together,2021,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-in-education-bringing-it-all-together_f54ea644-en.pdf?itemId=%2Fcontent%2Fcomponent%2Ff54ea644-en&mimeType=pdf
271
+ OECD iLibrary,Artificial intelligence in science: Overview and policy proposals,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-in-science-overview-and-policy-proposals_a2817e1f-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fa2817e1f-en&mimeType=pdf
272
+ OECD iLibrary,Artificial intelligence in scientific discovery: Challenges and opportunities,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-in-scientific-discovery-challenges-and-opportunities_ca841465-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fca841465-en&mimeType=pdf
273
+ OECD iLibrary,Lessons from shortcomings in machine learning for medical imaging,2023,OECD,https://www.oecd-ilibrary.org/lessons-from-shortcomings-in-machine-learning-for-medical-imaging_b885eecd-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fb885eecd-en&mimeType=pdf
274
+ OECD iLibrary,Artificial Intelligence: Managing the ethical challenges,2021,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-managing-the-ethical-challenges_54c51e69-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F54c51e69-en&mimeType=pdf
275
+ OECD iLibrary,Artificial Intelligence: Regulation Can Support Innovation,2021,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-regulation-can-support-innovation_f7fe0e1d-en.pdf?itemId=%2Fcontent%2Fmultimedia%2Ff7fe0e1d-en&mimeType=pdf
276
+ OECD iLibrary,Assessing artificial intelligence capabilities,2021,OECD,https://www.oecd-ilibrary.org/assessing-artificial-intelligence-capabilities_47d04fe3-en.pdf?itemId=%2Fcontent%2Fcomponent%2F47d04fe3-en&mimeType=pdf
277
+ OECD iLibrary,Assessing collective intelligence in human groups,2021,OECD,https://www.oecd-ilibrary.org/assessing-collective-intelligence-in-human-groups_ce863473-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fce863473-en&mimeType=pdf
278
+ OECD iLibrary,Assessing Household Financial Vulnerability: Empirical evidence from the U.S. using machine learning,2019,OECD,https://www.oecd-ilibrary.org/assessing-household-financial-vulnerability-empirical-evidence-from-the-u-s-using-machine-learning_75c63aa1-en.pdf?itemId=%2Fcontent%2Fcomponent%2F75c63aa1-en&mimeType=pdf
279
+ OECD iLibrary,Assessing Natural Language Processing,2021,OECD,https://www.oecd-ilibrary.org/assessing-natural-language-processing_fcd5e244-en.pdf?itemId=%2Fcontent%2Fcomponent%2Ffcd5e244-en&mimeType=pdf
280
+ OECD iLibrary,Artificial Intelligence-enabled adaptive assessments with Intelligent Tutors,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-enabled-adaptive-assessments-with-intelligent-tutors_22731ca8-en.pdf?itemId=%2Fcontent%2Fcomponent%2F22731ca8-en&mimeType=pdf
281
+ OECD iLibrary,Assessment and recommendations,2019,OECD,https://www.oecd-ilibrary.org/assessment-and-recommendations_cdb89610-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fcdb89610-en&mimeType=pdf
282
+ OECD iLibrary,Bringing space to earth with data-driven activities,2019,OECD,https://www.oecd-ilibrary.org/bringing-space-to-earth-with-data-driven-activities_07d50927-en.pdf?itemId=%2Fcontent%2Fcomponent%2F07d50927-en&mimeType=pdf
283
+ OECD iLibrary,Building an assessment of artificial intelligence capabilities,2021,OECD,https://www.oecd-ilibrary.org/building-an-assessment-of-artificial-intelligence-capabilities_01421d08-en.pdf?itemId=%2Fcontent%2Fcomponent%2F01421d08-en&mimeType=pdf
284
+ OECD iLibrary,Building key governance capacities,2022,OECD,https://www.oecd-ilibrary.org/building-key-governance-capacities_56a4e1d2-en.pdf?itemId=%2Fcontent%2Fcomponent%2F56a4e1d2-en&mimeType=pdf
285
+ OECD iLibrary,Case study: Planning for the future of work,2021,OECD,https://www.oecd-ilibrary.org/case-study-planning-for-the-future-of-work_ff4b8bd3-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fff4b8bd3-en&mimeType=pdf
286
+ OECD iLibrary,Changes in AI capabilities in literacy and numeracy between 2016 and 2021,2023,OECD,https://www.oecd-ilibrary.org/changes-in-ai-capabilities-in-literacy-and-numeracy-between-2016-and-2021_4301a378-en.pdf?itemId=%2Fcontent%2Fcomponent%2F4301a378-en&mimeType=pdf
287
+ OECD iLibrary,Combining collective and machine intelligence at the knowledge frontier,2023,OECD,https://www.oecd-ilibrary.org/combining-collective-and-machine-intelligence-at-the-knowledge-frontier_dbbd48a9-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fdbbd48a9-en&mimeType=pdf
288
+ OECD iLibrary,Common sense skills: Artificial intelligence and the workplace,2021,OECD,https://www.oecd-ilibrary.org/common-sense-skills-artificial-intelligence-and-the-workplace_5fe70a0c-en.pdf?itemId=%2Fcontent%2Fcomponent%2F5fe70a0c-en&mimeType=pdf
289
+ OECD iLibrary,Competence assessment in German vocational education and training,2021,OECD,https://www.oecd-ilibrary.org/competence-assessment-in-german-vocational-education-and-training_c3d7aa74-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc3d7aa74-en&mimeType=pdf
290
+ OECD iLibrary,Competition and AI,2021,OECD,https://www.oecd-ilibrary.org/competition-and-ai_3acbe1cd-en.pdf?itemId=%2Fcontent%2Fcomponent%2F3acbe1cd-en&mimeType=pdf
291
+ OECD iLibrary,Conclusion and recommendations,2022,OECD,https://www.oecd-ilibrary.org/conclusion-and-recommendations_c61de01b-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc61de01b-en&mimeType=pdf
292
+ OECD iLibrary,Consumer policy in the digital transformation,2020,OECD,https://www.oecd-ilibrary.org/consumer-policy-in-the-digital-transformation_7570fa4a-en.pdf?itemId=%2Fcontent%2Fcomponent%2F7570fa4a-en&mimeType=pdf
293
+ OECD iLibrary,Cracking the glass ceiling on wages: Gapsquare CEO Zara Nanu,2019,OECD,https://www.oecd-ilibrary.org/cracking-the-glass-ceiling-on-wages-gapsquare-ceo-zara-nanu_ca12b314-en.pdf?itemId=%2Fcontent%2Fmultimedia%2Fca12b314-en&mimeType=pdf
294
+ OECD iLibrary,Bringing health into the 21st century,2019,OECD,https://www.oecd-ilibrary.org/bringing-health-into-the-21st-century_e130fcc2-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fe130fcc2-en&mimeType=pdf
295
+ OECD iLibrary,"Artificial intelligence, digital technology and advanced production",2020,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-digital-technology-and-advanced-production_629af843-en.pdf?itemId=%2Fcontent%2Fcomponent%2F629af843-en&mimeType=pdf
296
+ OECD iLibrary,"Artificial intelligence, job quality and inclusiveness",2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-job-quality-and-inclusiveness_99c4c123-en.pdf?itemId=%2Fcontent%2Fcomponent%2F99c4c123-en&mimeType=pdf
297
+ OECD iLibrary,Artificial intelligence: Changing landscape for SMEs,2021,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-changing-landscape-for-smes_01a4ae9d-en.pdf?itemId=%2Fcontent%2Fcomponent%2F01a4ae9d-en&mimeType=pdf
298
+ OECD iLibrary,Korea,2021,OECD,https://www.oecd-ilibrary.org/korea_88d8a0bc-en.pdf?itemId=%2Fcontent%2Fcomponent%2F88d8a0bc-en&mimeType=pdf
299
+ OECD iLibrary,Knowledge and power,2022,OECD,https://www.oecd-ilibrary.org/knowledge-and-power_6cca2555-en.pdf?itemId=%2Fcontent%2Fcomponent%2F6cca2555-en&mimeType=pdf
300
+ OECD iLibrary,Is there a narrowing of AI research?,2023,OECD,https://www.oecd-ilibrary.org/is-there-a-narrowing-of-ai-research_77709ef0-en.pdf?itemId=%2Fcontent%2Fcomponent%2F77709ef0-en&mimeType=pdf
301
+ OECD iLibrary,Introduction,2022,OECD,https://www.oecd-ilibrary.org/introduction_c545040a-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc545040a-en&mimeType=pdf
302
+ OECD iLibrary,Interpretability: Should – and can – we understand the reasoning of machine-learning systems?,2023,OECD,https://www.oecd-ilibrary.org/interpretability-should-and-can-we-understand-the-reasoning-of-machine-learning-systems_2d0478b9-en.pdf?itemId=%2Fcontent%2Fcomponent%2F2d0478b9-en&mimeType=pdf
303
+ OECD iLibrary,A framework for evaluating the AI-driven automation of science,2023,OECD,https://www.oecd-ilibrary.org/a-framework-for-evaluating-the-ai-driven-automation-of-science_63faa850-en.pdf?itemId=%2Fcontent%2Fcomponent%2F63faa850-en&mimeType=pdf
304
+ OECD iLibrary,"Artificial intelligence, developing-country science and bilateral co‑operation",2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-developing-country-science-and-bilateral-co-operation_4edb761e-en.pdf?itemId=%2Fcontent%2Fcomponent%2F4edb761e-en&mimeType=pdf
305
+ OECD iLibrary,Abilities and skills: Assessing humans and artificial intelligence/robotics systems,2021,OECD,https://www.oecd-ilibrary.org/abilities-and-skills-assessing-humans-and-artificial-intelligence-robotics-systems_497be8f9-en.pdf?itemId=%2Fcontent%2Fcomponent%2F497be8f9-en&mimeType=pdf
306
+ OECD iLibrary,Adaptability,2019,OECD,https://www.oecd-ilibrary.org/adaptability_3d8ecdfb-en.pdf?itemId=%2Fcontent%2Fcomponent%2F3d8ecdfb-en&mimeType=pdf
307
+ OECD iLibrary,Addressing labour-market disruptions from trade and automation,2019,OECD,https://www.oecd-ilibrary.org/addressing-labour-market-disruptions-from-trade-and-automation_e309eca0-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fe309eca0-en&mimeType=pdf
308
+ OECD iLibrary,AI and scientific productivity: Considering policy and governance challenges,2023,OECD,https://www.oecd-ilibrary.org/ai-and-scientific-productivity-considering-policy-and-governance-challenges_84cde28b-en.pdf?itemId=%2Fcontent%2Fcomponent%2F84cde28b-en&mimeType=pdf
309
+ OECD iLibrary,AI applications,2019,OECD,https://www.oecd-ilibrary.org/ai-applications_79edf9d8-en.pdf?itemId=%2Fcontent%2Fcomponent%2F79edf9d8-en&mimeType=pdf
310
+ OECD iLibrary,AI in drug discovery,2023,OECD,https://www.oecd-ilibrary.org/ai-in-drug-discovery_6717b361-en.pdf?itemId=%2Fcontent%2Fcomponent%2F6717b361-en&mimeType=pdf
311
+ OECD iLibrary,A tale of two worlds: Machine learning approaches at the intersection with educational measurement,2023,OECD,https://www.oecd-ilibrary.org/a-tale-of-two-worlds-machine-learning-approaches-at-the-intersection-with-educational-measurement_d01eb8a4-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fd01eb8a4-en&mimeType=pdf
312
+ OECD iLibrary,AI policies and initiatives,2019,OECD,https://www.oecd-ilibrary.org/ai-policies-and-initiatives_cf3f3be0-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fcf3f3be0-en&mimeType=pdf
313
+ OECD iLibrary,AI in finance,2021,OECD,https://www.oecd-ilibrary.org/ai-in-finance_39b6299a-en.pdf?itemId=%2Fcontent%2Fcomponent%2F39b6299a-en&mimeType=pdf
314
+ OECD iLibrary,"Artificial intelligence, blockchain and quantum computing",2020,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-blockchain-and-quantum-computing_c51bcfeb-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fc51bcfeb-en&mimeType=pdf
315
+ OECD iLibrary,Artificial intelligence and the labour market: Introduction,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-and-the-labour-market-introduction_63bcc69a-en.pdf?itemId=%2Fcontent%2Fcomponent%2F63bcc69a-en&mimeType=pdf
316
+ OECD iLibrary,Artificial intelligence and jobs: No signs of slowing labour demand (yet),2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-and-jobs-no-signs-of-slowing-labour-demand-yet_9c86de40-en.pdf?itemId=%2Fcontent%2Fcomponent%2F9c86de40-en&mimeType=pdf
317
+ OECD iLibrary,Artificial intelligence and development projects: A case study in funding mechanisms to optimise research excellence in sub-Saharan Africa,2023,OECD,https://www.oecd-ilibrary.org/artificial-intelligence-and-development-projects-a-case-study-in-funding-mechanisms-to-optimise-research-excellence-in-sub-saharan-africa_f41aa308-en.pdf?itemId=%2Fcontent%2Fcomponent%2Ff41aa308-en&mimeType=pdf
318
+ OECD iLibrary,Identifying artificial intelligence capabilities: What and how to test,2021,OECD,https://www.oecd-ilibrary.org/identifying-artificial-intelligence-capabilities-what-and-how-to-test_85aeb432-en.pdf?itemId=%2Fcontent%2Fcomponent%2F85aeb432-en&mimeType=pdf
319
+ OECD iLibrary,Implications of evolving AI capabilities for employment and education,2023,OECD,https://www.oecd-ilibrary.org/implications-of-evolving-ai-capabilities-for-employment-and-education_ed919621-en.pdf?itemId=%2Fcontent%2Fcomponent%2Fed919621-en&mimeType=pdf
320
+ OECD iLibrary,ANPAL Guidelines,2019,OECD,https://www.oecd-ilibrary.org/anpal-guidelines_1373d882-en.pdf?itemId=%2Fcontent%2Fcomponent%2F1373d882-en&mimeType=pdf
321
+ OECD iLibrary,An overview of key developments and policies,2020,OECD,https://www.oecd-ilibrary.org/an-overview-of-key-developments-and-policies_5260917b-en.pdf?itemId=%2Fcontent%2Fcomponent%2F5260917b-en&mimeType=pdf
322
+ OECD iLibrary,An occupational taxonomic approach to assessing AI capabilities,2021,OECD,https://www.oecd-ilibrary.org/an-occupational-taxonomic-approach-to-assessing-ai-capabilities_2c1d8961-en.pdf?itemId=%2Fcontent%2Fcomponent%2F2c1d8961-en&mimeType=pdf
323
+ OECD iLibrary,All about AI: Should we be concerned about artificial intelligence?,2020,OECD,https://www.oecd-ilibrary.org/all-about-ai-should-we-be-concerned-about-artificial-intelligence_1b63c3bd-en.pdf?itemId=%2Fcontent%2Fmultimedia%2F1b63c3bd-en&mimeType=pdf
324
+ OECD iLibrary,AI use cases in LAC governments,2022,OECD,https://www.oecd-ilibrary.org/ai-use-cases-in-lac-governments_08955f48-en.pdf?itemId=%2Fcontent%2Fcomponent%2F08955f48-en&mimeType=pdf
325
+ OECD iLibrary,"Applying AI to real-world health-care settings and the life sciences: Tackling data privacy, security and policy challenges with federated learning",2023,OECD,https://www.oecd-ilibrary.org/applying-ai-to-real-world-health-care-settings-and-the-life-sciences-tackling-data-privacy-security-and-policy-challenges-with-federated-learning_058a23d0-en.pdf?itemId=%2Fcontent%2Fcomponent%2F058a23d0-en&mimeType=pdf
326
+ OECD publications,What are the OECD Principles on AI,2020,OECD,https://www.oecd-ilibrary.org/docserver/6ff2a1c4-en.pdf?expires=1684155213&id=id&accname=guest&checksum=47DD8B6DF2D9129E96D568E8A575B941
327
+ OECD publications,Examples of LAC AI instruments aligned with OECD AI values-based principles,2021,OECD,https://www.oecd-ilibrary.org/sites/5fbd4ea6-en/index.html?itemId=/content/component/5fbd4ea6-en
328
+ Policy documents,A Framework for Developing a National Artificial Intelligence Strategy Centre for Fourth Industrial Revolution,2019,World Economic Forum,https://www3.weforum.org/docs/WEF_National_AI_Strategy.pdf?_gl=1*1ycz2tk*_up*MQ..&gclid=EAIaIQobChMI2LuIqrL1_gIVSazVCh3WZA_0EAAYASAAEgKY9fD_BwE
329
+ Policy documents,Memorandum for the Heads of Executive Departments and Agencies,2020,United States,https://www.whitehouse.gov/wp-content/uploads/2020/01/Draft-OMB-Memo-on-Regulation-of-AI-1-7-19.pdf
330
+ Policy documents,AI Voor Nederland,2018,Netherlands,https://www.vno-ncw.nl/sites/default/files/aivnl_20181106_0.pdf
331
+ Policy documents,Preparing for the Future of Transportation,2019,United States,https://www.transportation.gov/sites/dot.gov/files/docs/policy-initiatives/automated-vehicles/320711/preparing-future-transportation-automated-vehicle-30.pdf
332
+ Policy documents,Outline for a German Strategy for Artificial Intelligence,2018,Germany,https://www.stiftung-nv.de/sites/default/files/outline_for_a_german_artificial_intelligence_strategy.pdf
333
+ Policy documents,AUS AI Standards Roadmap,2020,Australia,https://www.standards.org.au/getmedia/ede81912-55a2-4d8e-849f-9844993c3b9d/O_1515-An-Artificial-Intelligence-Standards-Roadmap-soft_1.pdf.aspx
334
+ Policy documents,AI Report,2019,Estonia,https://www.ria.ee/en/media/580/download
335
+ Policy documents,Draft AI R&D Guidelines for International Discussions,2017,Japan,https://www.soumu.go.jp/main_content/000507517.pdf
336
+ Policy documents,Saudi Vision 2030,2019,Saudi Arabia,https://www.saudiembassy.net/sites/default/files/u66/Saudi_Vision2030_EN.pdf
337
+ Policy documents,National Strategy for Artificial Intelligence,2020,Norway,https://www.regjeringen.no/contentassets/1febbbb2c4fd4b7d92c67ddd353b6ae8/en-gb/pdfs/ki-strategi_en.pdf
338
+ Policy documents,Saudi PWC Economic Potential AI Middle East,2018,Saudi Arabia,https://www.pwc.com/m1/en/publications/documents/economic-potential-ai-middle-east.pdf
339
+ Policy documents,Model Artificial Intelligence Governance Framework Second Edition,2020,Singapore,https://www.pdpc.gov.sg/-/media/files/pdpc/pdf-files/resource-for-organisation/ai/sgmodelaigovframework2.pdf
340
+ Policy documents,Draft AI Utilization Principles,2018,Japan,https://www.soumu.go.jp/main_content/000581310.pdf
341
+ Policy documents,Examples Of Ai National Policies,2020,OECD,https://www.oecd.org/sti/examples-of-ai-national-policies.pdf
342
+ Policy documents,Unlocking Armenia’s AI Potential,2021,Armenia,/
343
+ Policy documents,Artificial Intelligence Research Agenda for the Netherlands,2019,Netherlands,https://www.nwo.nl/sites/nwo/files/documents/AIREA-NL%20AI%20Research%20Agenda%20for%20the%20Netherlands.pdf
344
+ Policy documents,National Strategy for Artificial Intelligence,2019,Denmark,https://eng.em.dk/media/13081/305755-gb-version_4k.pdf
345
+ Policy documents,DIRECTIVE (EU) 2019/1024 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 20 June 2019 on open data and the re-use of public sector information,2019,European Union,https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32019L1024
346
+ Policy documents,"Report on the safety and liability implications of Artificial Intelligence, the Internet of Things and robotics",2020,European Union,https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52020DC0064
347
+ Policy documents,2030 Digital Compass: the European way for the Digital Decade,2021,European Union,https://eur-lex.europa.eu/resource.html?uri=cellar:12e835e2-81af-11eb-9ac9-01aa75ed71a1.0001.02/DOC_1&format=PDF
348
+ Policy documents,Singapore Compendium of Use Cases Vol 2,2020,Singapore,https://file.go.gov.sg/ai-gov-use-cases-2.pdf
349
+ Policy documents,Zonas Libres Tecnologicas,2020,Portugal,https://files.dre.pt/1s/2020/04/07800/0000600032.pdf
350
+ Policy documents,New Generation of Artificial Intelligence Development Plan,2017,China,https://flia.org/wp-content/uploads/2017/07/A-New-Generation-of-Artificial-Intelligence-Development-Plan-1.pdf
351
+ Policy documents,Towards an AI Strategy In Mexico: Harnessing the AI Revolution,2018,Mexico,https://go.wizeline.com/rs/571-SRN-279/images/Towards-an-AI-strategy-in-Mexico.pdf
352
+ Policy documents,Strategy for Denmark’s Digital Growth,2018,Denmark,https://eng.em.dk/media/10566/digital-growth-strategy-report_uk_web-2.pdf
353
+ Policy documents,Artificial Intelligence: A Strategic Vision for Luxembourg.,2019,Luxembourg,https://gouvernement.lu/dam-assets/fr/publications/rapport-etude-analyse/minist-digitalisation/Artificial-Intelligence-a-strategic-vision-for-Luxembourg.pdf
354
+ Policy documents,Artificial Ingelligence at the service of citizens,2018,Italy,https://ia.italia.it/assets/whitepaper.pdf
355
+ Policy documents,Portugal INCoDe 2030,2021,Portugal,https://incode2030.pt/sites/default/files/incode2030_en_0.pdf
356
+ Policy documents,National Strategy for Artificial Ingelligence,2018,India,https://indiaai.gov.in/documents/pdf/NationalStrategy-for-AI-Discussion-Paper.pdf
357
+ Policy documents,China AI Development Report,2018,China,https://indianstrategicknowledgeonline.com/web/China_AI_development_report_2018.pdf
358
+ Policy documents,Finland’s Age of Artificial Intelligence,2017,Finland,https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/160391/TEMrap_47_2017_verkkojulkaisu.pdf
359
+ Policy documents,Work in the age of artificial intelligence,2018,Finland,https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/160980/TEMjul_21_2018_Work_in_the_age.pdf
360
+ Policy documents,Malta Towards an AI Strategy: High-level policy document for public consultation,2019,Malta,https://malta.ai/wp-content/uploads/2019/04/Draft_Policy_document_-_online_version.pdf
361
+ Policy documents,Summary of the 2018 Department of Defense Artificial Intelligence Strategy,2018,United States,https://media.defense.gov/2019/Feb/12/2002088963/-1/-1/1/SUMMARY-OF-DOD-AI-STRATEGY.PDF
362
+ Policy documents,The Data-Driven Innovation Strategy for the Development of a Trusted and Sustainable Economy in Luxembourg,2019,Luxembourg,https://gouvernement.lu/dam-assets/fr/publications/rapport-etude-analyse/minist-economie/The-Data-driven-Innovation-Strategy.pdf
363
+ Policy documents,Data for the Benefit of the People: Recommendations from the Danish Expert Group on Data Ethics,2018,Denmark,https://em.dk/media/12190/dataethics-v2.pdf
364
+ Policy documents,Shaping Europe's digital future – Questions and Answers,2020,European Union,https://ec.europa.eu/commission/presscorner/api/files/document/print/en/qanda_20_264/QANDA_20_264_EN.pdf
365
+ Policy documents,Member States and Commission to work together to boost artificial intelligence made in Europe,2018,European Union,https://ec.europa.eu/commission/presscorner/api/files/document/print/en/ip_18_6689/IP_18_6689_EN.pdf
366
+ Policy documents,"AUS Tech Future: Delivering a Strong, Safe and Inclusive Digital Economy",2018,Australia,Missing
367
+ Policy documents,The Effective and Ethical Development of Artificial Intelligence,2019,Australia,https://acola.org/wp-content/uploads/2019/07/hs4_artificial-intelligence-report.pdf
368
+ Policy documents,AI 4 Belgium,2019,Belgium,https://ai4belgium.be/wp-content/uploads/2019/04/report_en.pdf
369
+ Policy documents,National AI Strategy,2021,United Kingdom,https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1020402/National_AI_Strategy_-_PDF_version.pdf
370
+ Policy documents,The Pathway to Driverless Cars Summary report and action plan,2015,United Kingdom,https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/401562/pathway-driverless-cars-summary.pdf
371
+ Policy documents,Growing the Artificial Intelligence Industry in the UK,2019,United Kingdom,https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/652097/Growing_the_artificial_intelligence_industry_in_the_UK.pdf
372
+ Policy documents,Industrial Strategy Artificial Intelligence Sector Deal,2018,United Kingdom,https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/702810/180425_BEIS_AI_Sector_Deal__4_.pdf
373
+ Policy documents,Realising the potential of technology in education: A strategy for education providers and the technology industry,2019,United Kingdom,https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/791931/DfE-Education_Technology_Strategy.pdf
374
+ Policy documents,"Strategic Research, Innovation and Deployment Agenda- AI, Data and Robotics Partnership",2020,European Union,https://bdva.eu/sites/default/files/AI-Data-Robotics-Partnership-SRIDA%20V3.1.pdf
375
+ Policy documents,Ethics Commission Automated And Connected Driving,2017,Germany,https://bmdv.bund.de/SharedDocs/EN/publications/report-ethics-commission-automated-and-connected-driving.pdf?__blob=publicationFile
376
+ Policy documents,Rebooting Regulation: Exploring the Future of AI Policy in Canada,2020,Canada,https://cifar.ca/wp-content/uploads/2020/01/rebooting-regulation-exploring-the-future-of-ai-policy-in-canada.pdf
377
+ Policy documents,AICan 2019- Annual Report of the CIFAR Pan-Canadian AI Strategy,2019,Canada,https://cifar.ca/wp-content/uploads/2020/04/ai_annualreport2019_web.pdf
378
+ Policy documents,Cifar Annual Report 2018-2019,2019,Canada,https://cifar.ca/wp-content/uploads/2020/04/annualreport20182019.pdf
379
+ Policy documents,Pan-Canadian AI Strategy Impact Assessment Report,2020,Canada,https://cifar.ca/wp-content/uploads/2020/11/Pan-Canadian-AI-Strategy-Impact-Assessment-Report.pdf
380
+ Policy documents,White Paper On Artificial Intelligence - A European approach to excellence and trust,2020,European Union,https://commission.europa.eu/system/files/2020-02/commission-white-paper-artificial-intelligence-feb2020_en.pdf
381
+ Policy documents,Estimating investments in General Purpose Technologies,2020,European Union,https://core.ac.uk/download/pdf/322748045.pdf
382
+ Policy documents,Artificial Intelligence Standardization White Paper,2018,China,https://cset.georgetown.edu/wp-content/uploads/t0120_AI_standardization_white_paper_EN.pdf
383
+ Policy documents,AI Technology Roadmap,2019,Australia,https://data61.csiro.au/~/media/D61/AI-Roadmap-assets/19-00346_DATA61_REPORT_AI-Roadmap_WEB_191111.pdf?la=en&hash=58386288921D9C21EC8C4861CDFD863F1FBCD457
384
+ Policy documents,Summary of the National Defense Strategy of The United States of America,2018,United States,https://dod.defense.gov/Portals/1/Documents/pubs/2018-National-Defense-Strategy-Summary.pdf
385
+ Policy documents,AI Principles: Recommendations on the Ethical Use of Artificial Intelligence by the Department of Defense,2019,United States,https://media.defense.gov/2019/Oct/31/2002204458/-1/-1/0/DIB_AI_PRINCIPLES_PRIMARY_DOCUMENT.PDF
386
+ Policy documents,Integrated Innovation Strategy,2018,Japan,https://www8.cao.go.jp/cstp/english/doc/integrated_main.pdf
387
+ Policy documents,"Artificial Intelligence, Automation, and the Economy",2016,United States,https://obamawhitehouse.archives.gov/sites/whitehouse.gov/files/documents/Artificial-Intelligence-Automation-Economy.PDF
388
+ Policy documents,The Age of Artificial Intelligence- Towards a European Strategy for Human-Centric Machines,2018,European Union,https://op.europa.eu/en/publication-detail/-/publication/f22f6811-1007-11ea-8c1f-01aa75ed71a1/language-en
389
+ Policy documents,Initial code of conduct for data-driven health and care technology,2018,United Kingdom,https://www.gov.uk/government/publications/code-of-conduct-for-data-driven-health-and-care-technology/initial-code-of-conduct-for-data-driven-health-and-care-technology
390
+ Policy documents,US Executive Order - Maintaining American Leadership in Artificial Intelligence,2019,United States,https://www.govinfo.gov/content/pkg/FR-2019-02-14/pdf/2019-02544.pdf
391
+ Policy documents,Better Together? Franco-German Cooperation on AI,2018,Germany,https://www.hertie-school.org/fileadmin/user_upload/20181218_Dt-frz-KI_Dittrich_neu.pdf
392
+ Policy documents,Industry 4.0 Testlabs in Australia Preparing for the Future,2017,Australia,https://www.industry.gov.au/sites/default/files/July%202018/document/pdf/industry-4.0-testlabs-report.pdf
393
+ Policy documents,AUS 2030-Prosperity Through Innovation,2017,Australia,https://www.industry.gov.au/sites/default/files/May%202018/document/pdf/australia-2030-prosperity-through-innovation-full-report.pdf
394
+ Policy documents,Digital Transformation towards 2030 in Portugal,2018,Portugal,https://www.itu.int/en/ITU-D/Regional-Presence/Europe/Documents/Events/2018/WSIS/Matos%20InnovWSIS-V1.pdf
395
+ Policy documents,For A Meaningful Artificial Intelligence,2018,France,https://www.jaist.ac.jp/~bao/AI/OtherAIstrategies/MissionVillani_Report_ENG-VF.pdf
396
+ Policy documents,Designing a Winning A.I. Strategy,2017,Saudi Arabia,https://www.kineticcs.com/wp-content/uploads/2017/11/Designing-A-Winning-AI-Strategy-_Kinetic-Consulting-Services.pdf
397
+ Policy documents,Nacionalni program spodbujanja razvoja in uporabe umetne inteligence v Republiki Sloveniji do leta 2025 (NpUI),2021,Slovenia,https://www.gov.si/assets/ministrstva/MJU/DID/NpUI-SI-2025.docx
398
+ Policy documents,Governance Guidelines for Implementation of AI Principles,2021,Japan,https://www.meti.go.jp/shingikai/mono_info_service/ai_shakai_jisso/pdf/20210709_9.pdf
399
+ Policy documents,Action plan for the digital transformation of Slovakia for 2019 – 2022,2019,Slovakia,https://www.mirri.gov.sk/wp-content/uploads/2019/10/AP-DT-English-Version-FINAL.pdf
400
+ Policy documents,Digital Economy Partnership Agreement,2020,New Zealand,https://www.mti.gov.sg/-/media/MTI/Microsites/DEAs/Digital-Economy-Partnership-Agreement/Digital-Economy-Partnership-Agreement.pdf
401
+ Policy documents,Concept for the Development of Artificial Intelligence in Bulgaria until 2030,2020,Bulgaria,https://www.mtitc.government.bg/sites/default/files/conceptforthedevelopmentofaiinbulgariauntil2030.pdf
402
+ Policy documents,The Dutch Digitalisation Strategy 2021,2021,Netherlands,https://www.nederlanddigitaal.nl/binaries/nederlanddigitaal-nl/documenten/publicaties/2021/06/22/the-dutch-digitalisation-strategy-2021-eng/210621-min-ezk-digitaliseringstrategie-en-v03.pdf
403
+ Policy documents,AIRAWAT- Establishing an AI Specific Cloud Computing Intrastructure for India,2020,India,https://www.niti.gov.in/sites/default/files/2020-01/AIRAWAT_Approach_Paper.pdf
404
+ Policy documents,American Artificial Intelligence Initiative: Year One Annual Report,2020,United States,https://www.nitrd.gov/nitrdgroups/images/c/c1/American-AI-Initiative-One-Year-Annual-Report.pdf
405
+ Policy documents,Recommendations for Leveraging Cloud Computing Resources for Federally Funded Artificial Intelligence Research and Development,2020,United States,https://www.nitrd.gov/pubs/Recommendations-Cloud-AI-RD-Nov2020.pdf
406
+ Policy documents,National Robotics Initiative 3.0: Innovations in Integration of Robotics (NRI-3.0),2021,United States,https://www.nsf.gov/pubs/2021/nsf21559/nsf21559.pdf
407
+ Policy documents,Proposte per una Strategia italiana per l'intelligenza artificiale,2020,Italy,https://www.mimit.gov.it/images/stories/documenti/Proposte_per_una_Strategia_italiana_AI.pdf
408
+ Policy documents,BRAZIL 2021 - National AI strategy,2021,Brazil,https://www.gov.br/mcti/pt-br/acompanhe-o-mcti/transformacaodigital/arquivosinteligenciaartificial/ebia-portaria_mcti_4-979_2021_anexo1.pdf
409
+ Policy documents,MEXICO Consolidado Comentarios Consulta,2019,Mexico,https://www.gob.mx/cms/uploads/attachment/file/415644/Consolidado_Comentarios_Consulta_IA__1_.pdf
410
+ Policy documents,Resolution Directing Use of Compulsory Process Regarding Abuse of Intellectual Property,2021,United States,https://www.ftc.gov/system/files/attachments/press-releases/ftc-streamlines-consumer-protection-competition-investigations-eight-key-enforcement-areas-enable/omnibus_resolutions_p859900.pdf
411
+ Policy documents,Robustness and Explainability of Artificial Intelligence,2020,European Union,https://publications.jrc.ec.europa.eu/repository/bitstream/JRC119336/dpad_report.pdf
412
+ Policy documents,"AI in the UK: ready, willing and able",2018,United Kingdom,https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf
413
+ Policy documents,AI in the UK: No Room for Complacency,2020,United Kingdom,https://publications.parliament.uk/pa/ld5801/ldselect/ldliaison/196/196.pdf
414
+ Policy documents,Developing Standards for Artificial Intelligence,2019,Australia,https://publicsectornetwork.com/wp-content/uploads/2019/11/Artificial-Intelligence-Discussion-Paper-004.pdf
415
+ Policy documents,Resource Guide on Artificial Ingelligence Strategies,2021,United Nations,https://sdgs.un.org/sites/default/files/2021-06/Resource%20Guide%20on%20AI%20Strategies_June%202021.pdf
416
+ Policy documents,Draft 2019-2020 Federal Data Strategy Action Plan,2019,United States,https://strategy.data.gov/assets/docs/draft-2019-2020-federal-data-strategy-action-plan.pdf
417
+ Policy documents,Science & Technology Highlights,2017,United States,https://trumpwhitehouse.archives.gov/wp-content/uploads/2018/03/Administration-2017-ST-Highlights.pdf
418
+ Policy documents,Towards a vibrant European Network of AI excellence centres,2020,European Union,https://umai.uma.es/shared/Commission-Presentation-H2020.pdf
419
+ Policy documents,Policy for the Development of Artificial Intelligence in Poland from 2020,2020,Poland,https://wp.oecd.ai/app/uploads/2021/12/Poland_Policy_for_Artificial_Intelligence_Development_in_Poland_from_2020_2020.pdf
420
+ Policy documents,National approach to artificial intelligence,2018,Sweden,https://wp.oecd.ai/app/uploads/2021/12/Sweden_National_Approach_to_Artificial_Intelligence_2018.pdf
421
+ Policy documents,Key points for a Federal Government Strategy on Artificial Intelligence,2018,Germany,https://www.bmwk.de/Redaktion/EN/Downloads/E/key-points-for-federal-government-strategy-on-artificial-intelligence.pdf?__blob=publicationFile&v=1
422
+ Policy documents,Strengthening international cooperation on AI,2021,United States,https://www.brookings.edu/wp-content/uploads/2021/10/Strengthening-International-Cooperation-AI_Oct21.pdf
423
+ Policy documents,Australia`s AI ethics framework- A discussion Paper,2019,Australia,https://www.csiro.au/-/media/D61/Reports/Artificial-Intelligence-ethics-framework.pdf
424
+ Policy documents,Algorithm Assessment Report,2018,New Zealand,https://www.data.govt.nz/assets/Uploads/Algorithm-Assessment-Report-Oct-2018.pdf
425
+ Policy documents,Recommendations when using supervised machine learning,2018,Denmark,https://www.dfsa.dk/-/media/Tilsyn/Recommendations_when_using_supervised_ML-pdf.pdf
426
+ Policy documents,Opportunities of Artificial Intelligence,2020,European Union,https://www.europarl.europa.eu/RegData/etudes/STUD/2020/652713/IPOL_STU(2020)652713_EN.pdf
427
+ Policy documents,Liability for Artificial Intelligence and other emerging digital technologies,2019,European Union,https://www.europarl.europa.eu/meetdocs/2014_2019/plmrep/COMMITTEES/JURI/DV/2020/01-09/AI-report_EN.pdf
428
+ Policy documents,Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan,2021,United States,https://www.fda.gov/media/145022/download
429
+ Policy documents,Deciphering China’s AI Dream,2018,China,https://www.fhi.ox.ac.uk/wp-content/uploads/Deciphering_Chinas_AI-Dream.pdf
430
+ Policy documents,"Statement on Artificial Intelligence, Robotics and Autonomous Systems",2018,European Union,https://op.europa.eu/en/publication-detail/-/publication/6b1bc507-af70-11e8-99ee-01aa75ed71a1/language-en/format-PDF
431
+ Policy documents,Social Principles of Human-centric AI (Draft),2019,Japan,https://www8.cao.go.jp/cstp/stmain/aisocialprinciples.pdf
requirements.txt ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ buster-doctalk==1.0.28
2
+ gradio==4.36.1
3
+ gradio_client==1.0.1
4
+ gunicorn==21.2.0
5
+ joblib>=1.3.2
6
+ pandas[output-formatting]>=2.1.3
7
+ pymongo>=4.6.1
src/app_utils.py ADDED
@@ -0,0 +1,191 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import logging
2
+ import os
3
+ import uuid
4
+ from datetime import datetime, timezone
5
+ from urllib.parse import quote_plus
6
+
7
+ import gradio as gr
8
+ import pandas as pd
9
+ import pymongo
10
+ from pymongo import MongoClient
11
+
12
+ from buster.completers import Completion, UserInputs
13
+ from buster.tokenizers import Tokenizer
14
+
15
+ logger = logging.getLogger(__name__)
16
+ logging.basicConfig(level=logging.INFO)
17
+
18
+
19
+ class WordTokenizer(Tokenizer):
20
+ """Naive word-level tokenizer
21
+
22
+ The original tokenizer from openAI eats way too much Ram.
23
+ This is a naive word count tokenizer to be used instead."""
24
+
25
+ def __init__(self, *args, **kwargs):
26
+ super().__init__(*args, **kwargs)
27
+
28
+ def encode(self, string):
29
+ return string.split()
30
+
31
+ def decode(self, encoded):
32
+ return " ".join(encoded)
33
+
34
+
35
+ def get_logging_db_name(instance_type: str) -> str:
36
+ assert instance_type in ["dev", "prod", "local", "test"], "Invalid instance_type declared."
37
+ return f"ai4h-databank-{instance_type}"
38
+
39
+
40
+ def get_session_id() -> str:
41
+ """Generate a uuid for each user."""
42
+ return str(uuid.uuid1())
43
+
44
+
45
+ def verify_required_env_vars(required_vars: list[str]):
46
+ unset_vars = [var for var in required_vars if os.getenv(var) is None]
47
+ if len(unset_vars) > 0:
48
+ logger.warning(f"Lisf of env. variables that weren't set: {unset_vars}")
49
+ else:
50
+ logger.info("All environment variables are set appropriately.")
51
+
52
+
53
+ def make_uri(username: str, password: str, cluster: str) -> str:
54
+ """Create mongodb uri."""
55
+ uri = (
56
+ "mongodb+srv://"
57
+ + quote_plus(username)
58
+ + ":"
59
+ + quote_plus(password)
60
+ + "@"
61
+ + cluster
62
+ + "/?retryWrites=true&w=majority"
63
+ )
64
+ return uri
65
+
66
+
67
+ def init_db(mongo_uri: str, db_name: str) -> pymongo.database.Database:
68
+ """
69
+ Initialize and return a connection to the specified MongoDB database.
70
+
71
+ Parameters:
72
+ - mongo_uri (str): The connection string for the MongoDB. This can be formed using `make_uri` function.
73
+ - db_name (str): The name of the MongoDB database to connect to.
74
+
75
+ Returns:
76
+ pymongo.database.Database: The connected database object.
77
+
78
+ Note:
79
+ If there's a problem with the connection, an exception will be logged and the program will terminate.
80
+ """
81
+
82
+ try:
83
+ mongodb_client = MongoClient(mongo_uri)
84
+ # Ping the database to make sure authentication is good
85
+ mongodb_client.admin.command("ping")
86
+ database = mongodb_client[db_name]
87
+ logger.info("Succesfully connected to the MongoDB database")
88
+ return database
89
+ except Exception as e:
90
+ logger.exception("Something went wrong connecting to mongodb")
91
+
92
+
93
+ def get_utc_time() -> str:
94
+ return str(datetime.now(timezone.utc))
95
+
96
+
97
+ def check_auth(username: str, password: str) -> bool:
98
+ """Check if authentication succeeds or not.
99
+
100
+ The authentication leverages the built-in gradio authentication. We use a shared password among users.
101
+ It is temporary for developing the PoC. Proper authentication needs to be implemented in the future.
102
+ We allow a valid username to be any username beginning with 'databank-', this will allow us to differentiate between users easily.
103
+ """
104
+
105
+ # get auth information from env. vars, they need to be set
106
+ USERNAME = os.environ["AI4H_APP_USERNAME"]
107
+ PASSWORD = os.environ["AI4H_APP_PASSWORD"]
108
+
109
+ valid_user = username.startswith(USERNAME)
110
+ valid_password = password == PASSWORD
111
+ is_auth = valid_user and valid_password
112
+ logger.info(f"Log-in attempted by {username=}. {is_auth=}")
113
+ return is_auth
114
+
115
+
116
+ def format_sources(matched_documents: pd.DataFrame) -> list[str]:
117
+ formatted_sources = []
118
+
119
+ # We first group on Title of the document, so that 2 chunks from a same doc get lumped together
120
+ grouped_df = matched_documents.groupby("title")
121
+
122
+ # Here we just rank the titles by highest to lowest similarity score...
123
+ ranked_titles = (
124
+ grouped_df.apply(lambda x: x.similarity_to_answer.max()).sort_values(ascending=False).index.to_list()
125
+ )
126
+
127
+ for title in ranked_titles:
128
+ df = grouped_df.get_group(title)
129
+
130
+ # Adds a link break between sources from a same chunk
131
+ chunks = "<br><br>".join(["πŸ”— " + chunk for chunk in df.content.to_list()])
132
+
133
+ url = df.url.to_list()[0]
134
+ source = df.source.to_list()[0]
135
+ year = df.year.to_list()[0]
136
+ country = df.country.to_list()[0]
137
+
138
+ formatted_sources.append(
139
+ f"""
140
+
141
+ ### Publication: [{title}]({url})
142
+ **Year of publication:** {year}
143
+ **Source:** {source}
144
+ **Country:** {country}
145
+
146
+ **Identified sections**:
147
+ {chunks}
148
+ """
149
+ )
150
+
151
+ return formatted_sources
152
+
153
+
154
+ def pad_sources(sources: list[str], max_sources: int) -> list[str]:
155
+ """Pad sources with empty strings to ensure that the number of sources is always max_sources."""
156
+ k = len(sources)
157
+ return sources + [""] * (max_sources - k)
158
+
159
+
160
+ def add_sources(completion, max_sources: int):
161
+ if not completion.question_relevant:
162
+ # Question was not relevant, don't bother doing anything else...
163
+ formatted_sources = [""]
164
+ else:
165
+ formatted_sources = format_sources(completion.matched_documents)
166
+
167
+ formatted_sources = pad_sources(formatted_sources, max_sources)
168
+
169
+ sources_textboxes = []
170
+ for source in formatted_sources:
171
+ visible = False if source == "" else True
172
+ t = gr.Markdown(source, latex_delimiters=[], elem_classes="source", visible=visible)
173
+ sources_textboxes.append(t)
174
+ return sources_textboxes
175
+
176
+
177
+ def debug_completion(user_input, reformulate_question):
178
+ """Generate a debug completion."""
179
+ user_inputs = UserInputs(original_input=user_input)
180
+ if reformulate_question:
181
+ user_inputs.reformulated_input = "This is your reformulated question?"
182
+
183
+ completion = Completion(
184
+ user_inputs=user_inputs,
185
+ error=False,
186
+ matched_documents=[],
187
+ answer_generator="This is the answer you'd expect a User to see.",
188
+ question_relevant=True,
189
+ answer_relevant=True,
190
+ )
191
+ return completion
src/buster/assets/index.html ADDED
@@ -0,0 +1 @@
 
 
1
+ <html><head><meta content="text/html; charset=UTF-8" http-equiv="content-type"><style type="text/css">@import url(https://themes.googleusercontent.com/fonts/css?kit=wAPX1HepqA24RkYW1AuHYA);ol.lst-kix_list_1-3{list-style-type:none}ol.lst-kix_list_1-4{list-style-type:none}.lst-kix_list_2-6>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) "." counter(lst-ctn-kix_list_2-3,decimal) "." counter(lst-ctn-kix_list_2-4,decimal) "." counter(lst-ctn-kix_list_2-5,decimal) "." counter(lst-ctn-kix_list_2-6,decimal) ". "}.lst-kix_list_2-7>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) "." counter(lst-ctn-kix_list_2-3,decimal) "." counter(lst-ctn-kix_list_2-4,decimal) "." counter(lst-ctn-kix_list_2-5,decimal) "." counter(lst-ctn-kix_list_2-6,decimal) "." counter(lst-ctn-kix_list_2-7,decimal) ". "}.lst-kix_list_2-7>li{counter-increment:lst-ctn-kix_list_2-7}ol.lst-kix_list_1-5{list-style-type:none}ol.lst-kix_list_1-6{list-style-type:none}.lst-kix_list_2-1>li{counter-increment:lst-ctn-kix_list_2-1}ol.lst-kix_list_1-0{list-style-type:none}.lst-kix_list_2-4>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) "." counter(lst-ctn-kix_list_2-3,decimal) "." counter(lst-ctn-kix_list_2-4,decimal) ". "}.lst-kix_list_2-5>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) "." counter(lst-ctn-kix_list_2-3,decimal) "." counter(lst-ctn-kix_list_2-4,decimal) "." counter(lst-ctn-kix_list_2-5,decimal) ". "}.lst-kix_list_2-8>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) "." counter(lst-ctn-kix_list_2-3,decimal) "." counter(lst-ctn-kix_list_2-4,decimal) "." counter(lst-ctn-kix_list_2-5,decimal) "." counter(lst-ctn-kix_list_2-6,decimal) "." counter(lst-ctn-kix_list_2-7,decimal) "." counter(lst-ctn-kix_list_2-8,decimal) ". "}ol.lst-kix_list_1-1{list-style-type:none}ol.lst-kix_list_1-2{list-style-type:none}.lst-kix_list_1-1>li{counter-increment:lst-ctn-kix_list_1-1}ol.lst-kix_list_2-6.start{counter-reset:lst-ctn-kix_list_2-6 0}ol.lst-kix_list_1-8.start{counter-reset:lst-ctn-kix_list_1-8 0}ol.lst-kix_list_2-3.start{counter-reset:lst-ctn-kix_list_2-3 0}ol.lst-kix_list_1-5.start{counter-reset:lst-ctn-kix_list_1-5 0}ol.lst-kix_list_1-7{list-style-type:none}.lst-kix_list_1-7>li{counter-increment:lst-ctn-kix_list_1-7}ol.lst-kix_list_1-8{list-style-type:none}ol.lst-kix_list_2-5.start{counter-reset:lst-ctn-kix_list_2-5 0}.lst-kix_list_2-0>li{counter-increment:lst-ctn-kix_list_2-0}.lst-kix_list_2-3>li{counter-increment:lst-ctn-kix_list_2-3}.lst-kix_list_2-6>li{counter-increment:lst-ctn-kix_list_2-6}ol.lst-kix_list_1-7.start{counter-reset:lst-ctn-kix_list_1-7 0}.lst-kix_list_1-2>li{counter-increment:lst-ctn-kix_list_1-2}ol.lst-kix_list_2-2.start{counter-reset:lst-ctn-kix_list_2-2 0}.lst-kix_list_1-5>li{counter-increment:lst-ctn-kix_list_1-5}.lst-kix_list_1-8>li{counter-increment:lst-ctn-kix_list_1-8}ol.lst-kix_list_1-4.start{counter-reset:lst-ctn-kix_list_1-4 0}ol.lst-kix_list_1-1.start{counter-reset:lst-ctn-kix_list_1-1 0}ol.lst-kix_list_2-2{list-style-type:none}ol.lst-kix_list_2-3{list-style-type:none}ol.lst-kix_list_2-4{list-style-type:none}ol.lst-kix_list_2-5{list-style-type:none}.lst-kix_list_1-4>li{counter-increment:lst-ctn-kix_list_1-4}ol.lst-kix_list_2-0{list-style-type:none}.lst-kix_list_2-4>li{counter-increment:lst-ctn-kix_list_2-4}ol.lst-kix_list_1-6.start{counter-reset:lst-ctn-kix_list_1-6 0}ol.lst-kix_list_2-1{list-style-type:none}ol.lst-kix_list_1-3.start{counter-reset:lst-ctn-kix_list_1-3 0}ol.lst-kix_list_2-8.start{counter-reset:lst-ctn-kix_list_2-8 0}ol.lst-kix_list_1-2.start{counter-reset:lst-ctn-kix_list_1-2 0}.lst-kix_list_1-0>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) ". "}ol.lst-kix_list_2-6{list-style-type:none}.lst-kix_list_1-1>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) ". "}.lst-kix_list_1-2>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) ". "}ol.lst-kix_list_2-0.start{counter-reset:lst-ctn-kix_list_2-0 0}ol.lst-kix_list_2-7{list-style-type:none}ol.lst-kix_list_2-8{list-style-type:none}.lst-kix_list_1-3>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) "." counter(lst-ctn-kix_list_1-3,decimal) ". "}.lst-kix_list_1-4>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) "." counter(lst-ctn-kix_list_1-3,decimal) "." counter(lst-ctn-kix_list_1-4,lower-latin) ") "}ol.lst-kix_list_1-0.start{counter-reset:lst-ctn-kix_list_1-0 0}.lst-kix_list_1-0>li{counter-increment:lst-ctn-kix_list_1-0}.lst-kix_list_1-6>li{counter-increment:lst-ctn-kix_list_1-6}.lst-kix_list_1-7>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) "." counter(lst-ctn-kix_list_1-3,decimal) "." counter(lst-ctn-kix_list_1-4,lower-latin) ")" counter(lst-ctn-kix_list_1-5,lower-roman) ")" counter(lst-ctn-kix_list_1-6,decimal) ")" counter(lst-ctn-kix_list_1-7,upper-latin) ") "}ol.lst-kix_list_2-7.start{counter-reset:lst-ctn-kix_list_2-7 0}.lst-kix_list_1-3>li{counter-increment:lst-ctn-kix_list_1-3}.lst-kix_list_1-5>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) "." counter(lst-ctn-kix_list_1-3,decimal) "." counter(lst-ctn-kix_list_1-4,lower-latin) ")" counter(lst-ctn-kix_list_1-5,lower-roman) ") "}.lst-kix_list_1-6>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) "." counter(lst-ctn-kix_list_1-3,decimal) "." counter(lst-ctn-kix_list_1-4,lower-latin) ")" counter(lst-ctn-kix_list_1-5,lower-roman) ")" counter(lst-ctn-kix_list_1-6,decimal) ") "}li.li-bullet-0:before{margin-left:-18pt;white-space:nowrap;display:inline-block;min-width:18pt}.lst-kix_list_2-0>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) ". "}.lst-kix_list_2-1>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) ". "}ol.lst-kix_list_2-1.start{counter-reset:lst-ctn-kix_list_2-1 0}.lst-kix_list_2-5>li{counter-increment:lst-ctn-kix_list_2-5}.lst-kix_list_2-8>li{counter-increment:lst-ctn-kix_list_2-8}.lst-kix_list_1-8>li:before{content:"" counter(lst-ctn-kix_list_1-0,decimal) "." counter(lst-ctn-kix_list_1-1,decimal) "." counter(lst-ctn-kix_list_1-2,decimal) "." counter(lst-ctn-kix_list_1-3,decimal) "." counter(lst-ctn-kix_list_1-4,lower-latin) ")" counter(lst-ctn-kix_list_1-5,lower-roman) ")" counter(lst-ctn-kix_list_1-6,decimal) ")" counter(lst-ctn-kix_list_1-7,upper-latin) ")" counter(lst-ctn-kix_list_1-8,upper-roman) ") "}.lst-kix_list_2-2>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) ". "}.lst-kix_list_2-3>li:before{content:"" counter(lst-ctn-kix_list_2-0,decimal) "." counter(lst-ctn-kix_list_2-1,decimal) "." counter(lst-ctn-kix_list_2-2,decimal) "." counter(lst-ctn-kix_list_2-3,decimal) ". "}.lst-kix_list_2-2>li{counter-increment:lst-ctn-kix_list_2-2}ol.lst-kix_list_2-4.start{counter-reset:lst-ctn-kix_list_2-4 0}ol{margin:0;padding:0}table td,table th{padding:0}.c14{margin-left:17.9pt;padding-top:0pt;text-indent:-17.9pt;padding-bottom:8pt;line-height:1.0791666666666666;orphans:2;widows:2;text-align:left;height:10pt}.c7{margin-left:18pt;padding-top:2pt;padding-left:0pt;padding-bottom:6pt;line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}.c1{margin-left:35.5pt;padding-top:0pt;text-indent:-35.5pt;padding-bottom:8pt;line-height:1.0791666666666666;orphans:2;widows:2;text-align:justify}.c3{margin-left:17.9pt;padding-top:0pt;padding-bottom:0pt;line-height:1.0;orphans:2;widows:2;text-align:left;height:10pt}.c15{margin-left:17.9pt;padding-top:12pt;padding-bottom:0pt;line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}.c10{margin-left:17.9pt;padding-top:0pt;padding-bottom:8pt;line-height:1.0791666666666666;orphans:2;widows:2;text-align:justify}.c8{font-weight:400;text-decoration:none;vertical-align:baseline;font-size:10pt;font-family:"Times New Roman";font-style:normal}.c11{font-weight:400;text-decoration:none;vertical-align:baseline;font-size:10.5pt;font-family:"Times New Roman";font-style:normal}.c12{text-decoration:none;vertical-align:baseline;font-size:16pt;font-family:"Times New Roman";font-style:normal}.c5{text-decoration:none;vertical-align:baseline;font-size:12pt;font-family:"Times New Roman";font-style:normal}.c6{text-decoration-skip-ink:none;-webkit-text-decoration-skip:none;text-decoration:underline}.c13{background-color:#ffffff;max-width:468pt;padding:72pt 72pt 72pt 72pt}.c0{padding:0;margin:0}.c4{font-weight:700}.c9{font-style:italic}.c2{color:#000000}.title{padding-top:24pt;color:#000000;font-weight:700;font-size:36pt;padding-bottom:6pt;font-family:"Times New Roman";line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}.subtitle{padding-top:18pt;color:#666666;font-size:24pt;padding-bottom:4pt;font-family:"Georgia";line-height:1.0791666666666666;page-break-after:avoid;font-style:italic;orphans:2;widows:2;text-align:left}li{color:#000000;font-size:10pt;font-family:"Times New Roman"}p{margin:0;color:#000000;font-size:10pt;font-family:"Times New Roman"}h1{padding-top:12pt;color:#2f5496;font-size:16pt;padding-bottom:0pt;font-family:"Calibri";line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}h2{padding-top:2pt;color:#000000;font-weight:700;font-size:12pt;padding-bottom:6pt;font-family:"Times New Roman";line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}h3{padding-top:2pt;color:#000000;font-weight:700;font-size:10pt;padding-bottom:6pt;font-family:"Times New Roman";line-height:1.0791666666666666;orphans:2;widows:2;text-align:justify}h4{padding-top:12pt;color:#000000;font-weight:700;font-size:12pt;padding-bottom:2pt;font-family:"Times New Roman";line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}h5{padding-top:11pt;color:#000000;font-weight:700;font-size:11pt;padding-bottom:2pt;font-family:"Times New Roman";line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}h6{padding-top:10pt;color:#000000;font-weight:700;font-size:10pt;padding-bottom:2pt;font-family:"Times New Roman";line-height:1.0791666666666666;page-break-after:avoid;orphans:2;widows:2;text-align:left}</style></head><body class="c13 doc-content"><div><p class="c3"><span class="c8 c2"></span></p></div><h1 class="c15"><span class="c4 c2 c12">Terms and Conditions (October 13, 2013)</span></h1><p class="c14"><span class="c8 c2"></span></p><p class="c10"><span>These terms and conditions (&ldquo;</span><span class="c4">Terms</span><span>&rdquo;) apply between Mila &ndash; Quebec Artificial Intelligence Institute (&ldquo;</span><span class="c4">Mila</span><span>&rdquo;, &ldquo;</span><span class="c4">we</span><span>&rdquo;, &ldquo;</span><span class="c4">us</span><span>&rdquo; or &ldquo;</span><span class="c4">our</span><span>&rdquo;) and you when you are accessing or using any artificial intelligence (AI)-powered service(s), feature(s), or functionality made available through the beta version of Search Engine for AI Policy platform (&ldquo;</span><span class="c4">AIR</span><span class="c8 c2">&rdquo;).</span></p><ol class="c0 lst-kix_list_2-0 start" start="1"><li class="c7 li-bullet-0"><h2 style="display:inline"><span class="c5 c4 c2">Your Use of AIR</span></h2></li></ol><p class="c10"><span>AIR is made available to you as a courtesy by Mila, subject to your compliance with these Terms, free of charge. You may not use AIR (i) to mislead any person to believe that the Output (as defined below) from AIR was solely human generated; (ii) in a way that would negatively affect the use of AIR by other users or its operation by Mila; (iii) to infringe on any third party&rsquo;s rights; or (iv) in a manner that would otherwise reasonably constitute misuse or abuse. You also shall not use AIR or its Output in a manner that creates harm or biased results. Biased results include using the Output, or a decision, recommendation or prediction that is made by AIR in a way that adversely differentiates, directly or indirectly and without justification, in relation to an individual on one or more of the prohibited grounds of discrimination set out in Section 3 of the </span><span class="c9">Canadian Human Rights Act</span><span>&nbsp;(&ldquo;</span><span class="c4">CHRA</span><span class="c8 c2">&rdquo;), or on a combination of such prohibited grounds.</span></p><ol class="c0 lst-kix_list_2-0" start="2"><li class="c7 li-bullet-0"><h2 id="h.gjdgxs" style="display:inline"><span class="c5 c4 c2">Intended Purpose and Accuracy</span></h2></li></ol><p class="c10"><span class="c9">Output of this tool may not be accurate. </span><span>AIR is an experimental AI tool that uses generative artificial intelligence technologies to generate an output (&ldquo;</span><span class="c4">Output</span><span>&rdquo;) in relation to and based on any content you may provide to it (&ldquo;</span><span class="c4">Input</span><span>&rdquo;), in order to answer queries related to artificial intelligence legislation and governance materials (&ldquo;</span><span class="c4">Intended Purpose</span><span>&rdquo;). AIR answers these questions by using content retrieved from third-party sources (&ldquo;</span><span class="c4">Third-Party Sources</span><span>&rdquo;), the accuracy of which is not endorsed nor confirmed by Mila. AIR is not intended to provide information outside of its Intended Purpose, and is not intended to be used to provide legal advice. In any case, even when using AIR for its Intended Purpose, you understand that AIR relies on experimental and exploratory technologies and that Mila cannot attest that the Output shall be accurate, complete or valid, and may as such not meet your desired use including, without limitation, that it may be duplicative of content generated by AIR for other users or third-parties. </span><span class="c6 c4">You must use caution and discretion when relying on, publishing, distributing, or otherwise using any Output resulting from the use of AIR and a human should always review the Output before any subsequent use of the Output is contemplated, including compliance with legislation, regulations or any other legal duties</span><span class="c8 c2">. Furthermore, you understand that Mila does not endorse the Output and you agree to not construe the Output as an official statement by Mila. You understand that some Third-Party Sources may be outdated at the time the Output is generated or later become outdated and, as such, the Output may be outdated or become so over time.</span></p><ol class="c0 lst-kix_list_2-0" start="3"><li class="c7 li-bullet-0"><h2 style="display:inline"><span class="c5 c4 c2">Intellectual Property</span></h2></li></ol><p class="c10"><span class="c9">You remain the owner of your Input and you become the owner of the Output</span><span>. You grant us a non-exclusive, irrevocable, worldwide, royalty-free, sublicensable licence to the Input and to the Output, including without limitation to provide you access to AIR, to improve AIR or to use in relation with other services developed by Mila, its affiliates or partners (&ldquo;</span><span class="c4">Mila Services</span><span class="c8 c2">&rdquo;), which includes the right to use, process, copy, reproduce, modify, create derivative works of, excerpt, translate, publish, display, transmit, perform, and distribute the Input and Output as necessary to operate, enhance, improve, and further develop AIR and to provide the Mila Services. To the extent it is permitted under law, and subject to third-party rights and compliance with your obligations under the Terms, we assign to you all rights, title and interest in and to the Output that may be held by Mila and waive, on behalf of any individuals associated thereto, all moral rights existing in such works.</span></p><p class="c10"><span class="c8 c2">You are solely responsible for any Input entered into AIR, including the accuracy, quality, appropriateness, and legality thereof, and will ensure that such Input and use thereof in AIR does not (i) violate any applicable law; (ii) contain any Personal Information (as defined below), as further described in Section 4 (Privacy); (iii) violate these Terms; or (iv) infringe, violate, or misappropriate the rights of Mila or of any third party. You represent and warrant that you possess all rights necessary to provide the Input into AIR and to grant the licenses to Mila set forth herein.</span></p><p class="c10"><span class="c8 c2">Subject to the foregoing and to third-party rights, either Mila or its licensors remain the owners of any intellectual property rights included in or embedded in AIR or its underlying technologies, including all branding elements and trademarks pertaining thereto. Despite anything to the contrary in these Terms, no rights are granted to you in regard to the Third-Party Sources.</span></p><p class="c10"><span>If you provide to us feedback, comments or ideas with respect to AIR or any Mila Services (&ldquo;</span><span class="c4">Feedback</span><span class="c8 c2">&rdquo;), you grant us a perpetual, non-exclusive, irrevocable, worldwide, royalty-free, sublicensable and fully transferable licence to this Feedback for any purpose without further consideration owed to you.</span></p><ol class="c0 lst-kix_list_2-0" start="4"><li class="c7 li-bullet-0"><h2 style="display:inline"><span class="c5 c4 c2">Privacy</span></h2></li></ol><p class="c10"><span class="c9">No Personal Information. </span><span>AIR is not intended to process personal information or any information that would be subject to applicable privacy laws (&ldquo;</span><span class="c4">Personal Information</span><span class="c8 c2">&rdquo;). Therefore, you represent that your Input does not contain Personal Information, including names, addresses, phone numbers, email addresses, or birth dates. If, despite the foregoing, your Input does contain Personal Information, you represent that you have obtained in writing the necessary consents, authorizations and licences to allow us to process this Personal Information through AIR in compliance with the Terms.</span></p><p class="c10"><span>Any collection, use and sharing of Personal Information by Mila on you and your use of AIR will be subject to, and undertaken in compliance with, the terms of our Privacy Policy, available on our website: </span><span class="c6">https://mila.quebec/en/resource/privacy-policy/</span><span class="c8 c2">.</span></p><ol class="c0 lst-kix_list_2-0" start="5"><li class="c7 li-bullet-0"><h2 style="display:inline"><span class="c5 c4 c2">Limitation of Liability</span></h2></li></ol><p class="c10"><span class="c8 c2">To the maximum extent permitted by applicable law, and except for its gross fault, willful misconduct or breach of an essential obligation of the agreement, neither Mila nor its respective directors, officers, researchers, employees, agents, or service providers, shall be liable under these Terms and under any legal theory for any direct or indirect damages, including without limitation lost revenues, profits or loss of clients or expected clients, damages for business interruption or failure to realize expected savings, loss of use, loss of goodwill, loss of data and any other incidental commercial or financial losses of any kind, whether caused by tort (including negligence), breach of contract or otherwise, even if Mila was allegedly advised or had reason to know, but excluding claims based on fraud. Without limiting the generality of the foregoing, Mila specifically disclaims any liability for any business, legal or compliance decision, act or omission made by you in reliance of the Output.</span></p><p class="c10"><span class="c8 c2">To the full extent permitted by the law, and subject to the paragraph above, the aggregate liability of Mila, its employees, directors, officers, affiliates, researchers and partners under these Terms shall be limited to 50 CAD. You understand that AIR is an experimental and preliminary tool and is provided to you &ldquo;as-is&rdquo; without any warranties (express, implied or otherwise) and we disclaim all warranties including without limitation merchantability, fitness for a particular purpose (even for the Intended Purpose), satisfactory quality and non-infringement. AIR may be interrupted without prior notification and data (including the Input) may be lost or altered. Mila disclaims all warranties and representations (express, implied or otherwise) in regard to the security or confidentiality of any content, including without limitation content provided to, licensed to, or licensed by Mila under these Terms.</span></p><ol class="c0 lst-kix_list_2-0" start="6"><li class="c7 li-bullet-0"><h2 style="display:inline"><span class="c5 c4 c2">Termination</span></h2></li></ol><p class="c10"><span class="c8 c2">You may terminate these Terms at any time by ceasing to use AIR. We may terminate these Terms without prior notice at our discretion. Section 2 (Intended Purpose and Accuracy) and Section 5 (Limitation of Liability) shall survive the termination of these Terms.</span></p><ol class="c0 lst-kix_list_2-0" start="7"><li class="c7 li-bullet-0"><h2 style="display:inline"><span class="c4 c2 c5">General</span></h2></li></ol><p class="c1"><span class="c4">7.1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Severability and Waiver</span><span class="c8 c2">. If any portion of these Terms is determined by a court of competent jurisdiction to be invalid or unenforceable, then any such invalid or unenforceable portion shall be deemed deleted from these Terms; provided that any such invalidity or unenforceability shall not affect the remaining portions of these Terms, all of which shall remain in full force and effect. Failure or delay by Mila to exercise any right under these Terms does not constitute a waiver of these Terms. All waivers must be in writing and signed by Mila.</span></p><p class="c1"><span class="c4">7.2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Modifications</span><span>. These Terms may be amended from time to time. Such amended version will be published on the website hosting AIR </span><span class="c2">and</span><span>&nbsp;will be effective immediately. Your subsequent use of AIR means that you agree to these changes.</span></p><p class="c1"><span class="c4">7.3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Independent Parties</span><span>. Mila and you are independent parties and nothing in these Terms shall constitute any party as the employer, principal or partner of, or joint venture with, the other party. No party has the authority to assume or create any obligation or liability, either expressed or implied, on behalf of any other party.</span></p><p class="c1"><span class="c4">7.4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Use of Brand</span><span class="c2">. Nothing in these Terms grant you any rights to use the brand (including trademarks and brand names) of Mila or any of its </span><span>partners</span><span class="c8 c2">.</span></p><p class="c1"><span class="c4">7.5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Entire Agreement</span><span class="c2">. These Terms </span><span>constitute</span><span class="c8 c2">&nbsp;the entire agreement between Mila and you with respect to the subject matter hereof and supersedes all other prior written or oral agreements hereto with respect to the subject matter hereof.</span></p><p class="c1"><span class="c4">7.6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Export Control</span><span>. The AIR platform, including any software, documentation, and any related technical data included with, or contained in, such platform, and any products utilizing any </span><span class="c4">such software</span><span>, documentation, or technical data (collectively, &ldquo;</span><span class="c4">Regulated Products</span><span>&rdquo;) may be subject to US export control laws and regulations, including the </span><span class="c9">Export Administration Regulations</span><span>&nbsp;and the </span><span class="c9">International Traffic in Arms Regulations</span><span class="c8 c2">. You shall not, and shall not permit any third parties to, directly or indirectly, export, reexport, or release any Regulated Products to any jurisdiction or country to which, or any party to whom, or for any use for which, the export, reexport, or release of any Regulated Products is prohibited by applicable federal or foreign law, regulation, or rule. You shall comply with all applicable federal and foreign laws, regulations, and rules, and complete all required undertakings (including obtaining any necessary export license or other governmental approval), prior to exporting, reexporting, or releasing any Regulated Products. You shall provide prior written notice of the need to comply with such laws, regulations, and rules to any person, firm, or entity which you have reason to believe is obtaining any such Regulated Products from you with the intent to export or reexport.</span></p><p class="c1"><span class="c4">7.7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Governing Law</span><span class="c2">. These Terms shall be construed in accordance with the laws of the Province of Quebec and the federal laws of Canada applicable </span><span>therein</span><span class="c8 c2">. Any dispute arising from or related to these Terms shall be submitted to a court of competent jurisdiction in the judicial district of Montreal, Quebec.</span></p><div><p class="c3"><span class="c2 c8"></span></p></div></body></html>
src/buster/buster_app.py ADDED
@@ -0,0 +1,668 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import logging
3
+ from typing import Optional, Union
4
+
5
+ import gradio as gr
6
+ import pandas as pd
7
+
8
+ import src.cfg as cfg
9
+ from buster.completers import Completion
10
+ from src.app_utils import add_sources, get_session_id, get_utc_time
11
+ from src.cfg import setup_buster
12
+ from src.feedback import FeedbackForm, Interaction
13
+
14
+ logger = logging.getLogger(__name__)
15
+ logging.basicConfig(level=logging.INFO)
16
+
17
+ # Typehint for chatbot history
18
+ ChatHistory = list[list[Optional[str], Optional[str]]]
19
+
20
+ app_name = cfg.app_name
21
+ example_questions = cfg.example_questions
22
+ disclaimer = cfg.disclaimer
23
+ mongo_db = cfg.mongo_db
24
+ buster_cfg = copy.deepcopy(cfg.buster_cfg)
25
+ buster = setup_buster(buster_cfg=buster_cfg)
26
+ max_sources = cfg.max_sources
27
+ data_dir = cfg.data_dir
28
+
29
+
30
+ # link to the terms and conditions to be rendered in markdown blocks
31
+ path_to_tncs = "file=src/buster/assets/index.html"
32
+ md_link_to_tncs = f"[terms and conditions]({path_to_tncs})"
33
+
34
+ # get documents metadata
35
+ documents_metadata_file = str(data_dir / "documents_metadata.csv")
36
+ documents_metadata = pd.read_csv(documents_metadata_file)
37
+
38
+ css = """
39
+ .source {
40
+ max-height: 250px; /* Set the maximum height for the textboxes */
41
+ overflow: auto; /* Enable scrollbars when content exceeds dimensions */
42
+ outline: 1px solid gray; /* Add a gray outline */
43
+ border-radius: 5px; /* Add rounded corners to the outline */
44
+ }
45
+ """
46
+
47
+
48
+ def add_disclaimer(completion: Completion, chat_history: ChatHistory, disclaimer: str = disclaimer):
49
+ """Add a disclaimer response if the answer was relevant."""
50
+ if completion.question_relevant:
51
+ chat_history.append([None, disclaimer])
52
+ return chat_history
53
+
54
+
55
+ def hide_about_panel(accept_checkbox):
56
+ # Stay open while not accepted
57
+ open = not bool(accept_checkbox)
58
+ return {about_panel: gr.update(open=open)}
59
+
60
+
61
+ def set_relevant_sources_selection(num_sources: int):
62
+ relevant_sources_selection = gr.CheckboxGroup(
63
+ choices=[f"Source {i+1}" for i in range(num_sources)],
64
+ label="Check all relevant sources (if any)",
65
+ )
66
+ return relevant_sources_selection
67
+
68
+
69
+ def setup_feedback_form(num_sources: int):
70
+ # Feedback
71
+ feedback_elems = {}
72
+ with gr.Row():
73
+ with gr.Column():
74
+ gr.Markdown(
75
+ f""" ## We would love your feedback!
76
+ Please submit feedback for each question asked.
77
+
78
+ Your feedback is anonymous and will help us make the tool as useful as possible for the community!
79
+ """
80
+ )
81
+ with gr.Row():
82
+ overall_experience = gr.Radio(choices=["πŸ‘", "πŸ‘Ž"], label=f"Did {app_name} help answer your question?")
83
+
84
+ # Currently, we show all feedback, but also support having a small portion of it display at first
85
+ show_additional_feedback = gr.Group(visible=True)
86
+ with show_additional_feedback:
87
+ with gr.Column():
88
+ clear_answer = gr.Radio(
89
+ choices=["πŸ‘", "πŸ‘Ž"], label="Was the generated answer clear and understandable?"
90
+ )
91
+ accurate_answer = gr.Radio(choices=["πŸ‘", "πŸ‘Ž"], label="Was the generated answer accurate?")
92
+ relevant_sources = gr.Radio(
93
+ choices=["πŸ‘", "πŸ‘Ž"],
94
+ label="Were the retrieved sources generally relevant to your query?",
95
+ )
96
+ relevant_sources_selection = set_relevant_sources_selection(num_sources=num_sources)
97
+ relevant_sources_order = gr.Radio(
98
+ choices=["πŸ‘", "πŸ‘Ž"],
99
+ label="Were the sources ranked appropriately, in order of relevance?",
100
+ )
101
+
102
+ extra_info = gr.Textbox(
103
+ label="Any other comments?",
104
+ lines=3,
105
+ placeholder="Please enter other feedback for improvement here...",
106
+ )
107
+
108
+ expertise = gr.Radio(
109
+ choices=["Beginner", "Intermediate", "Expert"],
110
+ label="How would you rate your knowledge of AI policy",
111
+ interactive=True,
112
+ )
113
+
114
+ submit_feedback_btn = gr.Button("Submit feedback", variant="primary", interactive=True)
115
+ with gr.Column(visible=False) as submitted_message:
116
+ gr.Markdown("Feedback recorded, thank you πŸ“! You can now ask a new question in the search bar.")
117
+
118
+ # fmt: off
119
+ submit_feedback_btn.click(
120
+ toggle_visibility,
121
+ inputs=gr.State(False),
122
+ outputs=submitted_message,
123
+ ).then(
124
+ submit_feedback,
125
+ inputs=[
126
+ overall_experience,
127
+ clear_answer,
128
+ accurate_answer,
129
+ relevant_sources,
130
+ relevant_sources_order,
131
+ relevant_sources_selection,
132
+ expertise,
133
+ extra_info,
134
+ last_completion,
135
+ session_id,
136
+ ],
137
+ ).success(
138
+ toggle_visibility,
139
+ inputs=gr.State(True),
140
+ outputs=submitted_message,
141
+ ).success(
142
+ toggle_interactivity,
143
+ inputs=gr.State(False),
144
+ outputs=submit_feedback_btn,
145
+ )
146
+
147
+ # fmt: on
148
+ feedback_elems = {
149
+ "overall_experience": overall_experience,
150
+ "clear_answer": clear_answer,
151
+ "accurate_answer": accurate_answer,
152
+ "relevant_sources": relevant_sources,
153
+ "relevant_sources_selection": relevant_sources_selection,
154
+ "relevant_sources_order": relevant_sources_order,
155
+ "submit_feedback_btn": submit_feedback_btn,
156
+ "submitted_message": submitted_message,
157
+ "show_additional_feedback": show_additional_feedback,
158
+ "expertise": expertise,
159
+ "extra_info": extra_info,
160
+ }
161
+
162
+ return feedback_elems
163
+
164
+
165
+ def to_md_link(title: str, link: str) -> str:
166
+ """Converts a title and link to markown link format"""
167
+ return f"[{title}]({link})"
168
+
169
+
170
+ def get_metadata_markdown(df) -> str:
171
+ """Converts the content from a dataframe to a markdown table string format."""
172
+ metadata = []
173
+
174
+ # Order articles by year, with latest first
175
+ df = df.sort_values(["Country", "Year"], ascending=True)
176
+
177
+ for _, item in df.iterrows():
178
+ # source = item["Source"]
179
+ link = item["Link"]
180
+ title = item["Title"]
181
+ year = item["Year"]
182
+ country = item["Country"]
183
+
184
+ metadata.append(f"{year} | {country} | {to_md_link(title, link)} ")
185
+ metadata_str = "\n".join(metadata)
186
+
187
+ markdown_text = f"""
188
+ | Year | Country | Report |
189
+ | --- | --- | --- |
190
+ {metadata_str}
191
+ """
192
+ return markdown_text
193
+
194
+
195
+ def add_user_question(user_question: str, chat_history: Optional[ChatHistory] = None) -> ChatHistory:
196
+ """Adds a user's question to the chat history.
197
+
198
+ If no history is provided, the first element of the history will be the user conversation.
199
+ """
200
+ if chat_history is None:
201
+ chat_history = []
202
+ chat_history.append([user_question, None])
203
+ return chat_history
204
+
205
+
206
+ def chat(chat_history: ChatHistory, reformulate_question: bool, top_k: Optional[int] = None):
207
+ """Answer a user's question using retrieval augmented generation."""
208
+
209
+ # Make sure top k is an int between 1 and 15
210
+ top_k = int(top_k)
211
+ top_k = max(top_k, 1)
212
+ top_k = min(top_k, max_sources)
213
+
214
+ # We assume that the question is the user's last interaction
215
+ user_input = chat_history[-1][0]
216
+
217
+ completion = buster.process_input(user_input, reformulate_question=reformulate_question, top_k=top_k)
218
+
219
+ if completion.question_relevant and not completion.error:
220
+ if reformulate_question and user_input not in cfg.example_questions:
221
+ assert completion.user_inputs.reformulated_input is not None
222
+
223
+ chat_history.append(
224
+ [
225
+ None,
226
+ f"{cfg.message_before_reformulation}{completion.user_inputs.reformulated_input}{cfg.message_after_reformulation}",
227
+ ]
228
+ )
229
+ chat_history.append([None, None])
230
+
231
+ # Stream tokens one at a time
232
+ chat_history[-1][1] = ""
233
+ for token in completion.answer_generator:
234
+ chat_history[-1][1] += token
235
+
236
+ yield chat_history, completion
237
+
238
+
239
+ def log_completion(
240
+ completion: Union[Completion, list[Completion]],
241
+ collection: str,
242
+ session_id: str,
243
+ request: gr.Request,
244
+ instance_type: Optional[str] = cfg.INSTANCE_TYPE,
245
+ instance_name: Optional[str] = cfg.INSTANCE_NAME,
246
+ mongo_db=cfg.mongo_db,
247
+ ):
248
+ """
249
+ Log user completions in a specified collection for analytics.
250
+
251
+ Parameters:
252
+ completion (Union[Completion, list[Completion]]): A single completion or a list of completions
253
+ to log. Completions can be instances of the Completion class.
254
+ collection (str): The name of the MongoDB collection where the interactions will be stored.
255
+ session_id (str): A unique identifier for the current session. In gradio this is reset every time a page is refreshed.
256
+ request (gr.Request): The gradio request object containing request metadata.
257
+ instance_type (str, optional): The type of instance where the completion took place.
258
+ Defaults to cfg.INSTANCE_TYPE.
259
+ instance_name (str, optional): The name of the instance where the completion took place.
260
+ Defaults to cfg.INSTANCE_NAME.
261
+ """
262
+
263
+ # TODO: add UID for each page visitor instead of username
264
+
265
+ # make sure it's always a list
266
+ if isinstance(completion, Completion):
267
+ user_completions = [completion]
268
+ else:
269
+ user_completions = completion
270
+
271
+ interaction = Interaction(
272
+ user_completions=user_completions,
273
+ time=get_utc_time(),
274
+ username=request.username,
275
+ session_id=session_id,
276
+ instance_name=instance_name,
277
+ instance_type=instance_type,
278
+ data_version=cfg.MONGO_DATABASE_DATA,
279
+ )
280
+ interaction.send(mongo_db, collection=collection)
281
+
282
+
283
+ def submit_feedback(
284
+ overall_experience: str,
285
+ clear_answer: str,
286
+ accuracte_answer: str,
287
+ relevant_sources: str,
288
+ relevant_sources_order: list[str],
289
+ relevant_sources_selection: str,
290
+ expertise: list[str],
291
+ extra_info: str,
292
+ completion: Union[Completion, list[Completion]],
293
+ session_id: str,
294
+ request: gr.Request,
295
+ instance_type: Optional[str] = cfg.INSTANCE_TYPE,
296
+ instance_name: Optional[str] = cfg.INSTANCE_NAME,
297
+ ):
298
+ feedback_form = FeedbackForm(
299
+ overall_experience=overall_experience,
300
+ clear_answer=clear_answer,
301
+ accurate_answer=accuracte_answer,
302
+ relevant_sources=relevant_sources,
303
+ relevant_sources_order=relevant_sources_order,
304
+ relevant_sources_selection=relevant_sources_selection,
305
+ expertise=expertise,
306
+ extra_info=extra_info,
307
+ )
308
+
309
+ # make sure it's always a list
310
+ if isinstance(completion, Completion):
311
+ user_completions = [completion]
312
+ else:
313
+ user_completions = completion
314
+
315
+ feedback = Interaction(
316
+ user_completions=user_completions,
317
+ form=feedback_form,
318
+ time=get_utc_time(),
319
+ username=request.username,
320
+ session_id=session_id,
321
+ instance_name=instance_name,
322
+ instance_type=instance_type,
323
+ )
324
+ feedback.send(mongo_db, collection=cfg.MONGO_COLLECTION_FEEDBACK)
325
+
326
+
327
+ def toggle_visibility(visible: bool):
328
+ """Toggles the visibility of the gradio element."""
329
+ return gr.update(visible=visible)
330
+
331
+
332
+ def toggle_interactivity(interactive: bool):
333
+ """Toggles the visibility of the gradio element."""
334
+ return gr.update(interactive=interactive)
335
+
336
+
337
+ def clear_user_input():
338
+ """Clears the contents of the user_input box."""
339
+ return gr.Textbox(value="")
340
+
341
+
342
+ def clear_sources():
343
+ """Clears all the documents in the tabs"""
344
+ return ["" for _ in range(max_sources)]
345
+
346
+
347
+ def clear_feedback_form():
348
+ """Clears the contents of the feedback form."""
349
+ return {
350
+ feedback_elems["overall_experience"]: gr.update(value=None),
351
+ feedback_elems["clear_answer"]: gr.update(value=None),
352
+ feedback_elems["accurate_answer"]: gr.update(value=None),
353
+ feedback_elems["relevant_sources"]: gr.update(value=None),
354
+ feedback_elems["relevant_sources_selection"]: gr.update(value=None),
355
+ feedback_elems["relevant_sources_order"]: gr.update(value=None),
356
+ feedback_elems["expertise"]: gr.update(value=None),
357
+ feedback_elems["extra_info"]: gr.update(value=None),
358
+ }
359
+
360
+
361
+ def reveal_app(choice: gr.SelectData):
362
+ return (
363
+ gr.Group(visible=False),
364
+ gr.Textbox(interactive=True, value=""),
365
+ gr.Button(interactive=True),
366
+ )
367
+
368
+
369
+ def display_sources():
370
+ with gr.Column():
371
+ gr.Markdown(
372
+ """## Relevant sources
373
+ All retrieved documents will be listed here in order of importance.
374
+ """
375
+ )
376
+ sources_textboxes = []
377
+ for i in range(max_sources):
378
+ t = gr.Markdown(latex_delimiters=[], elem_classes="source", visible=False)
379
+ sources_textboxes.append(t)
380
+ return sources_textboxes
381
+
382
+
383
+ def setup_about_panel():
384
+ with gr.Accordion(label=f"About {app_name}", open=False) as about_panel:
385
+ with gr.Row(variant="panel"):
386
+ gr.Markdown(
387
+ f"""
388
+
389
+ ## Welcome
390
+ Artificial intelligence is a field that's developing fast! In response, policy makers from around the world are creating guidelines, rules and regulations to keep up.
391
+
392
+ Finding accurate and up-to-date information about regulatory changes can be difficult but crucial to share best practices, ensure interoperability and promote adherence to local laws and regulations. That's why we've created {app_name}.
393
+
394
+ {app_name} is a Q&A search engine designed to provide relevant and high quality information about AI policies from around the world. Using this tool, your AI policy questions will be answered, accompanied by relevant analyses by the OECD's AI Observatory!
395
+
396
+ ## How it works (and doesn't)
397
+
398
+ {app_name} uses Large Language Models (AI algorithms that work with text) to pinpoint sections of policy documents that are relevant to your question. Rather than presenting you with the specific policy section verbatim, {app_name} has been designed to summarize the information in a digestible format, so that the response you receive more naturally fits with the question you've posed.
399
+
400
+ It's helpful to keep in mind that {app_name} is entirely restricted to our database (see β€œAvailable Sources” below). These sources are from the [OECD.AI](http://oecd.ai/) Database (containing national AI policies) and AI-related reports from the OECD iLibrary. If the answer to your question is not contained in these policy documents, the model won't be able to respond.
401
+
402
+ Since we restrict the model to information found in the documentation, it has a hard time with questions that require more generalized knowledge. Therefore, if you ask the model for information about AI policies in Asia, the model won't necessarily show you Japanese policy documentation. To overcome this limitation, it's best to be as specific as possible in your question, referencing the particular country you're looking for information on.
403
+
404
+ For more information about the tool's strengths and limitations, please see our website [here](https://mila.quebec/en/project/sai/).
405
+ """
406
+ )
407
+
408
+ gr.Markdown(
409
+ f"""
410
+ ## Risks
411
+
412
+ We have done our best to make sure that the AI algorithms are __only__ taking information from what is available in the OECD AI Observatory's Database; but, of course, Large Language Models (LLMs) are prone to fabrication. This means LLMs can make things up and present this made up information as if it were real, making it seem as if the information was found in a policy document. We therefore advise you to check the sources provided by the model to validate that the answer is in fact true. If you'd like to know exactly which documents the model can reference in its response, please see below.
413
+
414
+
415
+ ## Recommended usage
416
+
417
+ {app_name} can only answer specific types of questions, for example:
418
+
419
+ * Questions about policy documents that are currently in the OECD AI Observatory's database
420
+ * Questions that are posed in English and target English language documents;
421
+ * Questions for which the answer can be found in the text (i.e. the thinking has already been done by the author) these AI models are not able to write their own research report combining information across policy documents and analyzing them itself).
422
+
423
+ If your question is outside the scope of the recommended use, the model has been instructed not to answer.
424
+
425
+ We are looking to create a tool that is as inclusive as possible.
426
+ While currently the tool only works with English language questions and documents we will continue assessing {app_name}'s capacity to perform as intended for users with different levels of fluency in English and plan to expand the functionality to ensure accessibility and impact across countries and user groups.
427
+ """
428
+ )
429
+
430
+ return about_panel
431
+
432
+
433
+ def setup_terms_and_conditions():
434
+ with gr.Group(visible=True) as accept_terms_group:
435
+ with gr.Column(scale=1):
436
+ gr.Markdown(
437
+ f"""
438
+ By using this tool you agree to our {md_link_to_tncs}
439
+ """,
440
+ )
441
+ accept_checkbox = gr.Checkbox(value=0, label="I accept", interactive=True, container=False, scale=1)
442
+ return accept_terms_group, accept_checkbox
443
+
444
+
445
+ def setup_additional_sources():
446
+ # Display additional sources
447
+ gr.Markdown(f"")
448
+
449
+ gr.Markdown(
450
+ f"""## πŸ“š Available sources
451
+ {app_name} has access to dozens of AI policy documents from various sources.
452
+ Below we list all of the sources that {app_name} has access to.
453
+ """
454
+ )
455
+ with gr.Accordion(open=False, label="Click to list all available sources πŸ“š"):
456
+ with gr.Column():
457
+ # Display the sources using a dataframe table
458
+ documents_metadata["Report"] = documents_metadata.apply(
459
+ lambda row: to_md_link(row["Title"], row["Link"]), axis=1
460
+ )
461
+ sub_df = documents_metadata[["Country", "Year", "Report"]]
462
+ gr.DataFrame(
463
+ sub_df, headers=list(sub_df.columns), interactive=False, datatype=["number", "str", "markdown"]
464
+ )
465
+
466
+ # Uncomment to display the sources instead as a simple markdown table
467
+ # gr.Markdown(get_metadata_markdown(documents_metadata))
468
+
469
+
470
+ def raise_flagging_message():
471
+ """Raises a red banner indicating that the content has been flagged."""
472
+ gr.Info(
473
+ "Thank you for flagging the content. Our moderation team will look closely at these samples. We apologize for any harm this might have caused you."
474
+ )
475
+
476
+
477
+ def setup_flag_button():
478
+ """Sets up a flag button with some accompanying text explaining why we have it."""
479
+ with gr.Column(variant="compact"):
480
+ gr.Markdown(
481
+ """# Report bugs and harmful content
482
+ While we took many steps to ensure the tool is safe, we still rely on third parties for some of the model's capabilities. Please let us know if any harmful content shows up by clicking the button below and sending screenshots/concerns to mila.databank@gmail.com"""
483
+ )
484
+ flag_button = gr.Button(value="Flag content 🚩")
485
+ return flag_button
486
+
487
+
488
+ def setup_user_settings(
489
+ reformulate_question: bool, visible: bool, num_sources: int, max_sources: int = 15, min_sources: int = 1
490
+ ) -> dict:
491
+ """Set up user interface elements for frontend user settings in a web application.
492
+
493
+ This function creates an accordion containing a slider and a checkbox to configure
494
+ the number of sources and the option to reformulate questions, respectively.
495
+ The values set here will also be the values used by default by the app.
496
+
497
+ Args:
498
+ reformulate_question (bool): Initial state of the checkbox for reformulating questions.
499
+ visible (bool, optional): Visibility state of the settings tab. Defaults to False.
500
+ num_sources (int): Initial value for the number of sources slider. Defaults to 3.
501
+ max_sources (int, optional): Maximum limit for the number of sources slider. Defaults to 15.
502
+ min_sources (int, optional): Minimum limit for the number of sources slider. Defaults to 1.
503
+
504
+ Returns:
505
+ dict: A dictionary containing the UI elements for the reformulate question checkbox and the sources slider.
506
+ """
507
+
508
+ with gr.Accordion(label=f"Settings βš™οΈ", open=False, visible=visible):
509
+ top_k_slider = gr.Slider(
510
+ minimum=min_sources,
511
+ maximum=max_sources,
512
+ interactive=True,
513
+ value=num_sources,
514
+ step=1,
515
+ label="Number of sources",
516
+ info="Number of documents to pass to the language model during its retrieval.",
517
+ )
518
+
519
+ reformulate_question_cbox = gr.Checkbox(
520
+ value=reformulate_question,
521
+ label="Reformulate Question (Beta)",
522
+ info="Reformulates a user's question to enhance source retrieval.",
523
+ )
524
+
525
+ settings_elems = {
526
+ "reformulate_question_cbox": reformulate_question_cbox,
527
+ "top_k_slider": top_k_slider,
528
+ }
529
+ return settings_elems
530
+
531
+
532
+ buster_app = gr.Blocks(css=css)
533
+ with buster_app:
534
+ # State variables are client-side and are reset every time a client refreshes the page
535
+ # Store the users' last completion here
536
+ last_completion = gr.State()
537
+
538
+ # A unique identifier that resets every time a page is refreshed
539
+ session_id = gr.State(get_session_id)
540
+
541
+ gr.Markdown(f"<h1><center>AIR: Q&A tool for AI Policy</center></h1>")
542
+
543
+ about_panel = setup_about_panel()
544
+
545
+ with gr.Row():
546
+ with gr.Column(scale=2, variant="panel"):
547
+ gr.Markdown(
548
+ f"""
549
+ Ask {app_name} your AI policy questions! Keep in mind this tool is a demo and can sometimes provide inaccurate information. Always verify the integrity of the information using the provided sources.
550
+ Since this tool is still in its early stages of development, please only engage with it as a demo.
551
+ """
552
+ )
553
+ accept_terms_group, accept_terms_checkbox = setup_terms_and_conditions()
554
+ with gr.Row():
555
+ with gr.Column(scale=20):
556
+ user_input = gr.Textbox(
557
+ label="",
558
+ placeholder="Ask your AI policy questions here...",
559
+ value="⚠️ Accept the terms and conditions to use the app",
560
+ lines=1,
561
+ interactive=True,
562
+ )
563
+ submit = gr.Button(value="Ask", variant="primary", size="lg", interactive=False)
564
+
565
+ gr.Examples(
566
+ examples=example_questions,
567
+ inputs=user_input,
568
+ label=f"Sample questions to ask {app_name}",
569
+ )
570
+
571
+ chatbot = gr.Chatbot(label="Generated Answer", show_share_button=False)
572
+ sources_textboxes = display_sources()
573
+
574
+ with gr.Column():
575
+ settings_elems = setup_user_settings(
576
+ reformulate_question=cfg.reformulate_question,
577
+ visible=cfg.reveal_user_settings,
578
+ num_sources=cfg.buster_cfg.retriever_cfg["top_k"],
579
+ max_sources=cfg.max_sources,
580
+ )
581
+ top_k_slider = settings_elems["top_k_slider"]
582
+ feedback_elems = setup_feedback_form(top_k_slider.value)
583
+ flag_button = setup_flag_button()
584
+
585
+ top_k_slider.change(
586
+ set_relevant_sources_selection, inputs=top_k_slider, outputs=feedback_elems["relevant_sources_selection"]
587
+ )
588
+
589
+ setup_additional_sources()
590
+
591
+ gr.HTML(
592
+ f"""
593
+ <center>
594
+ <div style='margin-bottom: 20px;'> <!-- Add margin to the bottom of this div -->
595
+ Powered by <a href='https://github.com/jerpint/buster'>Buster</a> πŸ€–
596
+ </div>
597
+
598
+ <div>
599
+ <a href='{path_to_tncs}'> Terms And Conditions </a>
600
+ </div>
601
+ </center>
602
+ """
603
+ )
604
+
605
+ # fmt: off
606
+ # Allow use of submit button and hide checkbox when accepted
607
+ accept_terms_checkbox.select(
608
+ reveal_app,
609
+ outputs=[accept_terms_group, user_input, submit]
610
+ )
611
+
612
+ gr.on(
613
+ triggers=[submit.click, user_input.submit],
614
+ fn=add_user_question,
615
+ inputs=[user_input],
616
+ outputs=[chatbot]
617
+ ).then(
618
+ clear_user_input,
619
+ outputs=[user_input]
620
+ ).then(
621
+ clear_sources,
622
+ outputs=[*sources_textboxes]
623
+ ).then(
624
+ toggle_visibility,
625
+ inputs=gr.State(False),
626
+ outputs=feedback_elems["submitted_message"],
627
+ ).then(
628
+ toggle_interactivity,
629
+ inputs=gr.State(True),
630
+ outputs=feedback_elems["submit_feedback_btn"],
631
+ ).then(
632
+ clear_feedback_form,
633
+ outputs=[
634
+ feedback_elems["overall_experience"],
635
+ feedback_elems["clear_answer"],
636
+ feedback_elems["accurate_answer"],
637
+ feedback_elems["relevant_sources"],
638
+ feedback_elems["relevant_sources_selection"],
639
+ feedback_elems["relevant_sources_order"],
640
+ feedback_elems["expertise"],
641
+ feedback_elems["extra_info"],
642
+ ]
643
+ ).then(
644
+ chat,
645
+ inputs=[chatbot, settings_elems["reformulate_question_cbox"], settings_elems["top_k_slider"]],
646
+ outputs=[chatbot, last_completion],
647
+ ).then(
648
+ add_disclaimer,
649
+ inputs=[last_completion, chatbot, gr.State(cfg.disclaimer)],
650
+ outputs=[chatbot]
651
+ ).then(
652
+ add_sources,
653
+ inputs=[last_completion, gr.State(max_sources)],
654
+ outputs=[*sources_textboxes]
655
+ ).then(
656
+ log_completion,
657
+ inputs=[last_completion, gr.State(cfg.MONGO_COLLECTION_INTERACTION), session_id]
658
+ )
659
+
660
+
661
+ flag_button.click(
662
+ log_completion,
663
+ inputs=[last_completion, gr.State(cfg.MONGO_COLLECTION_FLAGGED), session_id]
664
+ ).then(
665
+ raise_flagging_message,
666
+ )
667
+
668
+ # fmt: on
src/buster/gradio_app.py ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ import os
2
+
3
+ from src.buster.buster_app import buster_app
4
+
5
+ if __name__ == "__main__":
6
+ buster_app.queue(api_open=False)
7
+ buster_app.launch(share=False, show_api=False)
src/cfg.py ADDED
@@ -0,0 +1,252 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import logging
2
+ import os
3
+ from pathlib import Path
4
+
5
+ import openai
6
+
7
+ from buster.busterbot import Buster, BusterConfig
8
+ from buster.completers import ChatGPTCompleter, DocumentAnswerer
9
+ from buster.formatters.documents import DocumentsFormatterJSON
10
+ from buster.formatters.prompts import PromptFormatter
11
+ from buster.llm_utils import QuestionReformulator
12
+ from buster.llm_utils.embeddings import get_openai_embedding_constructor
13
+ from buster.retriever import Retriever, ServiceRetriever
14
+ from buster.tokenizers import GPTTokenizer
15
+ from buster.validators import Validator
16
+ from src.app_utils import get_logging_db_name, init_db
17
+
18
+ logger = logging.getLogger(__name__)
19
+ logging.basicConfig(level=logging.INFO)
20
+
21
+ # Note: The app will not launch if the environment variables aren't set. This is intentional.
22
+ # Set OpenAI Configurations
23
+ openai.api_key = os.environ["OPENAI_API_KEY"]
24
+ openai.organization = os.environ["OPENAI_ORG_ID"]
25
+
26
+ # the embedding function that will get used throughout the app
27
+ embedding_fn = get_openai_embedding_constructor(
28
+ model="text-embedding-ada-002", client_kwargs={"timeout": 2, "max_retries": 2}
29
+ )
30
+
31
+ # Pinecone Configurations
32
+ PINECONE_API_KEY = os.environ["PINECONE_API_KEY"]
33
+ PINECONE_INDEX = "sai-hf"
34
+ PINECONE_NAMESPACE = "data-2024-02-07"
35
+
36
+ # MongoDB Configurations
37
+ MONGO_URI = os.environ["MONGO_URI"]
38
+
39
+ # Instance Configurations
40
+ INSTANCE_NAME = os.environ["INSTANCE_NAME"] # e.g., huggingface, heroku
41
+ INSTANCE_TYPE = os.environ["INSTANCE_TYPE"] # e.g. ["dev", "prod", "local"]
42
+
43
+ # MongoDB Databases
44
+ MONGO_DATABASE_LOGGING = get_logging_db_name(INSTANCE_TYPE) # Where all interactions will be stored
45
+ MONGO_DATABASE_DATA = "data-2024-02-07" # Where documents are stored
46
+
47
+ # Check that data chunks are aligned on Mongo and Pinecone
48
+ if MONGO_DATABASE_DATA != PINECONE_NAMESPACE:
49
+ logger.warning(
50
+ f"""The collection is different on pinecone and Mongo, is this expected?
51
+
52
+ {MONGO_DATABASE_DATA=}
53
+ {PINECONE_NAMESPACE=}
54
+ """
55
+ )
56
+
57
+ # MongoDB Collections
58
+ # Naming convention: Collection name followed by purpose.
59
+ MONGO_COLLECTION_FEEDBACK = "feedback" # Feedback form
60
+ MONGO_COLLECTION_INTERACTION = "interaction" # User interaction
61
+ MONGO_COLLECTION_FLAGGED = "flagged" # Flagged interactions
62
+
63
+ # Make the connections to the databases
64
+ mongo_db = init_db(mongo_uri=MONGO_URI, db_name=MONGO_DATABASE_LOGGING)
65
+
66
+
67
+ # Set relative path to data dir
68
+ current_dir = Path(__file__).resolve().parent
69
+ data_dir = current_dir.parent / "data" # ../data
70
+
71
+ app_name = "AIR οΈπŸ’¬"
72
+
73
+ # User settings default values
74
+ reveal_user_settings = False # Wheter to display settings to the user or not
75
+ max_sources = 15 # maximum number of sources that can be set by a user for retrieval
76
+ reformulate_question = False # Default setting for reformulating a user's question
77
+
78
+ # sample questions
79
+ example_questions = [
80
+ "Are there any AI policies related to AI adoption in the public sector in the UK?",
81
+ "How is Canada evaluating the success of its AI strategy?",
82
+ "Has the EU proposed specific legislation on AI?",
83
+ ]
84
+
85
+
86
+ disclaimer = f"""
87
+ **Use the feedback form on the right to help us improve** πŸ‘‰
88
+
89
+ **Always verify the integrity of {app_name} responses using the sources provided below** πŸ‘‡
90
+ """
91
+
92
+ message_before_reformulation = "I reformulated your answer to: '"
93
+ message_after_reformulation = (
94
+ "'\n\nThis is done automatically to increase performance of the tool. You can disable this in the Settings βš™οΈ tab."
95
+ )
96
+
97
+ # default client config for OpenAI Completions
98
+ client_kwargs = {
99
+ "api_key": os.environ["OPENAI_API_KEY"],
100
+ "organization": os.environ["OPENAI_ORG_ID"],
101
+ "timeout": 10,
102
+ "max_retries": 2,
103
+ }
104
+
105
+
106
+ buster_cfg = BusterConfig(
107
+ validator_cfg={
108
+ "use_reranking": True, # Reranks documents according to generated answer
109
+ "validate_documents": False, # Validates documents using chatGPT (expensive)
110
+ "answer_validator_cfg": {
111
+ "unknown_response_templates": [
112
+ "I cannot answer this question based on the information I have available",
113
+ "The information I have access to does not address the question",
114
+ ],
115
+ "unknown_threshold": 0.84,
116
+ "embedding_fn": embedding_fn,
117
+ },
118
+ "question_validator_cfg": {
119
+ "invalid_question_response": """Thank you for your question! Unfortunately, I haven't been able to find the information you're looking for. Your question might be:
120
+ * Outside the scope of AI policy documents
121
+ * Too recent (i.e. draft policies) or about the future
122
+ * Building on my previous answer (I have no memory of previous conversations)
123
+ * Vague (i.e not affiliated with a specific country)
124
+ * Asking the model to perform its own assessment of the policies (i.e. What is the best/worst AI policy)
125
+ You can always try rewording your question and ask again!
126
+ """,
127
+ "check_question_prompt": """You are a chatbot answering questions on behalf of the OECD specifically on AI policies.
128
+ Your first job is to determine whether or not a question is valid, and should be answered.
129
+ For a question to be considered valid, it must be related to AI and policies.
130
+ More general questions are not considered valid, even if you might know the response.
131
+ A user will submit a question. Respond 'true' if it is valid, respond 'false' if it is invalid.
132
+ Do not judge the tone of the question. As long as it is relevant to the topic, respond 'true'.
133
+
134
+ For example:
135
+ Q: What policies did countries like Canada put in place with respect to artificial intelligence?
136
+ true
137
+
138
+ Q: What policies are put in place to ensure the wellbeing of agriculture?
139
+ false
140
+
141
+ Q:
142
+ """,
143
+ "completion_kwargs": {
144
+ "model": "gpt-3.5-turbo-0613",
145
+ "stream": False,
146
+ "temperature": 0,
147
+ },
148
+ "client_kwargs": client_kwargs,
149
+ },
150
+ },
151
+ retriever_cfg={
152
+ "pinecone_api_key": PINECONE_API_KEY,
153
+ "pinecone_index": PINECONE_INDEX,
154
+ "pinecone_namespace": PINECONE_NAMESPACE,
155
+ "mongo_uri": MONGO_URI,
156
+ "mongo_db_name": MONGO_DATABASE_DATA,
157
+ "top_k": 3,
158
+ "thresh": 0.7,
159
+ "embedding_fn": embedding_fn,
160
+ },
161
+ documents_answerer_cfg={
162
+ "no_documents_message": "No documents are available for this question.",
163
+ },
164
+ completion_cfg={
165
+ "completion_kwargs": {
166
+ "model": "gpt-3.5-turbo-0613",
167
+ "stream": True,
168
+ "temperature": 0,
169
+ },
170
+ "client_kwargs": client_kwargs,
171
+ },
172
+ tokenizer_cfg={
173
+ "model_name": "gpt-3.5-turbo-0613",
174
+ },
175
+ documents_formatter_cfg={
176
+ "max_tokens": 3500,
177
+ "columns": ["content", "source", "title"],
178
+ },
179
+ question_reformulator_cfg={
180
+ "completion_kwargs": {
181
+ "model": "gpt-3.5-turbo",
182
+ "stream": False,
183
+ "temperature": 0,
184
+ },
185
+ "system_prompt": """
186
+ Your role is to reformat a user's input into a question that is useful in the context of a semantic retrieval system.
187
+ Reformulate the question in a way that captures the original essence of the question while also adding more relevant details that can be useful in the context of semantic retrieval.""",
188
+ },
189
+ prompt_formatter_cfg={
190
+ "max_tokens": 4000,
191
+ "text_before_docs": (
192
+ "You are a chatbot assistant answering questions about artificial intelligence (AI) policies and laws. "
193
+ "You represent the OECD AI Policy Observatory. "
194
+ "You can only respond to a question if the content necessary to answer the question is contained in the information provided to you. "
195
+ "The information will be provided in a json format. "
196
+ "If the answer is found in the information provided, summarize it in a helpful way to the user. "
197
+ "If it isn't, simply reply that you cannot answer the question. "
198
+ "Do not refer to the documents directly, but use the information provided within it to answer questions. "
199
+ "Always cite which document you pulled information from. "
200
+ "Do not say 'according to the documentation' or related phrases. "
201
+ "Do not mention the documents directly, but use the information available within them to answer the question. "
202
+ "You are forbidden from using the expressions 'according to the documentation' and 'the provided documents'. "
203
+ "Here is the information available to you in a json table:\n"
204
+ ),
205
+ "text_after_docs": (
206
+ "REMEMBER:\n"
207
+ "You are a chatbot assistant answering questions about artificial intelligence (AI) policies and laws. "
208
+ "You represent the OECD AI Policy Observatory. "
209
+ "Here are the rules you must follow:\n"
210
+ "1) You must only respond with information contained in the documents above. Say you do not know if the information is not provided.\n"
211
+ "2) Make sure to format your answers in Markdown format, including code block and snippets.\n"
212
+ "3) Do not reference any links, urls or hyperlinks in your answers.\n"
213
+ "4) Do not mention the documentation directly, but use the information provided within it to answer questions.\n"
214
+ "5) You are forbidden from using the expressions 'according to the documentation' and 'the provided documents'.\n"
215
+ "6) If you do not know the answer to a question, or if it is completely irrelevant to the library usage, simply reply with:\n"
216
+ "'I'm sorry, but I am an AI language model trained to assist with questions related to AI policies and laws. I cannot answer that question as it is not relevant to AI policies and laws. Is there anything else I can assist you with?'\n"
217
+ "For example:\n"
218
+ "Q: What is the meaning of life for a qa bot?\n"
219
+ "A: I'm sorry, but I am an AI language model trained to assist with questions related to AI policies and laws. I cannot answer that question as it is not relevant to AI policies and laws. Is there anything else I can assist you with?\n"
220
+ "7) Always cite which document you pulled information from. Do this directly in the text. You can refer directly to the title in-line with your answer. Make it clear when information came directly from a source. "
221
+ "8) If the information available to you does not directly address the question, simply state that you do not have the information required to answer. Do not summarize what is available to you. "
222
+ "For example, say: 'I cannot answer this question based on the information I have available.'\n"
223
+ "9) Keep a neutral tone, and put things into context.\n"
224
+ "For example:\n"
225
+ "Q: What do African countries say about data privacy?\n"
226
+ "A: There are currently 28 countries in Africa that have personal data protection legislation. However, limited resources, a lack of clear leadership in the region and localized approaches with regard to data-driven technology run the risk of creating an unfavorable environment for data privacy regulation from a business and data rights standpoint. For example, without policies for data sharing across countries, multinational data companies may choose to move their foreign direct investment to more favorable destinations. African countries also grapple with issues of higher priority, which stunts progress in the field of data privacy. According to one regional policy expert, β€œa government that is still battling [to set up a] school feeding programme in 2019 is not going to be the one to prioritise data and data protection policies with respect to AI.”\n"
227
+ "Now answer the following question:\n"
228
+ ),
229
+ },
230
+ )
231
+
232
+
233
+ def setup_buster(buster_cfg):
234
+ retriever: Retriever = ServiceRetriever(**buster_cfg.retriever_cfg)
235
+ tokenizer = GPTTokenizer(**buster_cfg.tokenizer_cfg)
236
+ document_answerer: DocumentAnswerer = DocumentAnswerer(
237
+ completer=ChatGPTCompleter(**buster_cfg.completion_cfg),
238
+ documents_formatter=DocumentsFormatterJSON(tokenizer=tokenizer, **buster_cfg.documents_formatter_cfg),
239
+ prompt_formatter=PromptFormatter(tokenizer=tokenizer, **buster_cfg.prompt_formatter_cfg),
240
+ **buster_cfg.documents_answerer_cfg,
241
+ )
242
+ validator = Validator(**buster_cfg.validator_cfg)
243
+
244
+ question_reformulator = QuestionReformulator(**buster_cfg.question_reformulator_cfg)
245
+
246
+ buster: Buster = Buster(
247
+ retriever=retriever,
248
+ document_answerer=document_answerer,
249
+ validator=validator,
250
+ question_reformulator=question_reformulator,
251
+ )
252
+ return buster
src/feedback.py ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import logging
4
+ from dataclasses import dataclass
5
+ from typing import Any, Type
6
+
7
+ import pandas as pd
8
+ import pymongo
9
+ from fastapi.encoders import jsonable_encoder
10
+ from pyparsing import Optional
11
+
12
+ from buster.completers import UserInputs
13
+ from buster.completers.base import Completion
14
+
15
+ logger = logging.getLogger(__name__)
16
+ logging.basicConfig(level=logging.INFO)
17
+
18
+
19
+ @dataclass
20
+ class StandardForm:
21
+ def to_json(self) -> Any:
22
+ return jsonable_encoder(self)
23
+
24
+ @classmethod
25
+ def from_dict(cls, interaction_dict: dict) -> StandardForm:
26
+ return cls(**interaction_dict)
27
+
28
+
29
+ @dataclass
30
+ class FeedbackForm(StandardForm):
31
+ """Form on the original Buster app."""
32
+
33
+ # Overall experience
34
+ overall_experience: str
35
+
36
+ # Answer Quality
37
+ clear_answer: str
38
+ accurate_answer: str
39
+
40
+ # Source Relevance
41
+ relevant_sources: str
42
+ relevant_sources_order: str
43
+ relevant_sources_selection: list
44
+
45
+ # beginner, intermediate, expert at AI policy?
46
+ expertise: list[str]
47
+
48
+ # Additional Feedback
49
+ extra_info: str
50
+
51
+
52
+ @dataclass
53
+ class ComparisonForm(StandardForm):
54
+ """Easily readable comparison result on the battle arena."""
55
+
56
+ question: str
57
+ model_left: str
58
+ model_right: str
59
+ vote: str
60
+ extra_info: str
61
+
62
+
63
+ @dataclass
64
+ class Interaction:
65
+ user_completions: list[Completion]
66
+ time: str
67
+ session_id: str # A unique identifier for each gradio session, e.g. UUID
68
+ username: Optional[str] = None
69
+ instance_type: Optional[str] = None # Dev or prod
70
+ instance_name: Optional[str] = None # Heroku, hf-space, etc.
71
+ data_version: Optional[str] = None # Which collection of the was used
72
+ form: Optional[StandardForm] = None
73
+
74
+ def send(self, mongo_db: pymongo.database.Database, collection: str):
75
+ feedback_json = self.to_json()
76
+ logger.info(feedback_json)
77
+
78
+ try:
79
+ mongo_db[collection].insert_one(feedback_json)
80
+ logger.info(f"response logged to mondogb {collection=}")
81
+ except Exception as err:
82
+ logger.exception(f"Something went wrong logging to mongodb {collection=}")
83
+ raise err
84
+
85
+ def flatten(self) -> dict:
86
+ """Flattens the Interaction object into a dict for easier reading."""
87
+ interaction_dict = self.to_json()
88
+
89
+ # Flatten user completions, only keep the most recent interaction
90
+ if len(interaction_dict["user_completions"]) > 0:
91
+ completion_dict = interaction_dict["user_completions"][-1]
92
+ # # TODO: add test for this...
93
+ for k in completion_dict.keys():
94
+ interaction_dict[f"completion_{k}"] = completion_dict[k]
95
+ del interaction_dict["user_completions"]
96
+
97
+ if self.form is not None:
98
+ # Flatten feedback form
99
+ for k in interaction_dict["form"].keys():
100
+ interaction_dict[f"form_{k}"] = interaction_dict["form"][k]
101
+ del interaction_dict["form"]
102
+
103
+ # Flatten matched documents
104
+ interaction_dict["matched_documents"] = self.user_completions[-1].matched_documents
105
+ interaction_dict["matched_documents"].reset_index(inplace=True)
106
+ interaction_dict["matched_documents"].drop(columns=["index"], inplace=True)
107
+ interaction_dict["matched_documents"] = interaction_dict["matched_documents"].T
108
+ if len(interaction_dict["matched_documents"]) > 0:
109
+ for k in interaction_dict["matched_documents"].keys():
110
+ interaction_dict[f"matched_documents_{k}"] = interaction_dict["matched_documents"][k].values
111
+ del interaction_dict["matched_documents"]
112
+
113
+ return interaction_dict
114
+
115
+ def to_json(self) -> Any:
116
+ custom_encoder = {
117
+ # Converts the matched_documents in the user_completions to json
118
+ Completion: lambda completion: completion.to_json(columns_to_ignore=["embedding", "_id"]),
119
+ }
120
+
121
+ to_encode = {
122
+ "username": self.username,
123
+ "session_id": self.session_id,
124
+ "user_completions": self.user_completions,
125
+ "time": self.time,
126
+ "instance_type": self.instance_type,
127
+ "instance_name": self.instance_name,
128
+ "data_version": self.data_version,
129
+ }
130
+
131
+ if self.form is not None:
132
+ to_encode["form"] = self.form.to_json()
133
+
134
+ return jsonable_encoder(to_encode, custom_encoder=custom_encoder)
135
+
136
+ @classmethod
137
+ def from_dict(cls, interaction_dict: dict, feedback_cls: Optional[Type[StandardForm]] = None) -> Interaction:
138
+ # remove the _id from mongodb
139
+ if "_id" in interaction_dict.keys():
140
+ del interaction_dict["_id"]
141
+
142
+ interaction_dict["user_completions"] = [Completion.from_dict(r) for r in interaction_dict["user_completions"]]
143
+
144
+ if "form" in interaction_dict.keys():
145
+ # The interaction contained a type of form, e.g. feedback form, parse it accordingly
146
+
147
+ # Make sure the user specified a feedback_cls
148
+ assert feedback_cls is not None, "You must specify which type of feedback it is"
149
+
150
+ interaction_dict["form"] = feedback_cls.from_dict(interaction_dict["form"])
151
+
152
+ return cls(**interaction_dict)
153
+
154
+
155
+ def read_collection(
156
+ mongo_db: pymongo.database.Database,
157
+ collection: str,
158
+ feedback_cls: Optional[Type[StandardForm]] = None,
159
+ filters: Optional[dict] = None,
160
+ ) -> pd.DataFrame:
161
+ """
162
+ Retrieve data from a MongoDB collection and return it as a pandas DataFrame.
163
+
164
+ Parameters:
165
+ - mongo_db (pymongo.database.Database): The MongoDB database instance.
166
+ - collection (str): The name of the MongoDB collection to read from.
167
+ - feedback_cls (Optional[Type[StandardForm]]): A class to which the retrieved data might be mapped.
168
+ If the collection contains instances of Interaction, this is not needed. If a form is attached
169
+ (i.e., interaction["form"] exists), it should be provided.
170
+ - filters (Optional[dict]): A dictionary of filters to apply to the mongodb query. If not provided,
171
+ all items in the collection are returned. E.g., to get interactions from a specific user,
172
+ use `filters={"username": <username>}`.
173
+
174
+ Returns:
175
+ - pd.DataFrame: A DataFrame containing the retrieved data. Data is flattened for convenience.
176
+
177
+ Notes:
178
+ - Interactions that cannot be processed are skipped, and a log message is generated with the
179
+ count of retrieved and skipped entries.
180
+ """
181
+ flattened_interactions = []
182
+ skipped_interactions = []
183
+ interactions = mongo_db[collection].find(filters)
184
+ for interaction in interactions:
185
+ try:
186
+ if user_input := interaction["user_completions"][0].get("user_input"):
187
+ # We used to only have a single key for user input
188
+ # This changed when we introduced question reformulation.
189
+ # Only useful to maintain backwards compatibility with data collected previously
190
+ interaction["user_completions"][0]["user_inputs"] = UserInputs(user_input)
191
+ del interaction["user_completions"][0]["user_input"]
192
+
193
+ flattened_interaction = Interaction.from_dict(interaction, feedback_cls=feedback_cls).flatten()
194
+ flattened_interactions.append(flattened_interaction)
195
+ except Exception as err:
196
+ skipped_interactions.append(interaction)
197
+
198
+ logger.info(f"Retrieved {len(flattened_interactions)} entries. Skipped {len(skipped_interactions)} entries")
199
+
200
+ return pd.DataFrame(flattened_interactions)