Upload 7 files

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Saied Alshahrani
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,13 +1,48 @@
----
-title: Wikipedia Corpora Report
-emoji: 📈
-colorFrom: gray
-colorTo: pink
-sdk: streamlit
-sdk_version: 1.33.0
-app_file: app.py
-pinned: false
-license: mit
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Wikipedia Corpora Meta Report
+We, in this repository, share with the community our Python scripts for the “**Wikipedia Corpora Meta Report**”, an online metadata report (dashboard), designed to shed light on how bots or humans generate or edit Wikipedia editions to provide the NLP community with detailed information (metadata) about each Wikipedia edition’s articles, enabling them to make informed decisions regarding using these Wikipedia articles for training their NLP tasks and systems. 
+
+This dashboard interactively displays the metadata of each Wikipedia edition using sunburst visualization and provides users with the options to view the metadata in a tabular format and to download the displayed metadata as a CSV file. The dashboard is open-sourced on GitHub with an MIT license and publicly hosted on Streamlit Community Cloud at [https://wikipedia-corpora-report.app](https://wikipedia-corpora-report.streamlit.app/).
+
+This dashboard was presented as a *transparency* tool in our **accepted** paper, [**Performance Implications of Using Unrepresentative Corpora in Arabic Natural Language Processing**](https://aclanthology.org/2023.arabicnlp-1.19.pdf), at [*The First Arabic Natural Language Processing Conference (ArabicNLP 2023)*](https://sites.google.com/view/wanlp2023), co-located with [EMNLP 2023](https://2023.emnlp.org/) in Singapore (hybrid conference), December 7, 2023.
+
+
+### Local Run of Dashboard
+The dashboard is publicly hosted online on Streamlit Community Cloud, yet if you desire to run the dashboard locally on your machine, follow these steps.
+
+1- Clone the dashboard's GitHub repository to your machine. Use this command in your terminal:
+
+```bash
+git clone https://github.com/SaiedAlshahrani/Wikipedia-Corpora-Report.git
+cd Wikipedia-Corpora-Report 
+```
+
+2- Download the required Python packages. Use this command in your terminal:
+
+```bash
+pip install -r requirements.txt
+```
+
+3- Run Streamlit local server. Use this command in your terminal:
+
+```bash
+streamlit run report.py
+```
+
+
+### BibTeX Citation:
+
+```bash
+@inproceedings{alshahrani-etal-2023-performance,
+    title = "{Performance Implications of Using Unrepresentative Corpora in {A}rabic Natural Language Processing}",
+    author = "Alshahrani, Saied  and Alshahrani, Norah  and Dey, Soumyabrata  and Matthews, Jeanna",
+    booktitle = "Proceedings of the The First Arabic Natural Language Processing Conference (ArabicNLP 2023)",
+    month = December,
+    year = "2023",
+    address = "Singapore (Hybrid)",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2023.arabicnlp-1.19",
+    doi = "10.18653/v1/2023.arabicnlp-1.19",
+    pages = "218--231",
+    abstract = "Wikipedia articles are a widely used source of training data for Natural Language Processing (NLP) research, particularly as corpora for low-resource languages like Arabic. However, it is essential to understand the extent to which these corpora reflect the representative contributions of native speakers, especially when many entries in a given language are directly translated from other languages or automatically generated through automated mechanisms. In this paper, we study the performance implications of using inorganic corpora that are not representative of native speakers and are generated through automated techniques such as bot generation or automated template-based translation. The case of the Arabic Wikipedia editions gives a unique case study of this since the Moroccan Arabic Wikipedia edition (ARY) is small but representative, the Egyptian Arabic Wikipedia edition (ARZ) is large but unrepresentative, and the Modern Standard Arabic Wikipedia edition (AR) is both large and more representative. We intrinsically evaluate the performance of two main NLP upstream tasks, namely word representation and language modeling, using word analogy evaluations and fill-mask evaluations using our two newly created datasets: Arab States Analogy Dataset (ASAD) and Masked Arab States Dataset (MASD). We demonstrate that for good NLP performance, we need both large and organic corpora; neither alone is sufficient. We show that producing large corpora through automated means can be a counter-productive, producing models that both perform worse and lack cultural richness and meaningful representation of the Arabic language and its native speakers.",
+}
+```

packages.txt

@@ -0,0 +1,2 @@
+wget
+firefox-esr

report.py

@@ -0,0 +1,313 @@
+import ssl
+import warnings
+import datasets
+import subprocess
+import pandas as pd
+import urllib.request
+from time import sleep
+import streamlit as st
+from datetime import date
+import plotly.express as px
+from urllib.error import HTTPError
+
+
+warnings.simplefilter("ignore", UserWarning)
+warnings.simplefilter("ignore", FutureWarning)
+pd.options.display.float_format = '{:.2f}'.format
+ssl._create_default_https_context = ssl._create_unverified_context
+
+st.set_page_config(page_title="Wikipedia Corpora Report", page_icon="https://webspace.clarkson.edu/~alshahsf/images/wikipedia1.png")
+
+st.markdown("""
+        <h1 style='text-align: center';>Wikipedia Corpora Meta Report</h1>
+        <h5 style='text-align: center';>A Metadata Report of How Wikipedia Editions Are Generated and Edited</h5>
+""", unsafe_allow_html=True)
+
+
+def fetch_wikis_codes():
+    try:
+        urls = [r'https://en.wikipedia.org/wiki/Statistics_of_Wikipedias',
+                r'https://meta.wikimedia.org/wiki/List_of_Wikipedias']
+
+        for url in urls:
+            try: tables = pd.read_html(url)
+            except urllib.error.HTTPError: continue
+
+            for i in range(len(tables)):
+                dataframe = tables[i]
+                columns = list(dataframe.columns.values)
+
+                if(set(['Language', 'Wiki']).issubset(set(columns))):
+                    wikis_codes = tables[i]
+                    break
+
+        wikis_codes = wikis_codes[['Wiki', 'Language']]
+        wikis_codes = wikis_codes[wikis_codes["Language"].str.contains("(closed)") == False]
+        wikis_codes = wikis_codes.set_index('Wiki').to_dict()['Language']
+        return wikis_codes
+
+    except:
+        wikis_codes = {'en': 'English', 'ceb': 'Cebuano', 'de': 'German', 'sv': 'Swedish', 'fr': 'French', 'nl': 'Dutch', 'ru': 'Russian',
+                       'es': 'Spanish', 'it': 'Italian', 'arz': 'Egyptian Arabic', 'pl': 'Polish', 'ja': 'Japanese', 'zh': 'Chinese', 'vi':
+                       'Vietnamese', 'uk': 'Ukrainian', 'war': 'Waray', 'ar': 'Arabic', 'pt': 'Portuguese', 'fa': 'Persian', 'ca': 'Catalan',
+                       'sr': 'Serbian', 'id': 'Indonesian', 'ko': 'Korean', 'no': 'Norwegian (Bokmål)', 'ce': 'Chechen', 'fi': 'Finnish', 'cs':
+                       'Czech', 'tr': 'Turkish', 'hu': 'Hungarian', 'tt': 'Tatar', 'sh': 'Serbo-Croatian', 'ro': 'Romanian', 'zh-min-nan':
+                       'Southern Min', 'eu': 'Basque', 'ms': 'Malay', 'eo': 'Esperanto', 'he': 'Hebrew', 'hy': 'Armenian', 'da': 'Danish', 'bg':
+                       'Bulgarian', 'cy': 'Welsh', 'sk': 'Slovak', 'azb': 'South Azerbaijani', 'uz': 'Uzbek', 'et': 'Estonian', 'simple':
+                       'Simple English', 'be': 'Belarusian', 'kk': 'Kazakh', 'min': 'Minangkabau', 'el': 'Greek', 'hr': 'Croatian', 'lt': 'Lithuanian',
+                       'gl': 'Galician', 'az': 'Azerbaijani', 'ur': 'Urdu', 'sl': 'Slovene', 'lld': 'Ladin', 'ka': 'Georgian', 'nn': 'Norwegian (Nynorsk)',
+                       'hi': 'Hindi', 'th': 'Thai', 'ta': 'Tamil', 'bn': 'Bengali', 'la': 'Latin', 'mk': 'Macedonian', 'zh-yue': 'Cantonese', 'ast':
+                       'Asturian', 'lv': 'Latvian', 'af': 'Afrikaans', 'tg': 'Tajik', 'my': 'Burmese', 'mg': 'Malagasy', 'mr': 'Marathi', 'sq': 'Albanian',
+                       'bs': 'Bosnian', 'oc': 'Occitan', 'te': 'Telugu', 'ml': 'Malayalam', 'nds': 'Low German', 'be-tarask': 'Belarusian (Taraškievica)',
+                       'br': 'Breton', 'ky': 'Kyrgyz', 'sw': 'Swahili', 'jv': 'Javanese', 'lmo': 'Lombard', 'new': 'Newar', 'pnb': 'Western Punjabi', 'vec':
+                       'Venetian', 'ht': 'Haitian Creole', 'pms': 'Piedmontese', 'ba': 'Bashkir', 'lb': 'Luxembourgish', 'su': 'Sundanese', 'ku': 'Kurdish (Kurmanji)',
+                       'ga': 'Irish', 'szl': 'Silesian', 'is': 'Icelandic', 'fy': 'West Frisian', 'cv': 'Chuvash', 'ckb': 'Kurdish (Sorani)', 'pa': 'Punjabi', 'tl':
+                       'Tagalog', 'an': 'Aragonese', 'wuu': 'Wu Chinese', 'diq': 'Zaza', 'io': 'Ido', 'sco': 'Scots', 'vo': 'Volapük', 'yo': 'Yoruba', 'ne': 'Nepali',
+                       'ia': 'Interlingua', 'kn': 'Kannada', 'gu': 'Gujarati', 'als': 'Alemannic German', 'ha': 'Hausa', 'avk': 'Kotava', 'bar': 'Bavarian', 'crh':
+                       'Crimean Tatar', 'scn': 'Sicilian', 'bpy': 'Bishnupriya Manipuri', 'qu': 'Quechua (Southern Quechua)', 'nv': 'Navajo', 'mn': 'Mongolian', 'xmf':
+                       'Mingrelian', 'ban': 'Balinese', 'si': 'Sinhala', 'tum': 'Tumbuka', 'ps': 'Pashto', 'frr': 'North Frisian', 'os': 'Ossetian', 'mzn': 'Mazanderani',
+                       'bat-smg': 'Samogitian', 'or': 'Odia', 'ig': 'Igbo', 'sah': 'Yakut', 'cdo': 'Eastern Min', 'gd': 'Scottish Gaelic', 'bug': 'Buginese', 'yi': 'Yiddish',
+                       'sd': 'Sindhi', 'ilo': 'Ilocano', 'am': 'Amharic', 'nap': 'Neapolitan', 'li': 'Limburgish', 'bcl': 'Central Bikol', 'fo': 'Faroese', 'gor': 'Gorontalo',
+                       'hsb': 'Upper Sorbian', 'map-bms': 'Banyumasan', 'mai': 'Maithili', 'shn': 'Shan', 'eml': 'Emilian-Romagnol', 'ace': 'Acehnese', 'zh-classical':
+                       'Classical Chinese', 'sa': 'Sanskrit', 'as': 'Assamese', 'wa': 'Walloon', 'ie': 'Interlingue', 'hyw': 'Western Armenian', 'lij': 'Ligurian', 'mhr':
+                       'Meadow Mari', 'zu': 'Zulu', 'sn': 'Shona', 'hif': 'Fiji Hindi', 'mrj': 'Hill Mari', 'bjn': 'Banjarese', 'mni': 'Meitei', 'km': 'Khmer', 'hak':
+                       'Hakka Chinese', 'roa-tara': 'Tarantino', 'pam': 'Kapampangan', 'sat': 'Santali', 'rue': 'Rusyn', 'nso': 'Northern Sotho', 'bh': 'Bihari (Bhojpuri)',
+                       'so': 'Somali', 'mi': 'Māori', 'se': 'Northern Sámi', 'myv': 'Erzya', 'vls': 'West Flemish', 'nds-nl': 'Dutch Low Saxon', 'dag': 'Dagbani', 'sc':
+                       'Sardinian', 'ary': 'Moroccan Arabic', 'co': 'Corsican', 'kw': 'Cornish', 'bo': 'Lhasa Tibetan', 'vep': 'Veps', 'glk': 'Gilaki', 'tk': 'Turkmen', 'kab':
+                       'Kabyle', 'gan': 'Gan Chinese', 'rw': 'Kinyarwanda', 'fiu-vro': 'Võro', 'ab': 'Abkhaz', 'gv': 'Manx', 'ug': 'Uyghur', 'nah': 'Nahuatl', 'zea': 'Zeelandic',
+                       'skr': 'Saraiki', 'frp': 'Franco-Provençal', 'udm': 'Udmurt', 'pcd': 'Picard', 'mt': 'Maltese', 'kv': 'Komi', 'csb': 'Kashubian', 'gn': 'Guarani', 'smn':
+                       'Inari Sámi', 'ay': 'Aymara', 'nrm': 'Norman', 'ks': 'Kashmiri', 'lez': 'Lezgian', 'lfn': 'Lingua Franca Nova', 'olo': 'Livvi-Karelian', 'mwl': 'Mirandese',
+                       'stq': 'Saterland Frisian', 'lo': 'Lao', 'ang': 'Old English', 'mdf': 'Moksha', 'fur': 'Friulian', 'rm': 'Romansh', 'lad': 'Judaeo-Spanish', 'kaa': 'Karakalpak',
+                       'gom': 'Konkani (Goan Konkani)', 'ext': 'Extremaduran', 'koi': 'Permyak', 'tyv': 'Tuvan', 'pap': 'Papiamento', 'av': 'Avar', 'dsb': 'Lower Sorbian', 'ln':
+                       'Lingala', 'dty': 'Doteli', 'tw': 'Twi', 'cbk-zam': 'Chavacano (Zamboanga)', 'dv': 'Maldivian', 'ksh': 'Ripuarian', 'za': 'Zhuang (Standard Zhuang)', 'gag':
+                       'Gagauz', 'bxr': 'Buryat (Russia Buriat)', 'pfl': 'Palatine German', 'lg': 'Luganda', 'szy': 'Sakizaya', 'pag': 'Pangasinan', 'blk': "Pa'O", 'pi': 'Pali',
+                       'tay': 'Atayal', 'haw': 'Hawaiian', 'awa': 'Awadhi', 'inh': 'Ingush', 'krc': 'Karachay-Balkar', 'xal': 'Kalmyk Oirat', 'pdc': 'Pennsylvania Dutch', 'to':
+                       'Tongan', 'atj': 'Atikamekw', 'tcy': 'Tulu', 'arc': 'Aramaic (Syriac)', 'mnw': 'Mon', 'jam': 'Jamaican Patois', 'shi': 'Shilha', 'kbp': 'Kabiye', 'wo':
+                       'Wolof', 'anp': 'Angika', 'kbd': 'Kabardian', 'nia': 'Nias', 'nov': 'Novial', 'om': 'Oromo', 'ki': 'Kikuyu', 'nqo': "N'Ko", 'bi': 'Bislama', 'xh': 'Xhosa',
+                       'tpi': 'Tok Pisin', 'tet': 'Tetum', 'ff': 'Fula', 'roa-rup': 'Aromanian', 'jbo': 'Lojban', 'fj': 'Fijian', 'kg': 'Kongo (Kituba)', 'lbe': 'Lak', 'ty': 'Tahitian',
+                       'guw': 'Gun', 'cu': 'Old Church Slavonic', 'trv': 'Seediq', 'ami': 'Amis', 'srn': 'Sranan Tongo', 'sm': 'Samoan', 'mad': 'Madurese', 'alt': 'Southern Altai',
+                       'ltg': 'Latgalian', 'gcr': 'French Guianese Creole', 'chr': 'Cherokee', 'tn': 'Tswana', 'ny': 'Chewa', 'st': 'Sotho', 'pih': 'Norfuk', 'rmy': 'Romani (Vlax Romani)',
+                       'got': 'Gothic', 'ee': 'Ewe', 'pcm': 'Nigerian Pidgin', 'bm': 'Bambara', 'ss': 'Swazi', 'ts': 'Tsonga', 've': 'Venda', 'kcg': 'Tyap', 'chy': 'Cheyenne', 'rn':
+                       'Kirundi', 'ch': 'Chamorro', 'gur': 'Frafra', 'ik': 'Iñupiaq', 'ady': 'Adyghe', 'pnt': 'Pontic Greek', 'guc': 'Wayuu', 'iu': 'Inuktitut', 'pwn': 'Paiwan', 'sg':
+                       'Sango', 'din': 'Dinka', 'ti': 'Tigrinya', 'kl': 'Greenlandic', 'dz': 'Dzongkha', 'cr': 'Cree', 'ak': 'Akan'}
+        return wikis_codes
+
+
+def run_daemon(args):
+    result = subprocess.run(args, capture_output=True, text=True)
+    try: result.check_returncode()
+    except subprocess.CalledProcessError as exception: raise exception
+
+
+labels = []
+wiki_codes = fetch_wikis_codes()
+for key, value in wiki_codes.items():
+    labels.append(f"{value} ({key})")
+
+# st.markdown("<br>",unsafe_allow_html=True)
+
+selected_language = st.selectbox("Select or Search for a Wikipedia language:", labels, placeholder="Select or Search for a Wikipedia language")
+
+
+@st.cache_data
+def fetch_metadata_dataset():
+        # HF_TOKEN = st.secrets["HF_TOKEN"]
+        dataset = datasets.load_dataset("SaiedAlshahrani/Wikipedia-Corpora-Report", split="train")#, use_auth_token=HF_TOKEN)
+        dataset = dataset.to_pandas()
+        return dataset
+
+dataset = fetch_metadata_dataset()
+
+metadata = dataset[dataset['Wiki'] == selected_language]
+
+retrieval_date = metadata['Retrieval-Date'].iloc[0]
+
+now_date = date.today()
+data_date = date(int(retrieval_date.split('-')[0]), int(retrieval_date.split('-')[1]), int(retrieval_date.split('-')[2]))
+delta = now_date - data_date
+
+# if delta.days > 45: run_daemon(["bash", "update-daemon.sh"])
+
+pages_content_bots = metadata['Values'].iloc[0]
+pages_content_humans = metadata['Values'].iloc[1]
+pages_non_content_bots = metadata['Values'].iloc[2]
+pages_non_content_humans = metadata['Values'].iloc[3]
+
+edits_content_bots = metadata['Values'].iloc[4]
+edits_content_humans = metadata['Values'].iloc[5]
+edits_non_content_bots = metadata['Values'].iloc[6]
+edits_non_content_humans = metadata['Values'].iloc[7]
+
+pages_content_pages = pages_content_bots+pages_content_humans
+pages_non_content_pages = pages_non_content_bots+pages_non_content_humans
+total_pages = pages_content_pages+pages_non_content_pages
+
+edits_content_pages = edits_content_bots+edits_content_humans
+edits_non_content_pages = edits_non_content_bots+edits_non_content_humans
+total_edits = edits_content_pages + edits_non_content_pages
+
+wiki_metadata = pd.DataFrame(metadata).reset_index(drop=True)
+
+col1 , cc, col2 = st.columns([1.5, 1.75, 1], gap="small")
+
+with col1:
+    display_data_table = st.checkbox(f'Display metadata in a table.', value=False)
+
+with cc:
+    st.markdown(f"<p style='color:lightgray;font-family:'IBM Plex Sans',sans-serif;font-size:18px;'> &#9432; Latest Metadata Update: {retrieval_date}</p>", unsafe_allow_html=True)
+
+with col2:
+    download_button = st.download_button(label="Download Metadata", data=wiki_metadata.to_csv().encode('utf-8'),
+                                        file_name=f'{selected_language.split("(")[0].strip(" ")}-Metadata-{retrieval_date}.csv', mime='text/csv',)
+
+fig = px.sunburst(data_frame=wiki_metadata,
+                  path=['Wiki','Metric', 'Sub-Metric', 'Editors'],
+                  values='Values',
+                  branchvalues="total",
+                  color_discrete_sequence=['darkgray', 'black'],
+                  template='xgridoff')
+
+fig.update_traces(textinfo='label+percent parent')
+fig.update_traces(hovertemplate="Label=%{label}<br>Value=%{value}<br>Parent=%{parent}</br>")
+fig.update_layout(margin=dict(t=0, l=0, r=0, b=0))
+# fig.update_layout(uniformtext=dict(minsize=12, mode='hide'))
+fig.add_layout_image(dict(x=.430, y=.615, sizex=0.23, sizey=0.23, opacity=0.22, layer="below",
+                    source="https://upload.wikimedia.org/wikipedia/commons/6/63/Wikipedia-logo.png"))
+
+# st.markdown("<br>",unsafe_allow_html=True)
+
+st.plotly_chart(fig, theme=None, use_container_width=True, config={'displayModeBar': False})
+
+# st.markdown("##")
+# st.markdown("<br>",unsafe_allow_html=True)
+
+
+if display_data_table:
+    table_st_style = """
+            <style>
+                table {
+                    border-collapse: collapse;
+                    border: 1px solid black;
+                    border-spacing: 0;
+                    margin-left: 0;
+                    margin-right: 0;
+                    width: 100%;}
+
+                page {
+                    border-collapse: collapse;}
+
+                td, th, tr {
+                    border: 1px solid black;
+                    padding: 0;}
+
+                .contentTableHeader {
+                    background-color: lightgray;
+                    color: black;}
+            </style>
+            """
+    st.markdown(table_st_style, unsafe_allow_html=True)
+
+    st.markdown(f"""
+                <table border="1" width="100%" cellpadding="0" cellspacing="0">
+                    <thead class="contentTableHeader">
+                        <tr>
+                            <td style="text-align:center"><b>Wikipedia</b></td>
+                            <td style="text-align:center"><b>Totals</b></td>
+                            <td style="text-align:center"><b>Pages</b></td>
+                            <td style="text-align:center"><b>Editors</b></td>
+                        </tr>
+                    </thead>
+                    <tbody style="margin: 0;padding: 0">
+                        <tr>
+                            <td style="text-align:center"; rowspan=8>{selected_language}</td>
+                            <td style="text-align:center"; rowspan=4>Pages ({total_pages:,})</td>
+                            <td style="text-align:center"; rowspan=2>Articles ({pages_content_pages:,})</td>
+                            <td style="text-align:center">Bots ({pages_content_bots:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center">Humans ({pages_content_humans:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center"; rowspan=2>Non-Articles ({pages_non_content_pages:,})</td>
+                            <td style="text-align:center">Bots ({pages_non_content_bots:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center">Humans ({pages_non_content_humans:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center"; rowspan=4>Edits ({total_edits:,})</td>
+                            <td style="text-align:center"; rowspan=2>Articles ({edits_content_pages:,})</td>
+                            <td style="text-align:center">Bots ({edits_content_bots:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center"; >Humans ({edits_content_humans:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center"; rowspan=2>Non-Articles ({edits_non_content_pages:,})</td>
+                            <td style="text-align:center">Bots ({edits_non_content_bots:,})</td>
+                        </tr>
+                        <tr>
+                            <td style="text-align:center">Humans ({edits_non_content_humans:,})</td>
+                        </tr>
+                    </tbody>
+                </table>
+                """, unsafe_allow_html=True)
+
+fonts_style = """
+            <style>
+                    @import url('https://fonts.googleapis.com/css2?family=IBM+Plex+Sans:wght@200&display=swap');
+            </style>
+            """
+st.markdown(fonts_style, unsafe_allow_html=True)
+
+hide_st_style = """
+            <style>
+                    MainMenu {visibility: hidden;}
+                    header {visibility: hidden;}
+                    footer {visibility: hidden;}
+                    button[title="View fullscreen"]{visibility: hidden;}
+            </style>
+            """
+st.markdown(hide_st_style, unsafe_allow_html=True)
+
+footer="""
+        <style>
+            .footer {
+                    position: fixed;
+                    left: 0;
+                    bottom: 0;
+                    width: 100%;
+                    background-color: white;
+                    color: #737373;
+                    text-align: center;}
+
+            .p1 {
+                    font-family: 'IBM Plex Sans', sans-serif;
+                    font-size: 12px}
+
+        </style>
+
+        <div class="footer"> <p class="p1">Copyright © 2023 by Saied Alshahrani<br>Hosted with Streamlit Community Cloud</p> </div>
+
+"""
+st.markdown(footer, unsafe_allow_html=True)
+
+st.markdown("""
+        <style>
+               .block-container {
+                    padding-top: 0rem;
+                    padding-bottom: 0rem;
+                    padding-left: 0rem;
+                    padding-right: 0rem;
+                }
+        </style>
+        """, unsafe_allow_html=True)
+
+st.markdown("""
+        <style>
+        .br {
+            display: block;
+            margin: 0px 0;
+            }
+        </style>
+        """, unsafe_allow_html=True)

requirements.txt

@@ -0,0 +1,8 @@
+lxml==4.9.1
+pandas==1.4.3
+plotly==5.15.0
+datasets==2.14.6
+streamlit==1.30.0
+selenium==3.141.0 
+geckodriver-autoinstaller==0.1.0
+

update-daemon.sh

@@ -0,0 +1,20 @@
+python update-metadata.py
+
+# git lfs install
+git clone https://huggingface.co/datasets/SaiedAlshahrani/Wikipedia-Corpora-Report
+cd Wikipedia-Corpora-Report/
+
+head -n1 ../English--Wikipedia--Metadata.csv > Wikipedia-Corpora-Report.csv
+sed -i '' 1d ../*--Wikipedia--Metadata.csv
+cat ../*--Wikipedia--Metadata.csv >> Wikipedia-Corpora-Report.csv
+# cp -r ../all-metadata .
+
+git add .
+git status
+git commit -m "Update Wikipedia-Corpora-Report.csv"
+git push
+
+rm ../*--Wikipedia--Metadata.csv
+cp Wikipedia-Corpora-Report.csv ..
+cd ..
+

update-metadata.py

@@ -0,0 +1,196 @@
+import selenium
+import os, warnings
+import urllib.request
+from time import sleep
+import pandas as pd, ssl
+from selenium import webdriver
+from urllib.error import HTTPError
+
+warnings.simplefilter("ignore", UserWarning)
+warnings.simplefilter("ignore", FutureWarning)
+pd.options.display.float_format = '{:.2f}'.format
+ssl._create_default_https_context = ssl._create_unverified_context
+
+def fetch_wikis_codes():
+    try:
+        urls = [r'https://en.wikipedia.org/wiki/Statistics_of_Wikipedias', 
+                r'https://meta.wikimedia.org/wiki/List_of_Wikipedias']
+        
+        for url in urls:
+            try: tables = pd.read_html(url)
+            except urllib.error.HTTPError: continue
+                
+            for i in range(len(tables)):
+                dataframe = tables[i]
+                columns = list(dataframe.columns.values)
+        
+                if(set(['Language', 'Wiki']).issubset(set(columns))):
+                    wikis_codes = tables[i]
+                    break
+        
+        wikis_codes = wikis_codes[['Wiki', 'Language']]
+        wikis_codes = wikis_codes[wikis_codes["Language"].str.contains("(closed)") == False]
+        wikis_codes = wikis_codes.set_index('Wiki').to_dict()['Language']
+        return wikis_codes
+    
+    except: 
+        wikis_codes = {'en': 'English', 'ceb': 'Cebuano', 'de': 'German', 'sv': 'Swedish', 'fr': 'French', 'nl': 'Dutch', 'ru': 'Russian',
+                       'es': 'Spanish', 'it': 'Italian', 'arz': 'Egyptian Arabic', 'pl': 'Polish', 'ja': 'Japanese', 'zh': 'Chinese', 'vi': 
+                       'Vietnamese', 'uk': 'Ukrainian', 'war': 'Waray', 'ar': 'Arabic', 'pt': 'Portuguese', 'fa': 'Persian', 'ca': 'Catalan', 
+                       'sr': 'Serbian', 'id': 'Indonesian', 'ko': 'Korean', 'no': 'Norwegian (Bokmål)', 'ce': 'Chechen', 'fi': 'Finnish', 'cs': 
+                       'Czech', 'tr': 'Turkish', 'hu': 'Hungarian', 'tt': 'Tatar', 'sh': 'Serbo-Croatian', 'ro': 'Romanian', 'zh-min-nan': 
+                       'Southern Min', 'eu': 'Basque', 'ms': 'Malay', 'eo': 'Esperanto', 'he': 'Hebrew', 'hy': 'Armenian', 'da': 'Danish', 'bg': 
+                       'Bulgarian', 'cy': 'Welsh', 'sk': 'Slovak', 'azb': 'South Azerbaijani', 'uz': 'Uzbek', 'et': 'Estonian', 'simple': 
+                       'Simple English', 'be': 'Belarusian', 'kk': 'Kazakh', 'min': 'Minangkabau', 'el': 'Greek', 'hr': 'Croatian', 'lt': 'Lithuanian', 
+                       'gl': 'Galician', 'az': 'Azerbaijani', 'ur': 'Urdu', 'sl': 'Slovene', 'lld': 'Ladin', 'ka': 'Georgian', 'nn': 'Norwegian (Nynorsk)', 
+                       'hi': 'Hindi', 'th': 'Thai', 'ta': 'Tamil', 'bn': 'Bengali', 'la': 'Latin', 'mk': 'Macedonian', 'zh-yue': 'Cantonese', 'ast': 
+                       'Asturian', 'lv': 'Latvian', 'af': 'Afrikaans', 'tg': 'Tajik', 'my': 'Burmese', 'mg': 'Malagasy', 'mr': 'Marathi', 'sq': 'Albanian', 
+                       'bs': 'Bosnian', 'oc': 'Occitan', 'te': 'Telugu', 'ml': 'Malayalam', 'nds': 'Low German', 'be-tarask': 'Belarusian (Taraškievica)', 
+                       'br': 'Breton', 'ky': 'Kyrgyz', 'sw': 'Swahili', 'jv': 'Javanese', 'lmo': 'Lombard', 'new': 'Newar', 'pnb': 'Western Punjabi', 'vec': 
+                       'Venetian', 'ht': 'Haitian Creole', 'pms': 'Piedmontese', 'ba': 'Bashkir', 'lb': 'Luxembourgish', 'su': 'Sundanese', 'ku': 'Kurdish (Kurmanji)', 
+                       'ga': 'Irish', 'szl': 'Silesian', 'is': 'Icelandic', 'fy': 'West Frisian', 'cv': 'Chuvash', 'ckb': 'Kurdish (Sorani)', 'pa': 'Punjabi', 'tl': 
+                       'Tagalog', 'an': 'Aragonese', 'wuu': 'Wu Chinese', 'diq': 'Zaza', 'io': 'Ido', 'sco': 'Scots', 'vo': 'Volapük', 'yo': 'Yoruba', 'ne': 'Nepali', 
+                       'ia': 'Interlingua', 'kn': 'Kannada', 'gu': 'Gujarati', 'als': 'Alemannic German', 'ha': 'Hausa', 'avk': 'Kotava', 'bar': 'Bavarian', 'crh': 
+                       'Crimean Tatar', 'scn': 'Sicilian', 'bpy': 'Bishnupriya Manipuri', 'qu': 'Quechua (Southern Quechua)', 'nv': 'Navajo', 'mn': 'Mongolian', 'xmf': 
+                       'Mingrelian', 'ban': 'Balinese', 'si': 'Sinhala', 'tum': 'Tumbuka', 'ps': 'Pashto', 'frr': 'North Frisian', 'os': 'Ossetian', 'mzn': 'Mazanderani', 
+                       'bat-smg': 'Samogitian', 'or': 'Odia', 'ig': 'Igbo', 'sah': 'Yakut', 'cdo': 'Eastern Min', 'gd': 'Scottish Gaelic', 'bug': 'Buginese', 'yi': 'Yiddish', 
+                       'sd': 'Sindhi', 'ilo': 'Ilocano', 'am': 'Amharic', 'nap': 'Neapolitan', 'li': 'Limburgish', 'bcl': 'Central Bikol', 'fo': 'Faroese', 'gor': 'Gorontalo', 
+                       'hsb': 'Upper Sorbian', 'map-bms': 'Banyumasan', 'mai': 'Maithili', 'shn': 'Shan', 'eml': 'Emilian-Romagnol', 'ace': 'Acehnese', 'zh-classical': 
+                       'Classical Chinese', 'sa': 'Sanskrit', 'as': 'Assamese', 'wa': 'Walloon', 'ie': 'Interlingue', 'hyw': 'Western Armenian', 'lij': 'Ligurian', 'mhr': 
+                       'Meadow Mari', 'zu': 'Zulu', 'sn': 'Shona', 'hif': 'Fiji Hindi', 'mrj': 'Hill Mari', 'bjn': 'Banjarese', 'mni': 'Meitei', 'km': 'Khmer', 'hak': 
+                       'Hakka Chinese', 'roa-tara': 'Tarantino', 'pam': 'Kapampangan', 'sat': 'Santali', 'rue': 'Rusyn', 'nso': 'Northern Sotho', 'bh': 'Bihari (Bhojpuri)', 
+                       'so': 'Somali', 'mi': 'Māori', 'se': 'Northern Sámi', 'myv': 'Erzya', 'vls': 'West Flemish', 'nds-nl': 'Dutch Low Saxon', 'dag': 'Dagbani', 'sc': 
+                       'Sardinian', 'ary': 'Moroccan Arabic', 'co': 'Corsican', 'kw': 'Cornish', 'bo': 'Lhasa Tibetan', 'vep': 'Veps', 'glk': 'Gilaki', 'tk': 'Turkmen', 'kab': 
+                       'Kabyle', 'gan': 'Gan Chinese', 'rw': 'Kinyarwanda', 'fiu-vro': 'Võro', 'ab': 'Abkhaz', 'gv': 'Manx', 'ug': 'Uyghur', 'nah': 'Nahuatl', 'zea': 'Zeelandic', 
+                       'skr': 'Saraiki', 'frp': 'Franco-Provençal', 'udm': 'Udmurt', 'pcd': 'Picard', 'mt': 'Maltese', 'kv': 'Komi', 'csb': 'Kashubian', 'gn': 'Guarani', 'smn': 
+                       'Inari Sámi', 'ay': 'Aymara', 'nrm': 'Norman', 'ks': 'Kashmiri', 'lez': 'Lezgian', 'lfn': 'Lingua Franca Nova', 'olo': 'Livvi-Karelian', 'mwl': 'Mirandese', 
+                       'stq': 'Saterland Frisian', 'lo': 'Lao', 'ang': 'Old English', 'mdf': 'Moksha', 'fur': 'Friulian', 'rm': 'Romansh', 'lad': 'Judaeo-Spanish', 'kaa': 'Karakalpak', 
+                       'gom': 'Konkani (Goan Konkani)', 'ext': 'Extremaduran', 'koi': 'Permyak', 'tyv': 'Tuvan', 'pap': 'Papiamento', 'av': 'Avar', 'dsb': 'Lower Sorbian', 'ln': 
+                       'Lingala', 'dty': 'Doteli', 'tw': 'Twi', 'cbk-zam': 'Chavacano (Zamboanga)', 'dv': 'Maldivian', 'ksh': 'Ripuarian', 'za': 'Zhuang (Standard Zhuang)', 'gag': 
+                       'Gagauz', 'bxr': 'Buryat (Russia Buriat)', 'pfl': 'Palatine German', 'lg': 'Luganda', 'szy': 'Sakizaya', 'pag': 'Pangasinan', 'blk': "Pa'O", 'pi': 'Pali', 
+                       'tay': 'Atayal', 'haw': 'Hawaiian', 'awa': 'Awadhi', 'inh': 'Ingush', 'krc': 'Karachay-Balkar', 'xal': 'Kalmyk Oirat', 'pdc': 'Pennsylvania Dutch', 'to': 
+                       'Tongan', 'atj': 'Atikamekw', 'tcy': 'Tulu', 'arc': 'Aramaic (Syriac)', 'mnw': 'Mon', 'jam': 'Jamaican Patois', 'shi': 'Shilha', 'kbp': 'Kabiye', 'wo': 
+                       'Wolof', 'anp': 'Angika', 'kbd': 'Kabardian', 'nia': 'Nias', 'nov': 'Novial', 'om': 'Oromo', 'ki': 'Kikuyu', 'nqo': "N'Ko", 'bi': 'Bislama', 'xh': 'Xhosa', 
+                       'tpi': 'Tok Pisin', 'tet': 'Tetum', 'ff': 'Fula', 'roa-rup': 'Aromanian', 'jbo': 'Lojban', 'fj': 'Fijian', 'kg': 'Kongo (Kituba)', 'lbe': 'Lak', 'ty': 'Tahitian', 
+                       'guw': 'Gun', 'cu': 'Old Church Slavonic', 'trv': 'Seediq', 'ami': 'Amis', 'srn': 'Sranan Tongo', 'sm': 'Samoan', 'mad': 'Madurese', 'alt': 'Southern Altai', 
+                       'ltg': 'Latgalian', 'gcr': 'French Guianese Creole', 'chr': 'Cherokee', 'tn': 'Tswana', 'ny': 'Chewa', 'st': 'Sotho', 'pih': 'Norfuk', 'rmy': 'Romani (Vlax Romani)', 
+                       'got': 'Gothic', 'ee': 'Ewe', 'pcm': 'Nigerian Pidgin', 'bm': 'Bambara', 'ss': 'Swazi', 'ts': 'Tsonga', 've': 'Venda', 'kcg': 'Tyap', 'chy': 'Cheyenne', 'rn': 
+                       'Kirundi', 'ch': 'Chamorro', 'gur': 'Frafra', 'ik': 'Iñupiaq', 'ady': 'Adyghe', 'pnt': 'Pontic Greek', 'guc': 'Wayuu', 'iu': 'Inuktitut', 'pwn': 'Paiwan', 'sg': 
+                       'Sango', 'din': 'Dinka', 'ti': 'Tigrinya', 'kl': 'Greenlandic', 'dz': 'Dzongkha', 'cr': 'Cree', 'ak': 'Akan'}
+        return wikis_codes
+        
+
+def fetch_wiki_metadata(wiki, metric, submetric, timeout):
+    options = webdriver.FirefoxOptions()
+    options.add_argument("--headless")
+    profile = webdriver.FirefoxProfile()
+    profile.set_preference("browser.download.folderList", 2)
+    profile.set_preference("browser.download.manager.showWhenStarting", False)
+    profile.set_preference("browser.download.dir", f"{os.getcwd()}")
+    profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream")
+    driver = webdriver.Firefox(options=options, firefox_profile=profile, executable_path='geckodriver', service_log_path=os.devnull)
+
+    if metric == 'pages':
+        base_url = f'https://stats.wikimedia.org/#/{wiki}.wikipedia.org/content/pages-to-date/full|table|'
+
+    elif metric == 'edits':
+        base_url = f'https://stats.wikimedia.org/#/{wiki}.wikipedia.org/contributing/edits/full|table|'
+
+    parameters = f'1-month|editor_type~anonymous*group-bot*name-bot*user+(page_type)~{submetric}|monthly'
+    request_url = "".join([base_url, parameters])
+
+    driver.implicitly_wait(3)
+    driver.get(request_url)
+    driver.page_source
+
+    sleep(timeout)
+
+    csvFilename = f"{wiki}--{metric}--{submetric}.csv"
+    csvFilename = csvFilename.replace(' ','-')
+    driver.find_element_by_class_name("ui.icon.button.tooltipped.tooltipped-n").click()
+    sleep(3) ; os.rename("undefined.csv", csvFilename)
+
+    driver.close()
+    driver.quit()
+
+    print(f'    [+] Metadata Exported to `{wiki}/{csvFilename}`.')
+    
+    return csvFilename
+
+
+wiki_codes = fetch_wikis_codes()
+labels = []
+for key, value in wiki_codes.items():
+    labels.append(f"{value} ({key})")
+    
+wikis = list(wiki_codes.keys())
+metrics = ['pages', 'edits']
+submetrics = ['content', 'non-content']
+
+timeout = 3
+counter = 1
+
+for wiki in wikis:
+
+    print(f'{counter}## {wiki_codes[wiki]} Wikipedia Files:')
+    if not os.path.exists(f'{wiki}'): os.makedirs(f'{wiki}')
+    if not os.path.exists('all-metadata'): os.makedirs('all-metadata')
+
+    for metric in metrics:
+        
+        for submetric in submetrics:
+            
+            try: 
+                csvFilename = fetch_wiki_metadata(wiki, metric, submetric, timeout)
+                dataframe = pd.read_csv(csvFilename).iloc[-1]
+                
+            except selenium.common.exceptions.ElementClickInterceptedException:
+                dataframe = pd.read_csv(fetch_wiki_metadata(wiki, metric, submetric, timeout*2)).iloc[-1]
+                timeout *= 2
+
+            retrieval_date = pd.to_datetime(dataframe['timeRange.end']).strftime('%Y-%m-%d')
+
+            if metric == 'pages':
+                if submetric == 'content':
+                    pages_content_bots = dataframe['total.group-bot']+dataframe['total.name-bot']
+                    pages_content_humans = dataframe['total.user']+dataframe['total.anonymous']
+                elif submetric == 'non-content':
+                    pages_non_content_bots = dataframe['total.group-bot']+dataframe['total.name-bot']
+                    pages_non_content_humans = dataframe['total.user']+dataframe['total.anonymous']
+                else: print(f'Error: this submetric: {submetric} is not supported!')
+            
+            elif metric == 'edits':
+                if submetric == 'content':
+                    edits_content_bots = dataframe['total.group-bot']+dataframe['total.name-bot']
+                    edits_content_humans = dataframe['total.user']+dataframe['total.anonymous']
+                elif submetric == 'non-content':
+                    edits_non_content_bots = dataframe['total.group-bot']+dataframe['total.name-bot']
+                    edits_non_content_humans = dataframe['total.user']+dataframe['total.anonymous']
+                else: print(f'Error: this submetric: {submetric} is not supported!')
+            
+            else: print(f'Error: this metric: {metric} is not supported!')
+            
+            os.system(f'mv {wiki}--{metric}--{submetric}.csv {wiki}/{wiki}--{metric}--{submetric}.csv')
+            
+    selected_language = f'{wiki_codes[wiki]} ({wiki})'
+
+    metadata = {'Wiki' : [selected_language, selected_language, selected_language, selected_language, 
+                          selected_language, selected_language, selected_language,selected_language],
+                
+                'Metric' : ['Pages', 'Pages', 'Pages', 'Pages', 'Edits', 'Edits', 'Edits', 'Edits'],
+                
+                'Sub-Metric' : ['Articles', 'Articles',  'Non-Articles', 'Non-Articles', 
+                                'Articles', 'Articles',  'Non-Articles', 'Non-Articles'],
+                
+                'Editors' : ['Bots', 'Humans', 'Bots', 'Humans', 'Bots', 'Humans', 'Bots', 'Humans'],
+                
+                'Values' : [pages_content_bots, pages_content_humans, pages_non_content_bots, pages_non_content_humans, 
+                            edits_content_bots, edits_content_humans, edits_non_content_bots, edits_non_content_humans]}
+
+    wiki_metadata = pd.DataFrame(metadata)
+    wiki_metadata['Retrieval-Date'] = retrieval_date
+    wiki_metadata.to_csv(f'{wiki_codes[wiki].replace(" ","-")}--Wikipedia--Metadata.csv', index=False)
+    
+    os.system(f'mv {wiki} all-metadata/')
+    counter = counter + 1
+    sleep(1)